refactor: use modernc sqlite driver as default by wxiaoguang · Pull Request #37562 · go-gitea/gitea

wxiaoguang · 2026-05-06T06:01:33Z

The mattn driver is still kept, can be enabled by TAGS="sqlite_mattn sqlite_unlock_notify"

Doc: Update SQLITE_TIMEOUT #419 https://gitea.com/gitea/docs/pulls/419

Co-authored-by: TheFox0x7 thefox0x7@gmail.com

Copilot

Pull request overview

This PR switches Gitea’s default SQLite backend to the pure-Go modernc.org/sqlite driver (removing the need for CGO by default), while keeping the mattn/go-sqlite3 driver available behind build tags. It also updates build tooling/docs/CI expectations accordingly and adjusts SQLite timeout/test behavior to better fit the new driver’s locking semantics.

Changes:

Make modernc.org/sqlite the default SQLite driver and gate mattn/go-sqlite3 behind sqlite_mattn && sqlite_unlock_notify build tags.
Remove “SQLite requires build tags” guidance from docs and build pipelines (Docker/Nix/Snap/CI), since SQLite support becomes default.
Adjust SQLite busy-timeout defaults and update unit/integration tests to reduce flakiness/locking issues.

Reviewed changes

Copilot reviewed 33 out of 34 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/integration/api_issue_test.go	Reduces sleep in a concurrency-related SQLite integration test and rewords comment marker.
snap/snapcraft.yaml	Removes SQLite-related build tags from the snap build.
routers/install/install.go	Simplifies DB init error handling by removing the sqlite3-not-available special case.
README.md	Removes “build with SQLite tags” instructions.
README.zh-cn.md	Removes “build with SQLite tags” instructions (zh-cn).
README.zh-tw.md	Removes “build with SQLite tags” instructions (zh-tw).
options/locale/locale_en-US.json	Removes the `install.sqlite3_not_available` translation key.
modules/setting/database.go	Raises default `SQLITE_TIMEOUT` and introduces minimum/override logic.
models/unittest/testdb.go	Changes unit test DB creation to use a temp sqlite file and adjusts connection options.
models/unittest/fixtures_test.go	Updates fixtures loader tests to use temp sqlite files with the new CreateTestEngine signature.
models/issues/issue_test.go	Adjusts concurrency patterns in issue tests and mocks max query parameter behavior for chunking.
models/issues/issue_stats.go	Changes `MaxQueryParameters` from const to var to enable test-time overrides.
models/db/driver_sqlite_modernc.go	Adds the default modernc SQLite driver and connection string builder.
models/db/driver_sqlite_mattn.go	Retags mattn driver behind `sqlite_mattn && sqlite_unlock_notify`.
models/db/conn.go	Updates the “SQLite not built in” error message and adds docs around busy timeout meaning.
Makefile	Stops auto-adding sqlite build tags for sqlite tests; changes CGO enablement to only `sqlite_mattn`/`pam`.
go.mod	Adds `modernc.org/sqlite` and related indirect deps.
go.sum	Updates dependency checksums for modernc sqlite stack and related indirect deps.
flake.nix	Removes default SQLite build tags from Nix build env.
Dockerfile	Removes default SQLite build tags from Docker build args.
Dockerfile.rootless	Removes default SQLite build tags from rootless Docker build args.
custom/conf/app.example.ini	Updates documented default SQLite timeout to 20000ms.
contrib/ide/vscode/tasks.json	Removes the “Build (with SQLite3)” VS Code task.
contrib/ide/vscode/settings.json	Clears Go build tags in VS Code settings (but currently sets an invalid literal).
contrib/ide/vscode/launch.json	Removes the “Launch (with SQLite3)” VS Code launch config.
assets/go-licenses.json	Adds license entries for new modernc/related deps.
AGENTS.md	Updates guidance for running single Go tests without SQLite tags.
.github/workflows/release-tag-version.yml	Removes SQLite build tags from release workflow build env.
.github/workflows/release-tag-rc.yml	Removes SQLite build tags from RC release workflow build env.
.github/workflows/release-nightly.yml	Removes SQLite build tags from nightly release workflow build env.
.github/workflows/pull-e2e-tests.yml	Removes SQLite build tags from PR e2e workflow builds/tests.
.github/workflows/pull-db-tests.yml	Removes SQLite build tags; tweaks race usage (pgsql keeps race, sqlite integration drops race).
.github/workflows/pull-compliance.yml	Removes SQLite build tags from lint/compliance workflows.
.github/workflows/cache-seeder.yml	Removes SQLite-tagged cache seeding builds, keeping only bindata/gogit variants.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 4 comments.

lunny · 2026-05-06T16:55:27Z

The official release will no longer depend on xgo compilation after this merged, which should make the release process faster.

lunny · 2026-05-06T17:31:47Z

~~When build with TAGS="sqlite_mattn,sqlite_unlock_notify,bindata" make build and run Gitea, there is no sqlite support.~~

wxiaoguang · 2026-05-06T17:43:01Z

When build with TAGS="sqlite_mattn,sqlite_unlock_notify,bindata" make build and run Gitea, there is no sqlite support.

TAGS="sqlite_mattn sqlite_unlock_notify bindata" make build

./gitea

TheFox0x7 · 2026-05-06T18:05:03Z

Thanks for taking time to bring it to close. Hope my tinkering was helpful as a reference :)

wxiaoguang · 2026-05-06T18:07:48Z

Hope my tinkering was helpful as a reference :)

Yes, it is really helpful. Thank you very much!

Will add "Co-authored-by" when squash merge.

silverwind · 2026-05-06T18:20:18Z

No race detector for integration test seems a bit worrysome, but it's also been a long time since it last detected anything.

wxiaoguang · 2026-05-06T18:21:01Z

No race detector for integration test seems a bit worrysome, but it's also been a long time since it last detected anything.

Are you sure? Have you read the code? Why "no race detector for integration test"?

wxiaoguang · 2026-05-07T05:46:28Z

To reduce CI time more: refactor: only reset a database table when the table's data was changed #37573

Saves about 3 minutes for each test suit

test-unit: 13min -> 10min (-race)
test-pgsql: 24min -> 20min (-race)
test-mysql: 15min -> 12min
test-mssql: 16min -> 12min

silverwind · 2026-05-07T08:54:11Z

More saving could be done by having cache-seeder also run some tests to seed go test cache. IIRC I skipped on that in its implementation of the seeder because with the 10GB space limitation of GHA cache, the cache is already very size-constrained, but maybe something can be done there with partial tests or we find a bigger cache, likely 20GB-30GB would be sufficient.

* origin/main: (41 commits) refactor: only reset a database table when the table's data was changed (go-gitea#37573) refactor: lint bare `fill`/`stroke` colors, add vars for git graph color series (go-gitea#37543) fix(actions): report individual step status in workflow job API response (go-gitea#37592) ci: allow `chore` type in PR title lint (go-gitea#37575) refactor: remove obsolete export (go-gitea#37579) fix: Invalid UTF-8 commit messages in JSON API responses (go-gitea#37542) fix: use consistent GetUser family functions (go-gitea#37553) fix(api): return 409 message instead of empty JSON for wrong commit id (go-gitea#37572) fix(actions): prevent panic when workflow contains null jobs (go-gitea#37570) [skip ci] Updated translations via Crowdin refactor: use modernc sqlite driver as default (go-gitea#37562) refactor(templates): remove ctxData from tmpl files, use ctx.RootData instead (go-gitea#37567) ci: increase renovate frequency and fix RENOVATE_ALLOWED_POST_UPGRADE_COMMANDS (go-gitea#37565) fix(deps): update go dependencies (go-gitea#37541) refactor(deps): migrate from `nektos/act` fork to `gitea/runner` (go-gitea#37557) ci: lint PR titles with commitlint (go-gitea#37498) Make ServeSetHeaders default to download attachment if filename exists (go-gitea#37552) fix(actions): validate workflow param to prevent 500 error (go-gitea#37546) Fix various problems (go-gitea#37547) docs: fix 4 typos in CHANGELOG.md (go-gitea#37549) ... # Conflicts: # templates/repo/actions/runs_list.tmpl # templates/repo/issue/view_content/pull_merge_box.tmpl # templates/repo/pulls/status.tmpl

* main: chore(deps): bump tool deps and pin, update `golangci-lint` (go-gitea#37574) build: update pnpm to v11 (go-gitea#37591) chore(renovate): detect Makefile tools, enable `go` bumps (go-gitea#37595) refactor: only reset a database table when the table's data was changed (go-gitea#37573) refactor: lint bare `fill`/`stroke` colors, add vars for git graph color series (go-gitea#37543) fix(actions): report individual step status in workflow job API response (go-gitea#37592) ci: allow `chore` type in PR title lint (go-gitea#37575) refactor: remove obsolete export (go-gitea#37579) fix: Invalid UTF-8 commit messages in JSON API responses (go-gitea#37542) fix: use consistent GetUser family functions (go-gitea#37553) fix(api): return 409 message instead of empty JSON for wrong commit id (go-gitea#37572) fix(actions): prevent panic when workflow contains null jobs (go-gitea#37570) [skip ci] Updated translations via Crowdin refactor: use modernc sqlite driver as default (go-gitea#37562) refactor(templates): remove ctxData from tmpl files, use ctx.RootData instead (go-gitea#37567) ci: increase renovate frequency and fix RENOVATE_ALLOWED_POST_UPGRADE_COMMANDS (go-gitea#37565) # Conflicts: # models/user/user.go

The modernc SQLite driver (default since #37562) returns `SQLITE_BUSY` once the busy timeout is reached, unlike mattn which waited indefinitely. `TestResourceIndex` fires many concurrent `NewIssue` writers, but SQLite serializes all writers, so they queue on a single `BEGIN IMMEDIATE` write lock. Under `-race` (modernc is much slower) the goroutines at the back of the queue exceeded the hardcoded 5s test timeout, producing `database is locked (5) (SQLITE_BUSY)`. Changes: - Reduce the concurrent inserts from 25 to 10. Since SQLite serializes writers, the extra goroutines only deepen the busy-lock queue without adding coverage. 10 still exercises concurrent index allocation while cutting the test's `-race` runtime ~3x (2.76s to 0.86s locally). - Share the busy-timeout constant: export `DefaultSQLiteBusyTimeout` (20s, the production default) and reference it from the test engine instead of the hardcoded `5000`. Observed flake: https://github.com/go-gitea/gitea/actions/runs/26394082930/job/77690496092 --- This PR was written with the help of Claude Opus 4.7 --------- Co-authored-by: Claude (Opus 4.7) <noreply@anthropic.com>

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label May 6, 2026

github-actions Bot added the docs-update-needed The document needs to be updated synchronously label May 6, 2026

wxiaoguang mentioned this pull request May 6, 2026

Switch to modernc/sqlite #37105

Closed

wxiaoguang force-pushed the refactor-sqlite branch from 0953b79 to dbc1a35 Compare May 6, 2026 06:43

wxiaoguang mentioned this pull request May 6, 2026

Benchmark SQLite drivers #37499

Closed

wxiaoguang force-pushed the refactor-sqlite branch 2 times, most recently from 1eaec86 to c6db8cd Compare May 6, 2026 06:59

wxiaoguang marked this pull request as draft May 6, 2026 07:22

fix

72ffe37

wxiaoguang force-pushed the refactor-sqlite branch from c6db8cd to 72ffe37 Compare May 6, 2026 07:31

wxiaoguang marked this pull request as ready for review May 6, 2026 07:43

fix busy timeout

3dc8651

wxiaoguang force-pushed the refactor-sqlite branch from 32bdc8c to 3dc8651 Compare May 6, 2026 08:05

silverwind requested a review from Copilot May 6, 2026 09:04

fix comment & test

1aa4330

Copilot started reviewing on behalf of silverwind May 6, 2026 09:05 View session

Copilot AI reviewed May 6, 2026

View reviewed changes

Comment thread modules/setting/database.go

Comment thread models/db/conn.go Outdated

Comment thread contrib/ide/vscode/settings.json Outdated

Comment thread .github/workflows/pull-db-tests.yml

Comment thread models/db/driver_sqlite_modernc.go Outdated

Comment thread routers/install/install.go

fine tune

df4b167

wxiaoguang requested a review from Copilot May 6, 2026 09:18

Copilot started reviewing on behalf of wxiaoguang May 6, 2026 09:19 View session

Copilot AI reviewed May 6, 2026

View reviewed changes

Comment thread models/issues/issue_test.go

Comment thread tests/integration/api_issue_test.go

Comment thread routers/install/install.go

Comment thread models/db/conn.go

wxiaoguang added 3 commits May 6, 2026 17:25

avoid magic string "sqlite3"

f5377af

fix comment typo

e0b5c62

Merge branch 'main' into refactor-sqlite

7afce4c

wxiaoguang added the type/refactoring Existing code has been cleaned up. There should be no new functionality. label May 6, 2026

wxiaoguang added this to the 1.27.0 milestone May 6, 2026

wxiaoguang requested a review from TheFox0x7 May 6, 2026 10:22

lunny approved these changes May 6, 2026

View reviewed changes

GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels May 6, 2026

TheFox0x7 approved these changes May 6, 2026

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels May 6, 2026

wxiaoguang enabled auto-merge (squash) May 6, 2026 18:09

wxiaoguang disabled auto-merge May 6, 2026 18:11

wxiaoguang enabled auto-merge (squash) May 6, 2026 18:12

silverwind reviewed May 6, 2026

View reviewed changes

Comment thread .github/workflows/pull-db-tests.yml

Merge branch 'main' into refactor-sqlite

5cdc6f9

wxiaoguang merged commit a39af1a into go-gitea:main May 6, 2026
23 checks passed

wxiaoguang deleted the refactor-sqlite branch May 6, 2026 18:59

lunny mentioned this pull request May 7, 2026

Build with CGO_ENABLED=0 #24658

Open

lunny mentioned this pull request May 8, 2026

Both stable and edge snaps are broken in different ways #37593

Closed

silverwind mentioned this pull request May 10, 2026

fix(build): fix snap release #37602

Closed

silverwind mentioned this pull request May 18, 2026

fix: default sqlite journal mode to WAL #37758

Closed

silverwind mentioned this pull request May 25, 2026

test: fix flaky TestResourceIndex and reduce its runtime #37847

Merged

Uh oh!

Conversation

wxiaoguang commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lunny commented May 6, 2026

Uh oh!

lunny commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wxiaoguang commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TheFox0x7 commented May 6, 2026

Uh oh!

wxiaoguang commented May 6, 2026

Uh oh!

silverwind commented May 6, 2026

Uh oh!

wxiaoguang commented May 6, 2026

Uh oh!

Uh oh!

Uh oh!

wxiaoguang commented May 7, 2026

Uh oh!

silverwind commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wxiaoguang commented May 6, 2026 •

edited

Loading

lunny commented May 6, 2026 •

edited

Loading

wxiaoguang commented May 6, 2026 •

edited

Loading

silverwind commented May 7, 2026 •

edited

Loading