Beyond coding. We forge.
Find a file
Chl 544cbc6f01 Optimization of labels handling in issue_search (#4228)
This PR optimizes the SQL query and de-duplicate the labels' ids when generating the query string, on the issue page.

<hr/>

### Background

Some time ago, BingBot and some other crawlers have been putting my instance on its knees with requests containing a lot of label ids, like this one :

```
[07/Aug/2023:11:28:37 +0200] "GET /Dolibarr/sendrecurringinvoicebymail/issues?q=&type=all&sort=&state=closed&labels=1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c2%2c10%2c2%2c1%2c1%2c10%2c10%2c7%2c6%2c10%2c10%2c3%2c2%2c1%2c5%2c10%2c1%2c6%2c2%2c7%2c3%2c7%2c6%2c10%2c1%2c10%2c1%2c1%2c7%2c7%2c1%2c1%2c1%2c1%2c10%2c10%2c1%2c2%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c1%2c2%2c1%2c12%2c6%2c6%2c10&milestone=0&project=-1&poster=0 HTTP/1.1" 499 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36"
```

Since each of the label ids implies a join, it grows exponentially expensive for the database engine (at least on PostgreSQL but SQLite suffers a little too).

Thus, this PR proposes two enhancements:

* rewrite the database query to use only one squashed condition,
* deduplicate the label ids when generating the URL.

### Performance comparison

Here are some timings on Postgresql-backed, Forgejo 7.0.4 instances :
```sh
$ time curl -s -o /dev/null "http://localhost:3000/toto/tata/issues?q=&type=all&sort=&labels=19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25&state=open&milestone=0&project=0&assignee=0&poster=0"

real    0m10,491s
user    0m0,017s
sys     0m0,008s
```
...and with the patch:
```sh
$ time curl -s -o /dev/null "http://localhost:3000/toto/tata/issues?q=&type=all&sort=&labels=19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25%2c19%2c25&state=open&milestone=0&project=0&assignee=0&poster=0"

real    0m0,094s
user    0m0,012s
sys     0m0,013s
```

### Annex

This issue was originally proposed to [Gitea](https://github.com/go-gitea/gitea/pull/26460) but didn't get much attention, and I switched to Forgejo in the meantime :)

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4228
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Chl <chl@xlii.si>
Co-committed-by: Chl <chl@xlii.si>
2024-06-28 05:11:57 +00:00
.devcontainer Update ghcr.io/devcontainers/features/git-lfs Docker tag to v1.2.1 2024-06-20 00:01:42 +00:00
.forgejo Merge pull request 'test(js): make test-frontend-coverage displays vitest coverage' (#4224) from earl-warren/forgejo:wip-frontend-coverage into forgejo 2024-06-24 08:41:59 +00:00
.gitea [CHORE] Remove Microsoft SQL Server Support 2024-04-05 23:37:36 +02:00
assets Update module code.forgejo.org/f3/gof3/v3 to v3.4.0 (#4196) 2024-06-21 13:51:59 +00:00
build s/Gitea/Forgejo in various log messages and comments 2024-04-21 21:26:15 +05:00
cmd test(dump): don't depend on directory listing order 2024-06-17 17:54:59 +02:00
contrib Add codespell support and fix a good number of typos with its help (#3270) 2024-05-09 13:49:37 +00:00
custom/conf feat(F3): CLI: f3 mirror to convert to/from Forgejo 2024-06-14 12:52:12 +02:00
docker fix(Dockerfile.rootless): revert to default path for app.ini 2024-04-26 21:30:10 +02:00
models Optimization of labels handling in issue_search (#4228) 2024-06-28 05:11:57 +00:00
modules Fix markdown math brackets render problem (#31420) 2024-06-23 12:40:06 +02:00
options i18n: Translations update from Weblate (#4168) 2024-06-26 11:07:32 +00:00
public [FEAT] sourcehut webhooks 2024-04-05 10:20:40 +02:00
release-notes Merge pull request 'Clarify author label in tooltip' (#4201) from 0ko/forgejo:ui-author-tooltip into forgejo 2024-06-26 08:38:55 +00:00
releases/images [DOCS] RELEASE-NOTES.md 2024-02-05 14:44:32 +01:00
routers Add cache test for admins (#31265) 2024-06-23 11:27:03 +02:00
services [Fix] Account Linking UpdateMigrationsByType (#31428) 2024-06-23 12:40:53 +02:00
templates Merge pull request 'Clarify author label in tooltip' (#4201) from 0ko/forgejo:ui-author-tooltip into forgejo 2024-06-26 08:38:55 +00:00
tests Merge pull request 'Clarify author label in tooltip' (#4201) from 0ko/forgejo:ui-author-tooltip into forgejo 2024-06-26 08:38:55 +00:00
tools Add lint-go-gopls (#30729) 2024-06-09 11:13:39 +02:00
web_src test: add test coverage for web_src/js/utils.js (#4235) 2024-06-27 15:46:46 +00:00
.air.toml Reduce air verbosity (#31417) 2024-06-23 12:30:09 +02:00
.deadcode-out Refactor markup code (#31399) 2024-06-23 11:53:36 +02:00
.dockerignore Add /public/assets/img/webpack to ignore files again (#30451) 2024-04-15 20:01:36 +02:00
.editorconfig fixed indentation style in editorconfig for go.mod 2024-05-14 00:24:18 +02:00
.eslintrc.yaml add built js files to eslint ignore (#30737) 2024-05-05 08:22:50 +01:00
.gitattributes Add interface{} to any replacement to make fmt, exclude *.pb.go (#30461) 2024-04-15 20:01:36 +02:00
.gitignore Add /public/assets/img/webpack to ignore files again (#30451) 2024-04-15 20:01:36 +02:00
.gitmodules cleanup(tests): remove manual testing submodule 2024-04-21 10:13:51 +02:00
.gitpod.yml Remove sqlite-viewer and using database client (#31223) 2024-06-09 11:13:39 +02:00
.golangci.yml Enable unparam linter (#31277) 2024-06-16 13:42:58 +02:00
.ignore Add /options/license and /options/gitignore to .ignore (#30219) 2024-04-07 15:40:31 +02:00
.markdownlint.yaml Update JS dependencies (#28537) 2023-12-30 05:29:03 +00:00
.npmrc Upgrade to npm lockfile v3 and explicitely set it (#23561) 2023-03-18 19:38:10 +01:00
.spectral.yaml
.yamllint.yaml fully replace drone with actions (#27556) 2023-10-11 06:39:32 +00:00
BSDmakefile Fix build errors on BSD (in BSDMakefile) (#27594) 2023-10-13 15:38:27 +00:00
build.go User/Org Feed render description as per web (#23887) 2023-04-04 04:39:47 +01:00
CODEOWNERS I feel responsible … (Codeowners) 2024-04-27 02:22:05 +02:00
CONTRIBUTING.md docs: contributing: avoid information duplication (#3454) 2024-04-25 19:10:43 +00:00
DCO Remove address from DCO (#22595) 2023-01-24 18:52:38 +00:00
Dockerfile Fix deprecated Dockerfile ENV format (#31450) 2024-06-23 13:20:40 +02:00
Dockerfile.rootless Fix deprecated Dockerfile ENV format (#31450) 2024-06-23 13:20:40 +02:00
flake.lock update nix flake and add gofumpt (#31320) 2024-06-16 13:42:58 +02:00
flake.nix update nix flake and add gofumpt (#31320) 2024-06-16 13:42:58 +02:00
go.mod Merge pull request 'Update module github.com/yuin/goldmark to v1.7.4' (#4240) from renovate/github.com-yuin-goldmark-1.x into forgejo 2024-06-26 07:51:29 +00:00
go.sum Merge pull request 'Update module github.com/yuin/goldmark to v1.7.4' (#4240) from renovate/github.com-yuin-goldmark-1.x into forgejo 2024-06-26 07:51:29 +00:00
LICENSE [DOCS] LICENSE: add Forgejo Authors 2024-02-05 14:44:32 +01:00
main.go [RELEASE] decouple the release name from the version number 2024-02-17 15:27:35 +01:00
Makefile chore(renovate): sync renovate versions 2024-06-25 08:27:54 +02:00
package-lock.json Update dependency minimatch to v9.0.5 2024-06-26 06:20:05 +00:00
package.json Update dependency minimatch to v9.0.5 2024-06-26 06:20:05 +00:00
playwright.config.js Enforce trailing comma in JS on multiline (#30002) 2024-03-26 19:04:27 +01:00
poetry.lock Update dependency codespell to v2.3.0 2024-05-25 02:07:08 +00:00
poetry.toml Clean up pyproject.toml and package.json, fix poetry options (#25327) 2023-06-18 18:13:08 +00:00
pyproject.toml Enable poetry non-package mode (#31282) 2024-06-09 16:04:57 +02:00
README.md [skip ci] IGNORE (#4106) 2024-06-11 16:06:50 +00:00
RELEASE-NOTES.md [skip ci] docs(release-notes): 1.21.11-2 2024-06-13 14:27:14 +02:00
renovate.json chore(renovate): add shared go preset 2024-06-26 09:24:17 +02:00
stylelint.config.js Merge pull request 'Port "Enable declaration-block-no-redundant-longhand-properties (#30950)' (#3769) from beowulf/gitea-port-pull-30950 into forgejo 2024-05-14 22:23:54 +00:00
tailwind.config.js [FEAT] folding results for repo search (#4134) 2024-06-15 20:16:18 +00:00
vitest.config.js Switch to happy-dom for testing (#29948) 2024-03-26 19:04:26 +01:00
webpack.config.js replace jquery-minicolors with coloris (#30055) 2024-04-07 11:02:33 +02:00

Welcome to Forgejo

Hi there! Tired of big platforms playing monopoly? Providing Git hosting for your project, friends, company or community? Forgejo (/for'd͡ʒe.jo/ inspired by forĝejo the Esperanto word for forge) has you covered with its intuitive interface, light and easy hosting and a lot of builtin functionality.

Forgejo was created in 2022 because we think that the project should be owned by an independent community. If you second that, then Forgejo is for you! Our promise: Independent Free/Libre Software forever!

What does Forgejo offer?

If you like any of the following, Forgejo is literally meant for you:

  • Lightweight: Forgejo can easily be hosted on nearly every machine. Running on a Raspberry? Small cloud instance? No problem!
  • Project management: Besides Git hosting, Forgejo offers issues, pull requests, wikis, kanban boards and much more to coordinate with your team.
  • Publishing: Have something to share? Use releases to host your software for download, or use the package registry to publish it for docker, npm and many other package managers.
  • Customizable: Want to change your look? Change some settings? There are many config switches to make Forgejo work exactly like you want.
  • Powerful: Organizations & team permissions, CI integration, Code Search, LDAP, OAuth and much more. If you have advanced needs, Forgejo has you covered.
  • Privacy: From update checker to default settings: Forgejo is built to be privacy first for you and your crew.
  • Federation: (WIP) We are actively working to connect software forges with each other through ActivityPub, and create a collaborative network of personal instances.

Learn more

Dive into the documentation, subscribe to releases and blog post on our website, find us on the Fediverse or hop into our Matrix room if you have any questions or want to get involved.

Get involved

If you are interested in making Forgejo better, either by reporting a bug or by changing the governance, please take a look at the contribution guide.