Commit graph

515 commits

Author SHA1 Message Date
Gusted
d623126103 Merge pull request '[PORT] Refactor the usage of batch catfile (gitea#31754)' (#5122) from gusted/forgejo-port-gt-31754 into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/5122
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
2024-08-26 16:04:47 +00:00
Gusted
1a68d14cf8
[FIX] Don't allow SSH authentication without ssh executable
- Follow up of #4819
- When no `ssh` executable is present, disable the UI and backend bits
that allow the creation of push mirrors that use SSH authentication. As
this feature requires the usage of the `ssh` binary.
- Integration test added.
2024-08-26 08:58:35 +02:00
Gusted
24bbf051c3
[TESTS] Add test for CatFileBatch(Check) 2024-08-26 04:48:52 +02:00
Lunny Xiao
0ca13c5eae
[PORT] Refactor the usage of batch catfile (gitea#31754)
When opening a repository, it will call `ensureValidRepository` and also
`CatFileBatch`. But sometimes these will not be used until repository
closed. So it's a waste of CPU to invoke 3 times git command for every
open repository.

This PR removed all of these from `OpenRepository` but only kept
checking whether the folder exists. When a batch is necessary, the
necessary functions will be invoked.

---
Conflict resolution: Because of the removal of go-git in (#4941)
`_nogogit.go` files were either renamed or merged into the 'common'
file. Git does handle the renames correctly, but for those that were
merged has to be manually copied pasted over. The patch looks the same,
201 additions 90 deletions as the original patch.

(cherry picked from commit c03baab678ba5b2e9d974aea147e660417f5d3f7)
2024-08-26 03:48:51 +02:00
Philip Peterson
03508b33a8
[FEAT] Allow pushmirror to use publickey authentication
- Continuation of https://github.com/go-gitea/gitea/pull/18835 (by
@Gusted, so it's fine to change copyright holder to Forgejo).
- Add the option to use SSH for push mirrors, this would allow for the
deploy keys feature to be used and not require tokens to be used which
cannot be limited to a specific repository. The private key is stored
encrypted (via the `keying` module) on the database and NEVER given to
the user, to avoid accidental exposure and misuse.
- CAVEAT: This does require the `ssh` binary to be present, which may
not be available in containerized environments, this could be solved by
adding a SSH client into forgejo itself and use the forgejo binary as
SSH command, but should be done in another PR.
- CAVEAT: Mirroring of LFS content is not supported, this would require
the previous stated problem to be solved due to LFS authentication (an
attempt was made at forgejo/forgejo#2544).
- Integration test added.
- Resolves #4416
2024-08-22 17:05:07 +02:00
Earl Warren
0c70e11df8 Merge pull request 'git-grep: refactor defaults' (#4964) from yoctozepto/git-grep-refactor-defaults into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4964
Reviewed-by: Shiny Nematoda <snematoda@noreply.codeberg.org>
2024-08-19 13:12:37 +00:00
Radosław Piliszek
f784260633 git-grep: refactor defaults
One method to set them all... or something like that.

The defaults for git-grep options were scattered over the run
function body. This change refactors them into a separate method.
The application of defaults is checked implicitly by existing
tests and linters, and the new approach makes it very easy
to inspect the desired defaults are set.
2024-08-19 14:28:01 +02:00
Gusted
fe18428806
Fix linting issues
Some checks failed
renovate / renovate (push) Has been cancelled
2024-08-18 16:25:13 +02:00
Earl Warren
fbb2252314 Merge pull request 'git-grep: fix for initial dashes in expressions' (#4967) from yoctozepto/git-grep-fix-words into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4967
Reviewed-by: Shiny Nematoda <snematoda@noreply.codeberg.org>
2024-08-17 07:07:40 +00:00
Otto
d26ad8dbe5 Merge pull request 'git-grep: set timeout to 2s by default and allow configuring it' (#4966) from yoctozepto/git-grep-timeout into forgejo
Some checks are pending
/ release (push) Waiting to run
testing / backend-checks (push) Waiting to run
testing / frontend-checks (push) Waiting to run
testing / test-unit (push) Blocked by required conditions
testing / test-remote-cacher (map[image:docker.io/bitnami/redis:7.2 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:docker.io/bitnami/valkey:7.2 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:ghcr.io/microsoft/garnet-alpine:1.0.14 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:registry.redict.io/redict:7.3.0-scratch port:6379]) (push) Blocked by required conditions
testing / test-mysql (push) Blocked by required conditions
testing / test-pgsql (push) Blocked by required conditions
testing / test-sqlite (push) Blocked by required conditions
testing / security-check (push) Blocked by required conditions
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4966
Reviewed-by: Otto <otto@codeberg.org>
2024-08-15 12:15:16 +00:00
Radosław Piliszek
824dd6bc5d git-grep: set timeout to 2s by default and allow configuring it
We need to shorten the timeout to bound effectively for
computation size. This protects against "too big" repos.
This also protects to some extent against too long lines
if kept to very low values (basically so that grep cannot run out
of memory beforehand).

Docs-PR: forgejo/docs#812
2024-08-14 07:55:05 +02:00
Jason Song
83565de2c0
Fix IsObjectExist with gogit (#31790) (tests only)
Fix #31271.

When gogit is enabled, `IsObjectExist` calls
`repo.gogitRepo.ResolveRevision`, which is not correct. It's for
checking references not objects, it could work with commit hash since
it's both a valid reference and a commit object, but it doesn't work
with blob objects.

So it causes #31271 because it reports that all blob objects do not
exist.

(cherry picked from commit f4d3120f9d1de6a260a5e625b3ffa6b35a069e9b)

Conflicts:
  trivial resolution because go-git support was dropped https://codeberg.org/forgejo/forgejo/pulls/4941
2024-08-13 09:05:43 +02:00
Radosław Piliszek
f4d86b4ab0 git-grep: fix for initial dashes in expressions
There is no reason to reject initial dashes in git-grep
expressions... other than the code not supporting it previously.
A new method is introduced to relax the security checks.
2024-08-12 22:12:47 +02:00
Gusted
a21128a734
[CHORE] Drop go-git support
See https://codeberg.org/forgejo/discussions/issues/164 for the
rationale and discussion of this change.

Everything related to the `go-git` dependency is dropped (Only a single
instance is left in a test file to test for an XSS, it requires crafting
an commit that Git itself refuses to craft). `_gogit` files have
been removed entirely, `go:build: !gogit` is removed, `XXX_nogogit.go` files
either have been renamed or had their code being merged into the
`XXX.go` file.
2024-08-12 19:11:09 +02:00
Otto
4132b18e59 Merge pull request 'git-grep: ensure bounded default for MatchesPerFile' (#4928) from yoctozepto/forgejo:git-grep-limit-matches-per-file into forgejo
Some checks are pending
/ release (push) Waiting to run
testing / test-remote-cacher (map[image:redis:7.2 port:6379]) (push) Blocked by required conditions
testing / backend-checks (push) Waiting to run
testing / frontend-checks (push) Waiting to run
testing / test-unit (push) Blocked by required conditions
testing / test-remote-cacher (map[image:docker.io/valkey/valkey:7.2.5-alpine3.19 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:ghcr.io/microsoft/garnet-alpine:1.0.14 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:registry.redict.io/redict:7.3.0-scratch port:6379]) (push) Blocked by required conditions
testing / test-mysql (push) Blocked by required conditions
testing / test-pgsql (push) Blocked by required conditions
testing / test-sqlite (push) Blocked by required conditions
testing / security-check (push) Blocked by required conditions
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4928
Reviewed-by: Shiny Nematoda <snematoda@noreply.codeberg.org>
Reviewed-by: Otto <otto@codeberg.org>
2024-08-12 14:37:43 +00:00
Radosław Piliszek
f4a7bf6d2a git-grep: skip binary files
It is a waste of resources to scan them looking for matches
because they are never returned back - they appear as empty
lines in the current format.

Notably, even if they were returned, it is unlikely that matching
in binary files makes sense when the goal is "code search".
2024-08-11 19:10:23 +02:00
Radosław Piliszek
2fbb51ceb2 git-grep: ensure bounded default for MatchesPerFile
Analogously to how it happens for MaxResultLimit.

The default of 20 is inspired by a well-known, commercial code
hosting platform.

Unbounded limits are risky because they expose Forgejo to a class
of DoS attacks where queries are crafted to take advantage of
missing bounds.
2024-08-11 14:59:46 +02:00
Radosław Piliszek
7dd7cc7ebc git-grep: update comment
It was outdated and missing detail.
2024-08-10 16:41:12 +02:00
Shiny Nematoda
06d2e90fa4 feat: highlighted code search results (#4749)
closes #4534

<details>
<summary>Screenshots</summary>

![](https://codeberg.org/attachments/0ab8a7b0-6485-46dc-a730-c016abb1f287)
</details>

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4749
Reviewed-by: 0ko <0ko@noreply.codeberg.org>
Co-authored-by: Shiny Nematoda <snematoda.751k2@aleeas.com>
Co-committed-by: Shiny Nematoda <snematoda.751k2@aleeas.com>
2024-08-06 05:57:25 +00:00
TheFox0x7
4de909747b Add testifylint to lint checks (#4535)
Some checks are pending
/ release (push) Waiting to run
testing / test-remote-cacher (map[image:redis:7.2 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:registry.redict.io/redict:7.3.0-scratch port:6379]) (push) Blocked by required conditions
testing / test-mysql (push) Blocked by required conditions
testing / test-pgsql (push) Blocked by required conditions
testing / backend-checks (push) Waiting to run
testing / frontend-checks (push) Waiting to run
testing / test-unit (push) Blocked by required conditions
testing / test-remote-cacher (map[image:docker.io/valkey/valkey:7.2.5-alpine3.19 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:ghcr.io/microsoft/garnet-alpine:1.0.14 port:6379]) (push) Blocked by required conditions
testing / test-sqlite (push) Blocked by required conditions
testing / security-check (push) Blocked by required conditions
go-require lint is ignored for now

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4535
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: TheFox0x7 <thefox0x7@gmail.com>
Co-committed-by: TheFox0x7 <thefox0x7@gmail.com>
2024-07-30 19:41:10 +00:00
Brecht Van Lommel
33f9fb8150
Fix slow patch checking with commits that add or remove many files (#31548)
Running git update-index for every individual file is slow, so add and
remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR
creation and patch checking for all open PRs to be slow, or time out
entirely. For example, a commit that removes 1383 files was measured to
take more than 60 seconds and timed out. With this change checking took
about a second.

This is related to #27967, though this will not help with commits that
change many lines in few files.

(cherry picked from commit b88e5fc72d99e9d4a0aa9c13f70e0a9e967fe057)
2024-07-07 07:33:01 +02:00
Twenty Panda
5561e80b04 fix(hook): ignore unknown push options instead of failing
If a repository has

git config --add push.pushOption submit=".sourcehut/*.yml"

it failed when pushed because of the unknown submit push
option. It will be ignored instead.

Filtering out the push options is done in an earlier stage, when the
hook command runs, before it submits the options map to the private
endpoint.

* move all the push options logic to modules/git/pushoptions
* add 100% test coverage for modules/git/pushoptions

Test coverage for the code paths from which code was moved to the
modules/git/pushoptions package:

* cmd/hook.go:runHookPreReceive
* routers/private/hook_pre_receive.go:validatePushOptions
  tests/integration/git_push_test.go:TestOptionsGitPush runs through
  both. The test verifying the option is rejected was removed and, if
  added again, will fail because the option is now ignored instead of
  being rejected.

* cmd/hook.go:runHookProcReceive
* services/agit/agit.go:ProcReceive
  tests/integration/git_test.go: doCreateAgitFlowPull runs through
  both. It uses variations of AGit related push options.

* cmd/hook.go:runHookPostReceive
* routers/private/hook_post_receive.go:HookPostReceive
  tests/integration/git_test.go:doPushCreate called by TestGit/HTTP/sha1/PushCreate
  runs through both.
  Note that although it provides coverage for this code path it does not use push options.

Fixes: https://codeberg.org/forgejo/forgejo/issues/3651
2024-07-02 21:39:01 +02:00
Mai-Lapyst
9b225b56a9
Fixes git references wrongly transmitted to the action run 2024-06-30 06:17:40 +02:00
silverwind
d8bc0495de
Enable unparam linter (#31277)
Enable [unparam](https://github.com/mvdan/unparam) linter.

Often I could not tell the intention why param is unused, so I put
`//nolint` for those cases like webhook request creation functions never
using `ctx`.

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
(cherry picked from commit fc2d75f86d77b022ece848acf2581c14ef21d43b)

Conflicts:
	modules/setting/config_env.go
	modules/storage/azureblob.go
	services/webhook/dingtalk.go
	services/webhook/discord.go
	services/webhook/feishu.go
	services/webhook/matrix.go
	services/webhook/msteams.go
	services/webhook/packagist.go
	services/webhook/slack.go
	services/webhook/telegram.go
	services/webhook/wechatwork.go

	run make lint-go and fix Forgejo specific warnings
2024-06-16 13:42:58 +02:00
wxiaoguang
5612cf32e5
Refactor sha1 and time-limited code (#31023)
Remove "EncodeSha1", it shouldn't be used as a general purpose hasher
(just like we have removed "EncodeMD5" in #28622)

Rewrite the "time-limited code" related code and write better tests, the
old code doesn't seem quite right.

(cherry picked from commit fb1ad920b769799aa1287441289d15477d9878c5)

Conflicts:
	modules/git/utils_test.go
	trivial context conflict because sha256 testing in Forgejo has diverged
2024-05-26 18:43:30 +02:00
Earl Warren
c7711dac6d
chore: fix typos identified by mispell 2024-05-25 09:43:50 +02:00
Shiny Nematoda
ec4f5495ba feat: wiki search using git-grep
+ add release note
2024-05-20 13:48:50 +00:00
oliverpool
5e73c67d67 fix: hook post-receive for sha256 repos 2024-05-15 16:43:16 +02:00
Shiny Nematoda
b6ca8abcfd [FEAT] support searching non default branches/tags when using git-grep (#3654)
resolves https://codeberg.org/forgejo/forgejo/pulls/3639#issuecomment-1806676 and https://codeberg.org/forgejo/forgejo/pulls/3513#issuecomment-1794990

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3654
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Shiny Nematoda <snematoda.751k2@aleeas.com>
Co-committed-by: Shiny Nematoda <snematoda.751k2@aleeas.com>
2024-05-14 15:41:03 +00:00
wxiaoguang
8cb8547532
Make "sync branch" also sync object format and add tests (#30878)
(cherry picked from commit 9c08637eae8c3a44d15e62d85144e07ae9dabbec)
2024-05-12 20:03:10 +02:00
Yaroslav Halchenko
2b2fd2728c Add codespell support and fix a good number of typos with its help (#3270)
More about codespell: https://github.com/codespell-project/codespell .

I personally introduced it to dozens if not hundreds of projects already and so far only positive feedback.

```
❯ grep lint-spell Makefile
	@echo " - lint-spell                       lint spelling"
	@echo " - lint-spell-fix                   lint spelling and fix issues"
lint: lint-frontend lint-backend lint-spell
lint-fix: lint-frontend-fix lint-backend-fix lint-spell-fix
.PHONY: lint-spell
lint-spell: lint-codespell
.PHONY: lint-spell-fix
lint-spell-fix: lint-codespell-fix
❯ git grep lint- -- .forgejo/
.forgejo/workflows/testing.yml:      - run: make --always-make -j$(nproc) lint-backend checks-backend # ensure the "go-licenses" make target runs
.forgejo/workflows/testing.yml:      - run: make lint-frontend
```
so how would you like me to invoke `lint-codespell` on CI? (without that would be IMHO very suboptimal and let typos sneak in)

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3270
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
Co-committed-by: Yaroslav Halchenko <debian@onerussian.com>
2024-05-09 13:49:37 +00:00
Chongyi Zheng
c504461b66
Resolve lint for unused parameter and unnecessary type arguments (#30750)
Resolve all cases for `unused parameter` and `unnecessary type
arguments`

Related: #30729

---------

Co-authored-by: Giteabot <teabot@gitea.io>
(cherry picked from commit e80466f7349164ce4cf3c07bdac30d736d20f035)

Conflicts:
	modules/markup/markdown/transform_codespan.go
	modules/setting/incoming_email.go
	routers/api/v1/admin/user_badge.go
	routers/private/hook_pre_receive.go
	tests/integration/repo_search_test.go
	resolved by discarding the change, this is linting only and
	for the sake of avoiding future conflicts
2024-05-05 08:38:16 +01:00
silverwind
12b199c5e5
Enable more revive linter rules (#30608)
Noteable additions:

- `redefines-builtin-id` forbid variable names that shadow go builtins
- `empty-lines` remove unnecessary empty lines that `gofumpt` does not
remove for some reason
- `superfluous-else` eliminate more superfluous `else` branches

Rules are also sorted alphabetically and I cleaned up various parts of
`.golangci.yml`.

(cherry picked from commit 74f0c84fa4245a20ce6fb87dac1faf2aeeded2a2)

Conflicts:
	.golangci.yml
	apply the linter recommendations to Forgejo code as well
2024-04-28 15:39:00 +02:00
oliverpool
20350846fc Merge pull request 'fix: git.ComputeHash did not write the content' (#3466) from oliverpool/forgejo:fix_compute_hash into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3466
Reviewed-by: Otto <otto@codeberg.org>
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
2024-04-26 10:15:23 +00:00
Earl Warren
c31ae1a651 fix(lfs): gogit /settings/lfs/find 500 error (#3472)
Refs: https://codeberg.org/forgejo/forgejo/pulls/3448
Refs: https://codeberg.org/forgejo/forgejo/issues/3438
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3472
Reviewed-by: oliverpool <oliverpool@noreply.codeberg.org>
Co-authored-by: Earl Warren <contact@earl-warren.org>
Co-committed-by: Earl Warren <contact@earl-warren.org>
2024-04-26 09:22:09 +00:00
oliverpool
5247fd50db fix: git.ComputeHash did not write the content 2024-04-26 10:16:59 +02:00
oliverpool
3dfa5ba43a test: LFS gc should not delete all metadata objects
and ComputeBlobHash should depend on the blob content (not only the
length)
2024-04-26 10:16:59 +02:00
wxiaoguang
27434acef6
Fix commit file status parser (#30602)
Try to fix  #30492

(cherry picked from commit 53cf46cae7475befa2dde554bbd9147e436072b9)
2024-04-21 16:28:16 +02:00
Gusted
0c42e3c755 Merge pull request '[CHORE] Remove println' (#3185) from gusted/forgejo-println into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3185
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
2024-04-12 12:35:38 +00:00
Gusted
706846b628
[CHORE] Remove println
- It's not recommended to use it per the documentation: https://pkg.go.dev/builtin#println.
2024-04-12 13:36:31 +02:00
Antonin Delpeuch
592c20c7d4 Fix spelling mistakes caught by golangci-lint 2024-04-11 17:16:26 +02:00
Gusted
668709a33f
[BUG] Handle bigger files in git grep
- The parser of `git grep`'s output uses `bufio.Scanner`, which is a good
choice overall, however it does have a limit that's usually not noticed,
it will not read more than `64 * 1024` bytes at once which can be hit in
practical scenarios.
- Use `bufio.Reader` instead which doesn't have this limitation, but is
a bit harder to work with as it's a more lower level primitive.
- Adds unit test.
- Resolves https://codeberg.org/forgejo/forgejo/issues/3149
2024-04-11 13:47:50 +02:00
Earl Warren
26fc7c3461 Merge pull request '[gitea] week 15 cherry pick' (#3091) from algernon/forgejo:wcp/week-15 into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3091
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
2024-04-09 05:31:44 +00:00
JakobDev
613e5387c5 Count downloads for tag archives 2024-04-08 12:51:27 +02:00
KN4CK3R
6925c0eee4
Include encoding in signature payload (#30174)
Fixes #30119

Include the encoding in the signature payload.

before

![grafik](https://github.com/go-gitea/gitea/assets/1666336/01ab94a3-8af5-4d6f-be73-a10b65a15421)

after

![grafik](https://github.com/go-gitea/gitea/assets/1666336/3a37d438-c70d-4d69-b178-d170e74aa683)

(cherry picked from commit b6a3cd4b8dc20ba48d0044a972f6ff0f0de6e49e)
2024-04-07 15:40:31 +02:00
Shiny Nematoda
baac15f316 [FEAT] Support Include/Exclude Filters for Grep (#3058)
fixes `TestRepoSearch` failing occasionally

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3058
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Shiny Nematoda <snematoda.751k2@aleeas.com>
Co-committed-by: Shiny Nematoda <snematoda.751k2@aleeas.com>
2024-04-06 13:25:39 +00:00
oliverpool
cef39b2940 [TESTS] fix flack git check-attr flacky subtest 2024-04-02 11:56:48 +02:00
oliverpool
0bb7758cb0 git attribute: test proper cancellation and unify nul-byte reader 2024-03-31 08:44:09 +02:00
Gusted
1ebf4abddc
[BUG] Use correct format
- `%w` is to wrap errors, but can only be used by `fmt.Errorf`. Instead
use `%v` to display the error.
- Regression of #2763

Before:
[E] failed to run attr-check. Error: %!w(*exec.ExitError=&{0xc006568e28 []})
Stderr: fatal: this operation must be run in a work tree

After:
[E] failed to run attr-check. Error: exit status 128
Stderr: fatal: this operation must be run in a work tree
2024-03-28 22:27:53 +01:00
Shiny Nematoda
db7e6948a1 replace IfZero with cmp.Or 2024-03-28 11:58:26 +00:00