Commit graph

182 commits

Author SHA1 Message Date
Gergely Nagy
67fa52dedb
feat(quota): Quota enforcement
The previous commit laid out the foundation of the quota engine, this
one builds on top of it, and implements the actual enforcement.

Enforcement happens at the route decoration level, whenever possible. In
case of the API, when over quota, a 413 error is returned, with an
appropriate JSON payload. In case of web routes, a 413 HTML page is
rendered with similar information.

This implementation is for a **soft quota**: quota usage is checked
before an operation is to be performed, and the operation is *only*
denied if the user is already over quota. This makes it possible to go
over quota, but has the significant advantage of being practically
implementable within the current Forgejo architecture.

The goal of enforcement is to deny actions that can make the user go
over quota, and allow the rest. As such, deleting things should - in
almost all cases - be possible. A prime exemption is deleting files via
the web ui: that creates a new commit, which in turn increases repo
size, thus, is denied if the user is over quota.

Limitations
-----------

Because we generally work at a route decorator level, and rarely
look *into* the operation itself, `size:repos:public` and
`size:repos:private` are not enforced at this level, the engine enforces
against `size:repos:all`. This will be improved in the future.

AGit does not play very well with this system, because AGit PRs count
toward the repo they're opened against, while in the GitHub-style fork +
pull model, it counts against the fork. This too, can be improved in the
future.

There's very little done on the UI side to guard against going over
quota. What this patch implements, is enforcement, not prevention. The
UI will still let you *try* operations that *will* result in a denial.

Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
2024-08-02 11:10:34 +02:00
TheFox0x7
4de909747b Add testifylint to lint checks (#4535)
Some checks are pending
/ release (push) Waiting to run
testing / test-remote-cacher (map[image:redis:7.2 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:registry.redict.io/redict:7.3.0-scratch port:6379]) (push) Blocked by required conditions
testing / test-mysql (push) Blocked by required conditions
testing / test-pgsql (push) Blocked by required conditions
testing / backend-checks (push) Waiting to run
testing / frontend-checks (push) Waiting to run
testing / test-unit (push) Blocked by required conditions
testing / test-remote-cacher (map[image:docker.io/valkey/valkey:7.2.5-alpine3.19 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:ghcr.io/microsoft/garnet-alpine:1.0.14 port:6379]) (push) Blocked by required conditions
testing / test-sqlite (push) Blocked by required conditions
testing / security-check (push) Blocked by required conditions
go-require lint is ignored for now

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4535
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: TheFox0x7 <thefox0x7@gmail.com>
Co-committed-by: TheFox0x7 <thefox0x7@gmail.com>
2024-07-30 19:41:10 +00:00
Twenty Panda
5561e80b04 fix(hook): ignore unknown push options instead of failing
If a repository has

git config --add push.pushOption submit=".sourcehut/*.yml"

it failed when pushed because of the unknown submit push
option. It will be ignored instead.

Filtering out the push options is done in an earlier stage, when the
hook command runs, before it submits the options map to the private
endpoint.

* move all the push options logic to modules/git/pushoptions
* add 100% test coverage for modules/git/pushoptions

Test coverage for the code paths from which code was moved to the
modules/git/pushoptions package:

* cmd/hook.go:runHookPreReceive
* routers/private/hook_pre_receive.go:validatePushOptions
  tests/integration/git_push_test.go:TestOptionsGitPush runs through
  both. The test verifying the option is rejected was removed and, if
  added again, will fail because the option is now ignored instead of
  being rejected.

* cmd/hook.go:runHookProcReceive
* services/agit/agit.go:ProcReceive
  tests/integration/git_test.go: doCreateAgitFlowPull runs through
  both. It uses variations of AGit related push options.

* cmd/hook.go:runHookPostReceive
* routers/private/hook_post_receive.go:HookPostReceive
  tests/integration/git_test.go:doPushCreate called by TestGit/HTTP/sha1/PushCreate
  runs through both.
  Note that although it provides coverage for this code path it does not use push options.

Fixes: https://codeberg.org/forgejo/forgejo/issues/3651
2024-07-02 21:39:01 +02:00
silverwind
d8bc0495de
Enable unparam linter (#31277)
Enable [unparam](https://github.com/mvdan/unparam) linter.

Often I could not tell the intention why param is unused, so I put
`//nolint` for those cases like webhook request creation functions never
using `ctx`.

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
(cherry picked from commit fc2d75f86d77b022ece848acf2581c14ef21d43b)

Conflicts:
	modules/setting/config_env.go
	modules/storage/azureblob.go
	services/webhook/dingtalk.go
	services/webhook/discord.go
	services/webhook/feishu.go
	services/webhook/matrix.go
	services/webhook/msteams.go
	services/webhook/packagist.go
	services/webhook/slack.go
	services/webhook/telegram.go
	services/webhook/wechatwork.go

	run make lint-go and fix Forgejo specific warnings
2024-06-16 13:42:58 +02:00
Earl Warren
09f3518069
fix(hook): repo admins are wrongly denied the right to force merge
The right to force merge is uses the wrong predicate and
applies to instance admins:

  ctx.user.IsAdmin

It must apply to repository admins and use the following predicate:

 ctx.userPerm.IsAdmin()

This regression is from the ApplyToAdmins implementation in
79b7089360.

Fixes: https://codeberg.org/forgejo/forgejo/issues/3780
2024-06-02 21:16:46 +02:00
Earl Warren
05f0007437
fix(hook): instance admins wrongly restricted by permissions checks
This exception existed for both instance admins and repo admins
before ApplyToAdmins was introduced in
79b7089360.

It should have been kept for instance admins only because they are not
subject to permission checks.
2024-06-01 23:50:57 +02:00
oliverpool
5e73c67d67 fix: hook post-receive for sha256 repos 2024-05-15 16:43:16 +02:00
Lunny Xiao
eb792d9f8a
Move database operations of merging a pull request to post receive hook and add a transaction (#30805)
Merging PR may fail because of various problems. The pull request may
have a dirty state because there is no transaction when merging a pull
request. ref
https://github.com/go-gitea/gitea/pull/25741#issuecomment-2074126393

This PR moves all database update operations to post-receive handler for
merging a pull request and having a database transaction. That means if
database operations fail, then the git merging will fail, the git client
will get a fail result.

There are already many tests for pull request merging, so we don't need
to add a new one.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit ebf0c969403d91ed80745ff5bd7dfbdb08174fc7)

Conflicts:
	modules/private/hook.go
	routers/private/hook_post_receive.go
	trivial conflicts because
	  263a716cb5 * Performance optimization for git push (#30104)
	was not cherry-picked and because of
	  998a431747 Do not update PRs based on events that happened before they existed
2024-05-12 20:03:10 +02:00
Yaroslav Halchenko
2b2fd2728c Add codespell support and fix a good number of typos with its help (#3270)
More about codespell: https://github.com/codespell-project/codespell .

I personally introduced it to dozens if not hundreds of projects already and so far only positive feedback.

```
❯ grep lint-spell Makefile
	@echo " - lint-spell                       lint spelling"
	@echo " - lint-spell-fix                   lint spelling and fix issues"
lint: lint-frontend lint-backend lint-spell
lint-fix: lint-frontend-fix lint-backend-fix lint-spell-fix
.PHONY: lint-spell
lint-spell: lint-codespell
.PHONY: lint-spell-fix
lint-spell-fix: lint-codespell-fix
❯ git grep lint- -- .forgejo/
.forgejo/workflows/testing.yml:      - run: make --always-make -j$(nproc) lint-backend checks-backend # ensure the "go-licenses" make target runs
.forgejo/workflows/testing.yml:      - run: make lint-frontend
```
so how would you like me to invoke `lint-codespell` on CI? (without that would be IMHO very suboptimal and let typos sneak in)

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3270
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
Co-committed-by: Yaroslav Halchenko <debian@onerussian.com>
2024-05-09 13:49:37 +00:00
silverwind
9792a377e4
Fix tautological conditions (#30735)
As discovered by https://github.com/go-gitea/gitea/pull/30729.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
(cherry picked from commit 610802df85933e7a190a705bc3f7800da87ce868)

Conflicts:
	tests/integration/git_test.go
	trivial conflict because of https://codeberg.org/forgejo/forgejo/pulls/2834
2024-05-05 08:59:40 +01:00
silverwind
12b199c5e5
Enable more revive linter rules (#30608)
Noteable additions:

- `redefines-builtin-id` forbid variable names that shadow go builtins
- `empty-lines` remove unnecessary empty lines that `gofumpt` does not
remove for some reason
- `superfluous-else` eliminate more superfluous `else` branches

Rules are also sorted alphabetically and I cleaned up various parts of
`.golangci.yml`.

(cherry picked from commit 74f0c84fa4245a20ce6fb87dac1faf2aeeded2a2)

Conflicts:
	.golangci.yml
	apply the linter recommendations to Forgejo code as well
2024-04-28 15:39:00 +02:00
Gergely Nagy
8eba631f8d
hooks: Harden when we accept push options that change repo settings
It is possible to change some repo settings (its visibility, and
template status) via `git push` options: `-o repo.private=true`, `-o
repo.template=true`.

Previously, there weren't sufficient permission checks on these, and
anyone who could `git push` to a repository - including via an AGit
workflow! - was able to change either of these settings. To guard
against this, the pre-receive hook will now check if either of these
options are present, and if so, will perform additional permission
checks to ensure that these can only be set by a repository owner or
an administrator. Additionally, changing these settings is disabled for
forks, even for the fork's owner.

There's still a case where the owner of a repository can change the
visibility of it, and it will not propagate to forks (it propagates to
forks when changing the visibility via the API), but that's an
inconsistency, not a security issue.

Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
Signed-off-by: Earl Warren <contact@earl-warren.org>
2024-04-19 16:53:14 +02:00
wxiaoguang
9b8de15797
Clean up log messages (#30313)
`log.Xxx("%v")` is not ideal, this PR adds necessary context messages.
Remove some unnecessary logs.

Co-authored-by: Giteabot <teabot@gitea.io>
(cherry picked from commit 83f83019ef3471b847a300f0821499b3896ec987)

Conflicts:
	- modules/util/util.go
          Conflict resolved by picking `util.Iif` from 654cfd1dfbd3f3f1d94addee50b6fe2b018a49c3
2024-04-15 20:01:35 +02:00
Earl Warren
998a431747
Do not update PRs based on events that happened before they existed
* Split TestPullRequest out of AddTestPullRequestTask
* A Created field is added to the Issue table
* The Created field is set to the time (with nano resolution) on creation
* Record the nano time repo_module.PushUpdateOptions is created by the hook
* The decision to update a pull request created before a commit was
  pushed is based on the time (with nano resolution) the git hook
  was run and the Created field

It ensures the following happens:

* commit C is pushed
* the git hook queues AddTestPullRequestTask for processing and returns with success
* TestPullRequest is not called yet
* a pull request P with commit C as the head is created
* TestPullRequest runs and ignores P because it was created after the commit was received

When the "created" column is NULL, no verification is done, pull
requests that were created before the column was created in the
database cannot be newer than the latest call to a git hook.

Fixes: https://codeberg.org/forgejo/forgejo/issues/2009
2024-04-11 11:16:23 +02:00
Gusted
79b7089360
[FEAT] Configure if protected branch rule should apply to admins
- Currently protected branch rules do not apply to admins, however in
some cases (like in the case of Forgejo project) you might also want to
apply these rules to admins to avoid accidental merges.
- Add new option to configure this on a per-rule basis.
- Adds integration tests.
- Resolves #65
2024-03-29 00:21:49 +01:00
Jason Song
18e95af24f
Restore deleted branches when syncing (#29898)
Regression of #29493. If a branch has been deleted, repushing it won't
restore it.

Lunny may have noticed that, but I didn't delve into the comment then
overlooked it:
https://github.com/go-gitea/gitea/pull/29493#discussion_r1509046867

The additional comments added are to explain the issue I found during
testing, which are unrelated to the fixes.

(cherry picked from commit f371f84fa3456c2a71470632b6458d81e4892a54)
2024-03-26 19:04:25 +01:00
Jason Song
375222a145 Sync branches first (gitea#29714)
Follow gitea#29493.

Sync branches to DB first, then trigger push events.
2024-03-18 17:39:34 +01:00
oliverpool
0a53eb838d use gitRepo.GetCommit directly
and give it a commitID instead of a branchName (a bit more correct and faster)
2024-03-18 17:38:58 +01:00
Jason Song
286d09203f Sync branches to DB immediately when handle git hook calling (gitea#29493)
Unlike other async processing in the queue, we should sync branches to
the DB immediately when handling git hook calling. If it fails, users
can see the error message in the output of the git command.

It can avoid potential inconsistency issues, and help #29494.

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2024-03-18 17:38:01 +01:00
Lunny Xiao
3d9afe8813
Move get/set default branch from git package to gitrepo package to hide repopath (#29126)
(cherry picked from commit 25b842df261452a29570ba89ffc3a4842d73f68c)

Conflicts:
	routers/web/repo/wiki.go
	services/repository/branch.go
	services/repository/migrate.go
	services/wiki/wiki.go
	also apply to Forgejo specific usage of the refactored functions
2024-03-11 23:36:59 +07:00
Lunny Xiao
894d9b2836
Move context from modules to services (#29440)
Since `modules/context` has to depend on `models` and many other
packages, it should be moved from `modules/context` to
`services/context` according to design principles. There is no logic
code change on this PR, only move packages.

- Move `code.gitea.io/gitea/modules/context` to
`code.gitea.io/gitea/services/context`
- Move `code.gitea.io/gitea/modules/contexttest` to
`code.gitea.io/gitea/services/contexttest` because of depending on
context
- Move `code.gitea.io/gitea/modules/upload` to
`code.gitea.io/gitea/services/context/upload` because of depending on
context

(cherry picked from commit 29f149bd9f517225a3c9f1ca3fb0a7b5325af696)

Conflicts:
	routers/api/packages/alpine/alpine.go
	routers/api/v1/repo/issue_reaction.go
	routers/install/install.go
	routers/web/admin/config.go
	routers/web/passkey.go
	routers/web/repo/search.go
	routers/web/repo/setting/default_branch.go
	routers/web/user/home.go
	routers/web/user/profile.go
	tests/integration/editor_test.go
	tests/integration/integration_test.go
	tests/integration/mirror_push_test.go
	trivial context conflicts
	also modified all other occurrences in Forgejo specific files
2024-03-06 12:10:43 +08:00
yp05327
480c61432a
Fix logic error from #28138 (#29417)
There's a miss in #28138:

![image](https://github.com/go-gitea/gitea/assets/18380374/b1e0c5fc-0e6e-44ab-9f6e-34bc8ffbe1cc)

https://github.com/go-gitea/gitea/pull/28138/files#diff-2556e62ad7204a230c91927a3f2115e25a2b688240d0ee1de6d34f0277f37dfeR162

@lunny
Not sure about the impact of this, but it will only effect 1.22, and
maybe we should fix it ASAP.

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit f8c1efe944c539396402fb014bbfdb67ba199ef2)
2024-03-06 12:10:43 +08:00
Lunny Xiao
6905540088
Use the database object format name but not read from git repoisitory everytime and fix possible migration wrong objectformat when migrating a sha256 repository (#29294)
Now we can get object format name from git command line or from the
database repository table. Assume the column is right, we don't need to
read from git command line every time.

This also fixed a possible bug that the object format is wrong when
migrating a sha256 repository from external.

<img width="658" alt="image"
src="https://github.com/go-gitea/gitea/assets/81045/6e9a9dcf-13bf-4267-928b-6bf2c2560423">

(cherry picked from commit b79c30435f439af8243ee281310258cdf141e27b)

Conflicts:
	routers/web/repo/blame.go
	services/agit/agit.go
	context
2024-02-26 22:30:26 +01:00
Earl Warren
e90db3f5cc
[CLI] implement forgejo-cli
(cherry picked from commit 2555e315f7)
(cherry picked from commit 51b9c9092e)

[CLI] implement forgejo-cli (squash) support initDB

(cherry picked from commit 5c31ae602a)
(cherry picked from commit bbf76489a7)

Conflicts:
	because of d0dbe52e76
	upgrade to https://pkg.go.dev/github.com/urfave/cli/v2
(cherry picked from commit b6c1bcc008)

[CLI] implement forgejo-cli actions

(cherry picked from commit 08be2b226e)
(cherry picked from commit b6cfa88c6e)
(cherry picked from commit 59704200de)

[CLI] implement forgejo-cli actions generate-secret

(cherry picked from commit 6f7905c8ec)
(cherry picked from commit e085d6d273)

[CLI] implement forgejo-cli actions generate-secret (squash) NoInit

(cherry picked from commit 962c944eb2)

[CLI] implement forgejo-cli actions register

(cherry picked from commit 2f95143000)
(cherry picked from commit 42f2f8731e)

[CLI] implement forgejo-cli actions register (squash) no private

Do not go through the private API, directly modify the database

(cherry picked from commit 1ba7c0d39d)

[CLI] implement forgejo-cli actions

(cherry picked from commit 6f7905c8ec)
(cherry picked from commit e085d6d273)

[CLI] implement forgejo-cli actions generate-secret (squash) NoInit

(cherry picked from commit 962c944eb2)
(cherry picked from commit 4c121ef022)

Conflicts:
	cmd/forgejo/actions.go
	tests/integration/cmd_forgejo_actions_test.go
(cherry picked from commit 36997a48e3)

[CLI] implement forgejo-cli actions (squash) restore --version

Refs: https://codeberg.org/forgejo/forgejo/issues/1134
(cherry picked from commit 9739eb52d8)

[CI] implement forgejo-cli (squash) the actions subcommand needs config

(cherry picked from commit def638475122a26082ab3835842c84cd03839154)

Conflicts:
	cmd/main.go
	https://codeberg.org/forgejo/forgejo/pulls/1209
(cherry picked from commit a1758a3910)
(cherry picked from commit 935fa650c7)
(cherry picked from commit cd21026bc9)
(cherry picked from commit 1700b8973a)
(cherry picked from commit 1def42a379)
(cherry picked from commit 839d97521d)
(cherry picked from commit fd8c13be6b)
(cherry picked from commit 588e5d552f)
(cherry picked from commit 151a726620)

[v1.22] [CLI] implement forgejo-cli

https://codeberg.org/forgejo/forgejo/pulls/1541
(cherry picked from commit 46708de7b9)
(cherry picked from commit a8e5c1369e)
(cherry picked from commit c8a32aaf24)

Conflicts:
	models/actions/main_test.go
	https://codeberg.org/forgejo/forgejo/pulls/1656
(cherry picked from commit 79f4553063)
(cherry picked from commit 0379da0cf5)
(cherry picked from commit 331d58c085)
(cherry picked from commit 89705502c4)
(cherry picked from commit 4723d5febf)
(cherry picked from commit e71b260130)
(cherry picked from commit 6a376a5b48)

Conflicts:
	cmd/main.go
	https://codeberg.org/forgejo/forgejo/pulls/1969
(cherry picked from commit 6ba97cf4b5)
(cherry picked from commit e0a6ebfeca)
(cherry picked from commit 5702aeab2d)
(cherry picked from commit f919c4d6c1)
(cherry picked from commit a26799a88a)
(cherry picked from commit b6ab473395)
(cherry picked from commit cf054a0461)
2024-02-05 13:33:58 +01:00
Lunny Xiao
5f82ead13c
Simplify how git repositories are opened (#28937)
## Purpose
This is a refactor toward building an abstraction over managing git
repositories.
Afterwards, it does not matter anymore if they are stored on the local
disk or somewhere remote.

## What this PR changes
We used `git.OpenRepository` everywhere previously.
Now, we should split them into two distinct functions:

Firstly, there are temporary repositories which do not change:

```go
git.OpenRepository(ctx, diskPath)
```

Gitea managed repositories having a record in the database in the
`repository` table are moved into the new package `gitrepo`:

```go
gitrepo.OpenRepository(ctx, repo_model.Repo)
```

Why is `repo_model.Repository` the second parameter instead of file
path?
Because then we can easily adapt our repository storage strategy.
The repositories can be stored locally, however, they could just as well
be stored on a remote server.

## Further changes in other PRs
- A Git Command wrapper on package `gitrepo` could be created. i.e.
`NewCommand(ctx, repo_model.Repository, commands...)`. `git.RunOpts{Dir:
repo.RepoPath()}`, the directory should be empty before invoking this
method and it can be filled in the function only. #28940
- Remove the `RepoPath()`/`WikiPath()` functions to reduce the
possibility of mistakes.

---------

Co-authored-by: delvh <dev.lh@web.de>
2024-01-27 21:09:51 +01:00
Lunny Xiao
4eb2a29910
Improve ObjectFormat interface (#28496)
The 4 functions are duplicated, especially as interface methods. I think
we just need to keep `MustID` the only one and remove other 3.

```
MustID(b []byte) ObjectID
MustIDFromString(s string) ObjectID
NewID(b []byte) (ObjectID, error)
NewIDFromString(s string) (ObjectID, error)
```

Introduced the new interfrace method `ComputeHash` which will replace
the interface `HasherInterface`. Now we don't need to keep two
interfaces.

Reintroduced `git.NewIDFromString` and `git.MustIDFromString`. The new
function will detect the hash length to decide which objectformat of it.
If it's 40, then it's SHA1. If it's 64, then it's SHA256. This will be
right if the commitID is a full one. So the parameter should be always a
full commit id.

@AdamMajer Please review.
2023-12-19 07:20:47 +00:00
Lunny Xiao
408a484224
Adjust object format interface (#28469)
- Remove `ObjectFormatID`
- Remove function `ObjectFormatFromID`.
- Use `Sha1ObjectFormat` directly but not a pointer because it's an
empty struct.
- Store `ObjectFormatName` in `repository` struct
2023-12-17 11:56:08 +00:00
Adam Majer
cbf923e87b
Abstract hash function usage (#28138)
Refactor Hash interfaces and centralize hash function. This will allow
easier introduction of different hash function later on.

This forms the "no-op" part of the SHA256 enablement patch.
2023-12-13 21:02:00 +00:00
Jason Song
0a2b93d411
Fix typo "GetLatestRunnerToken" (#27680) 2023-10-18 15:52:44 +00:00
JakobDev
76a85a4ce9
Final round of db.DefaultContext refactor (#27587)
Last part of #27065
2023-10-14 08:37:24 +00:00
JakobDev
ebe803e514
Penultimate round of db.DefaultContext refactor (#27414)
Part of #27065

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-10-11 04:24:07 +00:00
Lunny Xiao
673cf6af76
make writing main test easier (#27270)
This PR removed `unittest.MainTest` the second parameter
`TestOptions.GiteaRoot`. Now it detects the root directory by current
working directory.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-09-28 01:38:53 +00:00
Lunny Xiao
3b2da7e4ae
Redefine the meaning of column is_active to make Actions Registration Token generation easier (#27143)
Partially Fix #25041

This PR redefined the meaning of column `is_active` in table
`action_runner_token`.
Before this PR, `is_active` means whether it has been used by any
runner. If it's true, other runner cannot use it to register again.

In this PR, `is_active` means whether it's validated to be used to
register runner. And if it's true, then it can be used to register
runners until it become false. When creating a new `is_active` register
token, any previous tokens will be set `is_active` to false.
2023-09-27 05:37:48 +00:00
JakobDev
76659b1114
Reduce usage of db.DefaultContext (#27073)
Part of #27065

This reduces the usage of `db.DefaultContext`. I think I've got enough
files for the first PR. When this is merged, I will continue working on
this.

Considering how many files this PR affect, I hope it won't take to long
to merge, so I don't end up in the merge conflict hell.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-09-14 17:09:32 +00:00
CaiCandong
815d267c80
Fix verifyCommits error when push a new branch (#26664)
> ### Description
> If a new branch is pushed, and the repository has a rule that would
require signed commits for the new branch, the commit is rejected with a
500 error regardless of whether it's signed.
> 
> When pushing a new branch, the "old" commit is the empty ID
(0000000000000000000000000000000000000000). verifyCommits has no
provision for this and passes an invalid commit range to git rev-list.
Prior to 1.19 this wasn't an issue because only pre-existing individual
branches could be protected.
> 
> I was able to reproduce with
[try.gitea.io/CraigTest/test](https://try.gitea.io/CraigTest/test),
which is set up with a blanket rule to require commits on all branches.


Fix #25565
Very thanks to @Craig-Holmquist-NTI for reporting the bug and suggesting
an valid solution!

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-08-30 02:27:53 +00:00
Lunny Xiao
b167f35113
Add context parameter to some database functions (#26055)
To avoid deadlock problem, almost database related functions should be
have ctx as the first parameter.
This PR do a refactor for some of these functions.
2023-07-22 22:14:27 +08:00
Earl Warren
ad4c09b59f
parseScope with owner/repo always sets owner to zero (#25987)
Refs: https://codeberg.org/forgejo/forgejo/pulls/1001
2023-07-19 21:21:51 +08:00
silverwind
88f835192d
Replace interface{} with any (#25686)
Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`.

Basically the same [as golang did](2580d0e08d).
2023-07-04 18:36:08 +00:00
Lunny Xiao
f9cfd6ce5b
Use the type RefName for all the needed places and fix pull mirror sync bugs (#24634)
This PR replaces all string refName as a type `git.RefName` to make the
code more maintainable.

Fix #15367
Replaces #23070 
It also fixed a bug that tags are not sync because `git remote --prune
origin` will not remove local tags if remote removed.

We in fact should use `git fetch --prune --tags origin` but not `git
remote update origin` to do the sync.

Some answer from ChatGPT as ref.

> If the git fetch --prune --tags command is not working as expected,
there could be a few reasons why. Here are a few things to check:
> 
>Make sure that you have the latest version of Git installed on your
system. You can check the version by running git --version in your
terminal. If you have an outdated version, try updating Git and see if
that resolves the issue.
> 
>Check that your Git repository is properly configured to track the
remote repository's tags. You can check this by running git config
--get-all remote.origin.fetch and verifying that it includes
+refs/tags/*:refs/tags/*. If it does not, you can add it by running git
config --add remote.origin.fetch "+refs/tags/*:refs/tags/*".
> 
>Verify that the tags you are trying to prune actually exist on the
remote repository. You can do this by running git ls-remote --tags
origin to list all the tags on the remote repository.
> 
>Check if any local tags have been created that match the names of tags
on the remote repository. If so, these local tags may be preventing the
git fetch --prune --tags command from working properly. You can delete
local tags using the git tag -d command.

---------

Co-authored-by: delvh <dev.lh@web.de>
2023-05-26 01:04:48 +00:00
wxiaoguang
979f08053f
Fix logger refactoring regression: manager logging add (#24847)
The `rotate` option is optional.

Test result:

![image](https://github.com/go-gitea/gitea/assets/2114189/7cff8207-dcf3-4988-adef-247f1c9316e7)
2023-05-24 08:09:33 +08:00
Brecht Van Lommel
3588edbb08
Add gitea manager reload-templates command (#24843)
This can be useful to update custom templates in production mode, when
they are updated frequently and a full Gitea restart each time is
disruptive.
2023-05-22 23:51:40 +08:00
wxiaoguang
ec2a01d1e2
Fix regression: access log template, gitea manager cli command (#24838)
Close #24836

![image](https://github.com/go-gitea/gitea/assets/2114189/95b025d2-f25f-4246-a08a-fe44ecb787a9)

![image](https://github.com/go-gitea/gitea/assets/2114189/c3afe1fa-2a23-420d-a016-3b67dcd04cd5)
2023-05-22 09:38:38 +08:00
wxiaoguang
4647660776
Rewrite logger system (#24726)
## ⚠️ Breaking

The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.

Although many legacy options still work, it's encouraged to upgrade to
the new options.

The SMTP logger is deleted because SMTP is not suitable to collect logs.

If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.

## Description

Close #12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)

Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)

There is a new document (with examples): `logging-config.en-us.md`

This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.

## The old problems

The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.


## The new design

See `logger.go` for documents.


## Screenshot

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff)


![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9)


![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee)

</details>

## TODO

* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)

---------

Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-21 22:35:11 +00:00
wxiaoguang
b9a97ccd0e
Refactor web route (#24080)
The old code is unnecessarily complex, and has many misuses.

Old code "wraps" a lot, wrap wrap wrap, it's difficult to understand
which kind of handler is used.

The new code uses a general approach, we do not need to write all kinds
of handlers into the "wrapper", do not need to wrap them again and
again.

New code, there are only 2 concepts:

1. HandlerProvider: `func (h any) (handlerProvider func (next)
http.Handler)`, it can be used as middleware
2. Use HandlerProvider to get the final HandlerFunc, and use it for
`r.Get()`


And we can decouple the route package from context package (see the
TODO).

# FAQ

## Is `reflect` safe?

Yes, all handlers are checked during startup, see the `preCheckHandler`
comment. If any handler is wrong, developers could know it in the first
time.

## Does `reflect` affect performance?

No. https://github.com/go-gitea/gitea/pull/24080#discussion_r1164825901

1. This reflect code only runs for each web handler call, handler is far
more slower: 10ms-50ms
2. The reflect is pretty fast (comparing to other code): 0.000265ms
3. XORM has more reflect operations already
2023-04-20 14:49:06 -04:00
techknowlogick
4014200021
add CLI command to register runner tokens (#23762)
This is a CLI command to generate new tokens for the runners to register
with

Fix https://github.com/go-gitea/gitea/issues/23643

---------

Co-authored-by: delvh <dev.lh@web.de>
2023-04-17 13:07:13 -04:00
wxiaoguang
f4538791f5
Refactor internal API for git commands, use meaningful messages instead of "Internal Server Error" (#23687)
# Why this PR comes

At first, I'd like to help users like #23636 (there are a lot)

The unclear "Internal Server Error" is quite anonying, scare users,
frustrate contributors, nobody knows what happens.

So, it's always good to provide meaningful messages to end users (of
course, do not leak sensitive information).

When I started working on the "response message to end users", I found
that the related code has a lot of technical debt. A lot of copy&paste
code, unclear fields and usages.

So I think it's good to make everything clear.

# Tech Backgrounds

Gitea has many sub-commands, some are used by admins, some are used by
SSH servers or Git Hooks. Many sub-commands use "internal API" to
communicate with Gitea web server.

Before, Gitea server always use `StatusCode + Json "err" field` to
return messages.

* The CLI sub-commands: they expect to show all error related messages
to site admin
* The Serv/Hook sub-commands (for git clients): they could only show
safe messages to end users, the error log could only be recorded by
"SSHLog" to Gitea web server.

In the old design, it assumes that:

* If the StatusCode is 500 (in some functions), then the "err" field is
error log, shouldn't be exposed to git client.
* If the StatusCode is 40x, then the "err" field could be exposed. And
some functions always read the "err" no matter what the StatusCode is.

The old code is not strict, and it's difficult to distinguish the
messages clearly and then output them correctly.

# This PR

To help to remove duplicate code and make everything clear, this PR
introduces `ResponseExtra` and `requestJSONResp`.

* `ResponseExtra` is a struct which contains "extra" information of a
internal API response, including StatusCode, UserMsg, Error
* `requestJSONResp` is a generic function which can be used for all
cases to help to simplify the calls.
* Remove all `map["err"]`, always use `private.Response{Err}` to
construct error messages.
* User messages and error messages are separated clearly, the `fail` and
`handleCliResponseExtra` will output correct messages.
* Replace all `Internal Server Error` messages with meaningful (still
safe) messages.

This PR saves more than 300 lines, while makes the git client messages
more clear.

Many gitea-serv/git-hook related essential functions are covered by
tests.

---------

Co-authored-by: delvh <dev.lh@web.de>
2023-03-29 14:32:26 +08:00
Jason Song
04347eb810
Use context parameter in services/repository (#23186)
Use context parameter in `services/repository`.

And use `cache.WithCacheContext(ctx)` to generate push action history
feeds.

Fix #23160
2023-02-28 16:17:51 -06:00
Lunny Xiao
c53ad052d8
Refactor the setting to make unit test easier (#22405)
Some bugs caused by less unit tests in fundamental packages. This PR
refactor `setting` package so that create a unit test will be easier
than before.

- All `LoadFromXXX` files has been splited as two functions, one is
`InitProviderFromXXX` and `LoadCommonSettings`. The first functions will
only include the code to create or new a ini file. The second function
will load common settings.
- It also renames all functions in setting from `newXXXService` to
`loadXXXSetting` or `loadXXXFrom` to make the function name less
confusing.
- Move `XORMLog` to `SQLLog` because it's a better name for that.

Maybe we should finally move these `loadXXXSetting` into the `XXXInit`
function? Any idea?

---------

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
2023-02-20 00:12:01 +08:00
Lunny Xiao
bd820aa9c5
Add context cache as a request level cache (#22294)
To avoid duplicated load of the same data in an HTTP request, we can set
a context cache to do that. i.e. Some pages may load a user from a
database with the same id in different areas on the same page. But the
code is hidden in two different deep logic. How should we share the
user? As a result of this PR, now if both entry functions accept
`context.Context` as the first parameter and we just need to refactor
`GetUserByID` to reuse the user from the context cache. Then it will not
be loaded twice on an HTTP request.

But of course, sometimes we would like to reload an object from the
database, that's why `RemoveContextData` is also exposed.

The core context cache is here. It defines a new context
```go
type cacheContext struct {
	ctx  context.Context
	data map[any]map[any]any
        lock sync.RWMutex
}

var cacheContextKey = struct{}{}

func WithCacheContext(ctx context.Context) context.Context {
	return context.WithValue(ctx, cacheContextKey, &cacheContext{
		ctx:  ctx,
		data: make(map[any]map[any]any),
	})
}
```

Then you can use the below 4 methods to read/write/del the data within
the same context.

```go
func GetContextData(ctx context.Context, tp, key any) any
func SetContextData(ctx context.Context, tp, key, value any)
func RemoveContextData(ctx context.Context, tp, key any)
func GetWithContextCache[T any](ctx context.Context, cacheGroupKey string, cacheTargetID any, f func() (T, error)) (T, error)
```

Then let's take a look at how `system.GetString` implement it.

```go
func GetSetting(ctx context.Context, key string) (string, error) {
	return cache.GetWithContextCache(ctx, contextCacheKey, key, func() (string, error) {
		return cache.GetString(genSettingCacheKey(key), func() (string, error) {
			res, err := GetSettingNoCache(ctx, key)
			if err != nil {
				return "", err
			}
			return res.SettingValue, nil
		})
	})
}
```

First, it will check if context data include the setting object with the
key. If not, it will query from the global cache which may be memory or
a Redis cache. If not, it will get the object from the database. In the
end, if the object gets from the global cache or database, it will be
set into the context cache.

An object stored in the context cache will only be destroyed after the
context disappeared.
2023-02-15 21:37:34 +08:00
Jason Song
4011821c94
Implement actions (#21937)
Close #13539.

Co-authored by: @lunny @appleboy @fuxiaohei and others.

Related projects:
- https://gitea.com/gitea/actions-proto-def
- https://gitea.com/gitea/actions-proto-go
- https://gitea.com/gitea/act
- https://gitea.com/gitea/act_runner

### Summary

The target of this PR is to bring a basic implementation of "Actions",
an internal CI/CD system of Gitea. That means even though it has been
merged, the state of the feature is **EXPERIMENTAL**, and please note
that:

- It is disabled by default;
- It shouldn't be used in a production environment currently;
- It shouldn't be used in a public Gitea instance currently;
- Breaking changes may be made before it's stable.

**Please comment on #13539 if you have any different product design
ideas**, all decisions reached there will be adopted here. But in this
PR, we don't talk about **naming, feature-creep or alternatives**.

### ⚠️ Breaking

`gitea-actions` will become a reserved user name. If a user with the
name already exists in the database, it is recommended to rename it.

### Some important reviews

- What is `DEFAULT_ACTIONS_URL` in `app.ini` for?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1055954954
- Why the api for runners is not under the normal `/api/v1` prefix?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1061173592
- Why DBFS?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1061301178
- Why ignore events triggered by `gitea-actions` bot?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1063254103
- Why there's no permission control for actions?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1090229868

### What it looks like

<details>

#### Manage runners

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205870657-c72f590e-2e08-4cd4-be7f-2e0abb299bbf.png">

#### List runs

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205872794-50fde990-2b45-48c1-a178-908e4ec5b627.png">


#### View logs

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205872501-9b7b9000-9542-4991-8f55-18ccdada77c3.png">



</details>

### How to try it

<details>

#### 1. Start Gitea

Clone this branch and [install from
source](https://docs.gitea.io/en-us/install-from-source).

Add additional configurations in `app.ini` to enable Actions:

```ini
[actions]
ENABLED = true
```

Start it.

If all is well, you'll see the management page of runners:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205877365-8e30a780-9b10-4154-b3e8-ee6c3cb35a59.png">


#### 2. Start runner

Clone the [act_runner](https://gitea.com/gitea/act_runner), and follow
the
[README](https://gitea.com/gitea/act_runner/src/branch/main/README.md)
to start it.

If all is well, you'll see a new runner has been added:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205878000-216f5937-e696-470d-b66c-8473987d91c3.png">

#### 3. Enable actions for a repo

Create a new repo or open an existing one, check the `Actions` checkbox
in settings and submit.

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205879705-53e09208-73c0-4b3e-a123-2dcf9aba4b9c.png">
<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205879383-23f3d08f-1a85-41dd-a8b3-54e2ee6453e8.png">

If all is well, you'll see a new tab "Actions":

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205881648-a8072d8c-5803-4d76-b8a8-9b2fb49516c1.png">

#### 4. Upload workflow files

Upload some workflow files to `.gitea/workflows/xxx.yaml`, you can
follow the [quickstart](https://docs.github.com/en/actions/quickstart)
of GitHub Actions. Yes, Gitea Actions is compatible with GitHub Actions
in most cases, you can use the same demo:

```yaml
name: GitHub Actions Demo
run-name: ${{ github.actor }} is testing out GitHub Actions 🚀
on: [push]
jobs:
  Explore-GitHub-Actions:
    runs-on: ubuntu-latest
    steps:
      - run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
      - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"
      - run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner."
      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
      - name: List files in the repository
        run: |
          ls ${{ github.workspace }}
      - run: echo "🍏 This job's status is ${{ job.status }}."
```

If all is well, you'll see a new run in `Actions` tab:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205884473-79a874bc-171b-4aaf-acd5-0241a45c3b53.png">

#### 5. Check the logs of jobs

Click a run and you'll see the logs:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205884800-994b0374-67f7-48ff-be9a-4c53f3141547.png">

#### 6. Go on

You can try more examples in [the
documents](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)
of GitHub Actions, then you might find a lot of bugs.

Come on, PRs are welcome.

</details>

See also: [Feature Preview: Gitea
Actions](https://blog.gitea.io/2022/12/feature-preview-gitea-actions/)

---------

Co-authored-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: ChristopherHX <christopher.homberger@web.de>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2023-01-31 09:45:19 +08:00