Commit graph

56 commits

Author SHA1 Message Date
Jason Song
0c40cff9a4
Clear up old Actions logs (#31735)
Part of #24256.

Clear up old action logs to free up storage space.

Users will see a message indicating that the log has been cleared if
they view old tasks.

<img width="1361" alt="image"
src="https://github.com/user-attachments/assets/9f0f3a3a-bc5a-402f-90ca-49282d196c22">

Docs: https://gitea.com/gitea/docs/pulls/40

---------

Co-authored-by: silverwind <me@silverwind.io>
(cherry picked from commit 687c1182482ad9443a5911c068b317a91c91d586)

Conflicts:
	custom/conf/app.example.ini
	routers/web/repo/actions/view.go
  trivial context conflict
2024-08-04 18:24:10 +02:00
Jason Song
43b184cf07
Move registerActionsCleanup to initActionsTasks (#31721)
There's already `initActionsTasks`; it will avoid additional check for
if Actions enabled to move `registerActionsCleanup` into it.

And we don't really need `OlderThanConfig`.

(cherry picked from commit f989f464386139592b6911cad1be4c901eb97fe5)
2024-08-04 10:14:34 +02:00
TheFox0x7
4de909747b Add testifylint to lint checks (#4535)
Some checks are pending
/ release (push) Waiting to run
testing / test-remote-cacher (map[image:redis:7.2 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:registry.redict.io/redict:7.3.0-scratch port:6379]) (push) Blocked by required conditions
testing / test-mysql (push) Blocked by required conditions
testing / test-pgsql (push) Blocked by required conditions
testing / backend-checks (push) Waiting to run
testing / frontend-checks (push) Waiting to run
testing / test-unit (push) Blocked by required conditions
testing / test-remote-cacher (map[image:docker.io/valkey/valkey:7.2.5-alpine3.19 port:6379]) (push) Blocked by required conditions
testing / test-remote-cacher (map[image:ghcr.io/microsoft/garnet-alpine:1.0.14 port:6379]) (push) Blocked by required conditions
testing / test-sqlite (push) Blocked by required conditions
testing / security-check (push) Blocked by required conditions
go-require lint is ignored for now

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4535
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: TheFox0x7 <thefox0x7@gmail.com>
Co-committed-by: TheFox0x7 <thefox0x7@gmail.com>
2024-07-30 19:41:10 +00:00
0ko
e498069fd5 Update checker setting updates
- revert c53f802778
- enable by default on install page
- save checkbox state between form submissions

Resolves https://codeberg.org/forgejo/forgejo/issues/2859
2024-03-31 10:52:24 +05:00
wxiaoguang
65248945c9
Refactor locale&string&template related code (#29165)
Clarify when "string" should be used (and be escaped), and when
"template.HTML" should be used (no need to escape)

And help PRs like  #29059 , to render the error messages correctly.

(cherry picked from commit f3eb835886031df7a562abc123c3f6011c81eca8)

Conflicts:
	modules/web/middleware/binding.go
	routers/web/feed/convert.go
	tests/integration/branches_test.go
	tests/integration/repo_branch_test.go
	trivial context conflicts
2024-02-16 15:20:52 +01:00
Gusted
6fc17ad37b
[PRIVACY] Add a DNS method to fetch new updates
- Use TXT records in order to determine the latest available version.
- This addresses a valid privacy issue, as with HTTP requests the server
can keep track(estimated) of how many instances are using Forgejo, with
DNS that's basically not possible as the server will never receive any
data, as the only ones receiving data are DNS resolvers.

(cherry picked from commit 0baefb546a)
(cherry picked from commit e8ee41880b)
(cherry picked from commit 7eca4f3bf1)
(cherry picked from commit 6dde3992dc)
(cherry picked from commit fb3a37fbfc)
(cherry picked from commit 8304af1e9d)
(cherry picked from commit 0543a7d12a)
(cherry picked from commit c3a22933b7)
(cherry picked from commit e243707694)
(cherry picked from commit 7eb6d1bcf7)
(cherry picked from commit 1d7b9535cd)
(cherry picked from commit 05920dce67)
(cherry picked from commit f173f27d7c)
(cherry picked from commit 90e1c9340e)
(cherry picked from commit de68610ea7)
(cherry picked from commit 8d5757ea04)
(cherry picked from commit c7a7fff316)
(cherry picked from commit 39ac8b8fc1)
(cherry picked from commit 9889203301)

[PRIVACY]: Adjust update checker description

- Resolves #323
- Adjust the description of the update check function on the
installation page to describe the privacy method instead of the HTTP
method by checking gitea.io

(cherry picked from commit 61eae5b105)
(cherry picked from commit 091def20a1)
(cherry picked from commit d5d11bf45a)
(cherry picked from commit 71863d4707)
(cherry picked from commit 11ece4aab1)
(cherry picked from commit afdd7e714f)
(cherry picked from commit 39170e2f1d)
(cherry picked from commit 4b3a52aab8)
(cherry picked from commit 9d763c5fc8)
(cherry picked from commit 638db15482)
(cherry picked from commit a52bfdd8e7)
(cherry picked from commit dc93d00e85)
(cherry picked from commit 0bc4b3508c)
(cherry picked from commit 3f760d85a4)
(cherry picked from commit ecc2716785)
(cherry picked from commit 6334d5677e)

Conflicts:
	modules/updatechecker/update_checker.go
	UpdateRemoteVersion now has a context argument. However, in
	the updated code from Gitea the context comes from the HTTP
	request and does not actually provide any useful context.
	Replace that with context.Background()
(cherry picked from commit ca2200767e)
(cherry picked from commit f46feca224)
(cherry picked from commit a800a66ca8)
(cherry picked from commit 21f017454e)
(cherry picked from commit 762d18e09a)
(cherry picked from commit 6d28e120c1)
(cherry picked from commit 4fcd7e7cbf)
(cherry picked from commit 9516285112)
(cherry picked from commit 0dd752a262)
(cherry picked from commit b1f1af7fe0)
(cherry picked from commit 71937deaa5)
(cherry picked from commit 93e41a7823)
(cherry picked from commit f497ed30cf)
(cherry picked from commit bff3346f7d)
(cherry picked from commit 906ae19c0f)
2024-02-05 15:38:36 +01:00
Gusted
c53f802778
[PRIVACY] Disable update_checker cron tasks
- This is being disabled as it will weekly connect to a domain.
- This only affects existing installations, as new installations will
have a explicit value being written into app.ini due to https://github.com/go-gitea/gitea/pull/21655

(cherry picked from commit cd0b8b6852)
(cherry picked from commit 58d43867ca)
(cherry picked from commit f290b91e50)
(cherry picked from commit 5bd6d75554)
(cherry picked from commit e5d0abb592)
(cherry picked from commit 4b381ce77e)
(cherry picked from commit b6a9b1390d)
(cherry picked from commit 3674362871)
(cherry picked from commit af0f606fc1)
(cherry picked from commit 49560da6fb)
(cherry picked from commit ca06e27c5b)
(cherry picked from commit 163df86df5)
(cherry picked from commit 0391c6ae42)
(cherry picked from commit f282f2d42f)
(cherry picked from commit 68d31cf247)
(cherry picked from commit cfd4a479e9)
(cherry picked from commit add969bdf6)
(cherry picked from commit c0f3f1f1e7)
(cherry picked from commit f11dca9090)
(cherry picked from commit 283810387f)
(cherry picked from commit 08af1f0ea8)
(cherry picked from commit 1c1a6da319)
(cherry picked from commit c1444bae65)
(cherry picked from commit 6361df634b)
(cherry picked from commit cc8bb3f946)
(cherry picked from commit c7ae4411fd)
(cherry picked from commit f73f4eef02)
(cherry picked from commit 2122896e85)
(cherry picked from commit d469c99671)
(cherry picked from commit 41dbbd731e)
(cherry picked from commit e8e742b437)
(cherry picked from commit 0fa9f87b5f)
(cherry picked from commit a92c724a56)
(cherry picked from commit 6a18d153e1)
(cherry picked from commit 0dd35a13fd)
(cherry picked from commit 5ae0cbe0ae)
(cherry picked from commit 13bd3cf37b)
(cherry picked from commit a163efa6cf)
(cherry picked from commit 422a6e8329)
(cherry picked from commit 125fb33e1b)
(cherry picked from commit fd39d15a92)
(cherry picked from commit b581474513)
(cherry picked from commit 987d198523)
(cherry picked from commit f06eed025f)
(cherry picked from commit 6f9f415644)
(cherry picked from commit eb2c878deb)
(cherry picked from commit 70daffa9fe)
(cherry picked from commit 3a6ae6fa49)
(cherry picked from commit 682a564b04)
2024-02-05 15:38:36 +01:00
wxiaoguang
11f0519ad8
Update go dependencies (#28518)
Update golang.org/x/crypto for CVE-2023-48795 and update other packages.
`go-git` is not updated because it needs time to figure out why some
tests fail.
2023-12-19 09:18:42 +08:00
Earl Warren
fab73e4a3a
Actually recover from a panic in cron task (#28409)
- Currently there's code to recover gracefully from panics that happen
within the execution of cron tasks. However this recover code wasn't
being run, because `RunWithShutdownContext` also contains code to
recover from any panic and then gracefully shutdown Forgejo. Because
`RunWithShutdownContext` registers that code as last, that would get run
first which in this case is not behavior that we want.
- Move the recover code to inside the function, so that is run first
before `RunWithShutdownContext`'s recover code (which is now a noop).

Fixes: https://codeberg.org/forgejo/forgejo/issues/1910

Co-authored-by: Gusted <postmaster@gusted.xyz>
2023-12-10 20:15:06 +00:00
Jason Song
1be49fdda6
Improve retrying index issues (#27554)
Fix #27540
2023-10-15 18:56:57 +00:00
wxiaoguang
1d155a43ad
Fix data-race bug when accessing task.LastRun (#27584) 2023-10-11 14:51:20 +00:00
Earl Warren
1050d7a78f
show manual cron run's last time (#27544)
- Currently in the cron tasks, the 'Previous Time' only displays the
previous time of when the cron library executes the function, but not
any of the manual executions of the task.
- Store the last run's time in memory in the Task struct and use that,
when that time is later than time that the cron library has executed
this task.
- This ensures that if an instance admin manually starts a task, there's
feedback that this task is/has been run, because the task might be run
that quick, that the status icon already has been changed to an
checkmark,
- Tasks that are executed at startup now reflect this as well, as the
time of the execution of that task on startup is now being shown as
'Previous Time'.
- Added integration tests for the API part, which is easier to test
because querying the HTML table of cron tasks is non-trivial.
- Resolves https://codeberg.org/forgejo/forgejo/issues/949

(cherry picked from commit fd34fdac14)

---------

Co-authored-by: Gusted <postmaster@gusted.xyz>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: silverwind <me@silverwind.io>
2023-10-11 07:28:16 +00:00
JakobDev
cf0df023be
More db.DefaultContext refactor (#27265)
Part of #27065

This PR touches functions used in templates. As templates are not static
typed, errors are harder to find, but I hope I catch it all. I think
some tests from other persons do not hurt.
2023-09-29 12:12:54 +00:00
JakobDev
7047df36d4
Another round of db.DefaultContext refactor (#27103)
Part of #27065

---------

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-09-25 13:17:37 +00:00
FuXiaoHei
460a2b0edf
Artifacts retention and auto clean up (#26131)
Currently, Artifact does not have an expiration and automatic cleanup
mechanism, and this feature needs to be added. It contains the following
key points:

- [x] add global artifact retention days option in config file. Default
value is 90 days.
- [x] add cron task to clean up expired artifacts. It should run once a
day.
- [x] support custom retention period from `retention-days: 5` in
`upload-artifact@v3`.
- [x] artifacts link in actions view should be non-clickable text when
expired.
2023-09-06 07:41:06 +00:00
Lunny Xiao
0d55f64e6c
chore(actions): support cron schedule task (#26655)
Replace #22751 

1. only support the default branch in the repository setting.
2. autoload schedule data from the schedule table after starting the
service.
3. support specific syntax like `@yearly`, `@monthly`, `@weekly`,
`@daily`, `@hourly`

## How to use

See the [GitHub Actions
document](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule)
for getting more detailed information.

```yaml
on:
  schedule:
    - cron: '30 5 * * 1,3'
    - cron: '30 5 * * 2,4'

jobs:
  test_schedule:
    runs-on: ubuntu-latest
    steps:
      - name: Not on Monday or Wednesday
        if: github.event.schedule != '30 5 * * 1,3'
        run: echo "This step will be skipped on Monday and Wednesday"
      - name: Every time
        run: echo "This step will always run"
```

Signed-off-by: Bo-Yi.Wu <appleboy.tw@gmail.com>

---------


Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-08-24 03:06:51 +00:00
Jason Song
47fddaadc8
Support rebuilding issue indexer manually (#26546)
Provide a way to rebuild issue indexer manually.

So if the indexer get outdated because of some bugs like #26539, we can
rebuild it.

<img width="1104" alt="image"
src="https://github.com/go-gitea/gitea/assets/9418365/ac242e29-6f04-47ca-b3d0-801a796448d3">

Co-authored-by: Giteabot <teabot@gitea.io>
2023-08-17 14:05:17 +00:00
KN4CK3R
0c6ae61229
Allow package cleanup from admin page (#25307)
Until now expired package data gets deleted daily by a cronjob. The
admin page shows the size of all packages and the size of unreferenced
data. The users (#25035, #20631) expect the deletion of this data if
they run the cronjob from the admin page but the job only deletes data
older than 24h.

This PR adds a new button which deletes all expired data.

![grafik](https://github.com/go-gitea/gitea/assets/1666336/b3e35d73-9496-4fa7-a20c-e5d30b1f6850)

---------

Co-authored-by: silverwind <me@silverwind.io>
2023-08-08 00:46:10 +00:00
Chongyi Zheng
f2138d6968
Replace gogs/cron with go-co-op/gocron (#25977)
Replace `github.com/gogs/cron` with `github.com/go-co-op/gocron` as the
former package is not maintained for many years.

---------

Co-authored-by: delvh <dev.lh@web.de>
2023-07-24 04:13:41 +00:00
silverwind
88f835192d
Replace interface{} with any (#25686)
Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`.

Basically the same [as golang did](2580d0e08d).
2023-07-04 18:36:08 +00:00
Lunny Xiao
e79ff50560
Use the new download domain replace the old (#25405)
As title.
2023-06-21 03:11:17 +00:00
KN4CK3R
df789d962b
Add Cargo package registry (#21888)
This PR implements a [Cargo registry](https://doc.rust-lang.org/cargo/)
to manage Rust packages. This package type was a little bit more
complicated because Cargo needs an additional Git repository to store
its package index.

Screenshots:

![grafik](https://user-images.githubusercontent.com/1666336/203102004-08d812ac-c066-4969-9bda-2fed818554eb.png)

![grafik](https://user-images.githubusercontent.com/1666336/203102141-d9970f14-dca6-4174-b17a-50ba1bd79087.png)

![grafik](https://user-images.githubusercontent.com/1666336/203102244-dc05743b-78b6-4d97-998e-ef76341a978f.png)

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-05 18:12:31 +08:00
wxiaoguang
6bc3079c00
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)

## Review without space diff

https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1

## Purpose of this PR

1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command

## The main idea of this PR

* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
    * Before: `AddArguments("-m").AddDynamicArguments(message)`
    * After: `AddOptionValues("-m", message)`
    * -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`

## FAQ

### Why these changes were not done in #21535 ?

#21535 is mainly a search&replace, it did its best to not change too
much logic.

Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.


### The naming of `AddOptionXxx`

According to git's manual, the `--xxx` part is called `option`.

### How can it guarantee that `internal.CmdArg` won't be not misused?

Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.

And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.

### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?

Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.

### Why there was a `CmdArgCheck` and why it's removed?

At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.


### Why many codes for `signArg == ""` is deleted?

Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-04 10:30:43 +08:00
Jason Song
4011821c94
Implement actions (#21937)
Close #13539.

Co-authored by: @lunny @appleboy @fuxiaohei and others.

Related projects:
- https://gitea.com/gitea/actions-proto-def
- https://gitea.com/gitea/actions-proto-go
- https://gitea.com/gitea/act
- https://gitea.com/gitea/act_runner

### Summary

The target of this PR is to bring a basic implementation of "Actions",
an internal CI/CD system of Gitea. That means even though it has been
merged, the state of the feature is **EXPERIMENTAL**, and please note
that:

- It is disabled by default;
- It shouldn't be used in a production environment currently;
- It shouldn't be used in a public Gitea instance currently;
- Breaking changes may be made before it's stable.

**Please comment on #13539 if you have any different product design
ideas**, all decisions reached there will be adopted here. But in this
PR, we don't talk about **naming, feature-creep or alternatives**.

### ⚠️ Breaking

`gitea-actions` will become a reserved user name. If a user with the
name already exists in the database, it is recommended to rename it.

### Some important reviews

- What is `DEFAULT_ACTIONS_URL` in `app.ini` for?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1055954954
- Why the api for runners is not under the normal `/api/v1` prefix?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1061173592
- Why DBFS?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1061301178
- Why ignore events triggered by `gitea-actions` bot?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1063254103
- Why there's no permission control for actions?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1090229868

### What it looks like

<details>

#### Manage runners

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205870657-c72f590e-2e08-4cd4-be7f-2e0abb299bbf.png">

#### List runs

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205872794-50fde990-2b45-48c1-a178-908e4ec5b627.png">


#### View logs

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205872501-9b7b9000-9542-4991-8f55-18ccdada77c3.png">



</details>

### How to try it

<details>

#### 1. Start Gitea

Clone this branch and [install from
source](https://docs.gitea.io/en-us/install-from-source).

Add additional configurations in `app.ini` to enable Actions:

```ini
[actions]
ENABLED = true
```

Start it.

If all is well, you'll see the management page of runners:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205877365-8e30a780-9b10-4154-b3e8-ee6c3cb35a59.png">


#### 2. Start runner

Clone the [act_runner](https://gitea.com/gitea/act_runner), and follow
the
[README](https://gitea.com/gitea/act_runner/src/branch/main/README.md)
to start it.

If all is well, you'll see a new runner has been added:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205878000-216f5937-e696-470d-b66c-8473987d91c3.png">

#### 3. Enable actions for a repo

Create a new repo or open an existing one, check the `Actions` checkbox
in settings and submit.

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205879705-53e09208-73c0-4b3e-a123-2dcf9aba4b9c.png">
<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205879383-23f3d08f-1a85-41dd-a8b3-54e2ee6453e8.png">

If all is well, you'll see a new tab "Actions":

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205881648-a8072d8c-5803-4d76-b8a8-9b2fb49516c1.png">

#### 4. Upload workflow files

Upload some workflow files to `.gitea/workflows/xxx.yaml`, you can
follow the [quickstart](https://docs.github.com/en/actions/quickstart)
of GitHub Actions. Yes, Gitea Actions is compatible with GitHub Actions
in most cases, you can use the same demo:

```yaml
name: GitHub Actions Demo
run-name: ${{ github.actor }} is testing out GitHub Actions 🚀
on: [push]
jobs:
  Explore-GitHub-Actions:
    runs-on: ubuntu-latest
    steps:
      - run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
      - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"
      - run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner."
      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
      - name: List files in the repository
        run: |
          ls ${{ github.workspace }}
      - run: echo "🍏 This job's status is ${{ job.status }}."
```

If all is well, you'll see a new run in `Actions` tab:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205884473-79a874bc-171b-4aaf-acd5-0241a45c3b53.png">

#### 5. Check the logs of jobs

Click a run and you'll see the logs:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205884800-994b0374-67f7-48ff-be9a-4c53f3141547.png">

#### 6. Go on

You can try more examples in [the
documents](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)
of GitHub Actions, then you might find a lot of bugs.

Come on, PRs are welcome.

</details>

See also: [Feature Preview: Gitea
Actions](https://blog.gitea.io/2022/12/feature-preview-gitea-actions/)

---------

Co-authored-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: ChristopherHX <christopher.homberger@web.de>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2023-01-31 09:45:19 +08:00
zeripath
2cc3a6381c
Add cron method to gc LFS MetaObjects (#22385)
This PR adds a task to the cron service to allow garbage collection of
LFS meta objects. As repositories may have a large number of
LFSMetaObjects, an updated column is added to this table and it is used
to perform a generational GC to attempt to reduce the amount of work.
(There may need to be a bit more work here but this is probably enough
for the moment.)

Fix #7045

Signed-off-by: Andrew Thornton <art27@cantab.net>
2023-01-16 13:50:53 -06:00
zeripath
651fe4bb7d
Add doctor command for full GC of LFS (#21978)
The recent PR adding orphaned checks to the LFS storage is not
sufficient to completely GC LFS, as it is possible for LFSMetaObjects to
remain associated with repos but still need to be garbage collected.

Imagine a situation where a branch is uploaded containing LFS files but
that branch is later completely deleted. The LFSMetaObjects will remain
associated with the Repository but the Repository will no longer contain
any pointers to the object.

This PR adds a second doctor command to perform a full GC.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-12-15 20:44:16 +00:00
flynnnnnnnnnn
e81ccc406b
Implement FSFE REUSE for golang files (#21840)
Change all license headers to comply with REUSE specification.

Fix #16132

Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-11-27 18:20:29 +00:00
wxiaoguang
dcd9fc7ee8
Refactor git command arguments and make all arguments to be safe to be used (#21535)
Follow #21464

Make all git command arguments strictly safe. Most changes are one-to-one replacing, keep all existing logic.
2022-10-23 22:44:45 +08:00
Lunny Xiao
f860a6d2e4
Add system setting table with cache and also add cache supports for user setting (#18058) 2022-10-17 07:29:26 +08:00
Lunny Xiao
1d8543e7db
Move some files into models' sub packages (#20262)
* Move some files into models' sub packages

* Move functions

* merge main branch

* Fix check

* fix check

* Fix some tests

* Fix lint

* Fix lint

* Revert lint changes

* Fix error comments

* Fix lint

Co-authored-by: 6543 <6543@obermui.de>
2022-08-25 10:31:57 +08:00
Gusted
5d3f99c7c6
Make better use of i18n (#20096)
* Prototyping

* Start work on creating offsets

* Modify tests

* Start prototyping with actual MPH

* Twiddle around

* Twiddle around comments

* Convert templates

* Fix external languages

* Fix latest translation

* Fix some test

* Tidy up code

* Use simple map

* go mod tidy

* Move back to data structure

- Uses less memory by creating for each language a map.

* Apply suggestions from code review

Co-authored-by: delvh <dev.lh@web.de>

* Add some comments

* Fix tests

* Try to fix tests

* Use en-US as defacto fallback

* Use correct slices

* refactor (#4)

* Remove TryTr, add log for missing translation key

* Refactor i18n

- Separate dev and production locale stores.
- Allow for live-reloading in dev mode.

Co-authored-by: zeripath <art27@cantab.net>

* Fix live-reloading & check for errors

* Make linter happy

* live-reload with periodic check (#5)

* Fix tests

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2022-06-26 22:19:22 +08:00
Lunny Xiao
110fc57cbc
Move some code into models/git (#19879)
* Move access and repo permission to models/perm/access

* fix test

* Move some git related files into sub package models/git

* Fix build

* fix git test

* move lfs to sub package

* move more git related functions to models/git

* Move functions sequence

* Some improvements per @KN4CK3R and @delvh
2022-06-12 23:51:54 +08:00
Paweł Bogusławski
14d96ff7ac
Disable unnecessary mirroring elements (#18527)
* Disable unnecessary mirroring elements

This mod fixes disabling unnecessary mirroring elements.

Related: https://github.com/go-gitea/gitea/pull/16957
Related: https://github.com/go-gitea/gitea/pull/13084
Author-Change-Id: IB#1105104

* Checkbox rendering disabled instead of hiding it

Fixes: 02b4505150
Related: https://github.com/go-gitea/gitea/pull/18527#pullrequestreview-878061913
Author-Change-Id: IB#1105104

* Update custom/conf/app.example.ini

Co-authored-by: silverwind <me@silverwind.io>

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

Co-authored-by: silverwind <me@silverwind.io>

* Mirror filter removed only when whole mirroring feature is disabled

Fixes: 02b4505150
Related: https://github.com/go-gitea/gitea/pull/18527#discussion_r883268890
Author-Change-Id: IB#1105104

Co-authored-by: silverwind <me@silverwind.io>
2022-06-04 19:42:17 +08:00
Lunny Xiao
fd7d83ace6
Move almost all functions' parameter db.Engine to context.Context (#19748)
* Move almost all functions' parameter db.Engine to context.Context
* remove some unnecessary wrap functions
2022-05-20 22:08:52 +08:00
Gusted
61f939359d
Use a better OlderThan for DeleteInactiveUsers (#19693)
* Use a better OlderThan for DeleteInactiveUsers

- Currently the OlderThan is zero, for instances that enable or run this
task this could actually delete just new users that still need to
confirm their email. This patch fixes that by setting the default to the
`ActiveCodeLives` setting, which corresponds to the amount of time that
a user can active their account, thus avoiding the issue of deleting
unactivated email users.

* Use correct duration
2022-05-13 14:49:22 +02:00
wxiaoguang
d242511e86
Remove legacy unmaintained packages, refactor to support change default locale (#19308)
Remove two unmaintained vendor packages `i18n` and `paginater`. Changes:
* Rewrite `i18n` package with a more clear fallback mechanism. Fix an unstable `Tr` behavior, add more tests.
* Refactor the legacy `Paginater` to `Paginator`, test cases are kept unchanged.

Trivial enhancement (no breaking for end users):
* Use the first locale in LANGS setting option as the default, add a log to prevent from surprising users.
2022-04-03 17:46:48 +08:00
zeripath
c88547ce71
Add Goroutine stack inspector to admin/monitor (#19207)
Continues on from #19202.

Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them. 

This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile.

The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines.

If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process. 

The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound.

In this way we should be able to more easily examine goroutines that have been stuck.

A manager command `gitea manager processes` is also provided that can export the processes (with or without stacktraces) to the command line.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-03-31 19:01:43 +02:00
KN4CK3R
1d332342db
Add Package Registry (#16510)
* Added package store settings.

* Added models.

* Added generic package registry.

* Added tests.

* Added NuGet package registry.

* Moved service index to api file.

* Added NPM package registry.

* Added Maven package registry.

* Added PyPI package registry.

* Summary is deprecated.

* Changed npm name.

* Sanitize project url.

* Allow only scoped packages.

* Added user interface.

* Changed method name.

* Added missing migration file.

* Set page info.

* Added documentation.

* Added documentation links.

* Fixed wrong error message.

* Lint template files.

* Fixed merge errors.

* Fixed unit test storage path.

* Switch to json module.

* Added suggestions.

* Added package webhook.

* Add package api.

* Fixed swagger file.

* Fixed enum and comments.

* Fixed NuGet pagination.

* Print test names.

* Added api tests.

* Fixed access level.

* Fix User unmarshal.

* Added RubyGems package registry.

* Fix lint.

* Implemented io.Writer.

* Added support for sha256/sha512 checksum files.

* Improved maven-metadata.xml support.

* Added support for symbol package uploads.

* Added tests.

* Added overview docs.

* Added npm dependencies and keywords.

* Added no-packages information.

* Display file size.

* Display asset count.

* Fixed filter alignment.

* Added package icons.

* Formatted instructions.

* Allow anonymous package downloads.

* Fixed comments.

* Fixed postgres test.

* Moved file.

* Moved models to models/packages.

* Use correct error response format per client.

* Use simpler search form.

* Fixed IsProd.

* Restructured data model.

* Prevent empty filename.

* Fix swagger.

* Implemented user/org registry.

* Implemented UI.

* Use GetUserByIDCtx.

* Use table for dependencies.

* make svg

* Added support for unscoped npm packages.

* Add support for npm dist tags.

* Added tests for npm tags.

* Unlink packages if repository gets deleted.

* Prevent user/org delete if a packages exist.

* Use package unlink in repository service.

* Added support for composer packages.

* Restructured package docs.

* Added missing tests.

* Fixed generic content page.

* Fixed docs.

* Fixed swagger.

* Added missing type.

* Fixed ambiguous column.

* Organize content store by sha256 hash.

* Added admin package management.

* Added support for sorting.

* Add support for multiple identical versions/files.

* Added missing repository unlink.

* Added file properties.

* make fmt

* lint

* Added Conan package registry.

* Updated docs.

* Unify package names.

* Added swagger enum.

* Use longer TEXT column type.

* Removed version composite key.

* Merged package and container registry.

* Removed index.

* Use dedicated package router.

* Moved files to new location.

* Updated docs.

* Fixed JOIN order.

* Fixed GROUP BY statement.

* Fixed GROUP BY #2.

* Added symbol server support.

* Added more tests.

* Set NOT NULL.

* Added setting to disable package registries.

* Moved auth into service.

* refactor

* Use ctx everywhere.

* Added package cleanup task.

* Changed packages path.

* Added container registry.

* Refactoring

* Updated comparison.

* Fix swagger.

* Fixed table order.

* Use token auth for npm routes.

* Enabled ReverseProxy auth.

* Added packages link for orgs.

* Fixed anonymous org access.

* Enable copy button for setup instructions.

* Merge error

* Added suggestions.

* Fixed merge.

* Handle "generic".

* Added link for TODO.

* Added suggestions.

* Changed temporary buffer filename.

* Added suggestions.

* Apply suggestions from code review

Co-authored-by: Thomas Boerger <thomas@webhippie.de>

* Update docs/content/doc/packages/nuget.en-us.md

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Thomas Boerger <thomas@webhippie.de>
2022-03-30 16:42:47 +08:00
zeripath
90e0a402c1
Show last cron messages on monitor page (#19223)
As discussed on #19221 we should store the results of the last task message on the
crontask and show them on the monitor page.

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-03-29 02:31:07 +01:00
Pilou
893c8938fc
New cron task: delete old system notices (#19219)
Add a new cron task which deletes the old system notices.
2022-03-28 13:54:59 +01:00
zeripath
acb9fc5f8e
Make cron task no notice on success (#19221)
Change all cron tasks to make them no notice on success default. Instead if a user
wants notices on success they need to add NOTICE_ON_SUCCESS=true instead.

 ## ⚠️ BREAKING ⚠️

This changes the cron config so that notices on success are no longer set by default
and breaks NO_SUCCESS_NOTICE settings. Instead users who want notices on success
must set NOTICE_ON_SUCCESS=true instead.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update custom/conf/app.example.ini

Co-authored-by: Norwin <noerw@users.noreply.github.com>

Co-authored-by: Norwin <noerw@users.noreply.github.com>
2022-03-26 21:13:04 +00:00
wxiaoguang
c2a3e38194
Fix the missing i18n key for update checker (#18646) 2022-02-07 15:43:53 +08:00
6543
54e9ee37a7
format with gofumpt (#18184)
* gofumpt -w -l .

* gofumpt -w -l -extra .

* Add linter

* manual fix

* change make fmt
2022-01-20 18:46:10 +01:00
Lunny Xiao
5723240490
Some repository refactors (#17950)
* some repository refactors

* remove unnecessary code

* Fix test

* Remove unnecessary banner
2021-12-12 23:48:20 +08:00
Lunny Xiao
3ca5dc7e32
Move keys to models/asymkey (#17917)
* Move keys to models/keys

* Rename models/keys -> models/asymkey

* change the missed package name

* Fix package alias

* Fix test

* Fix docs

* Fix test

* Fix test

* merge
2021-12-10 16:14:24 +08:00
Lunny Xiao
dcdb4873c8
Move repo archiver to models/repo (#17913)
* Move repo archiver to models/repo

* Move archiver service into services/repository/

* Fix imports

* Fix test

* Fix test
2021-12-06 15:19:28 +08:00
zeripath
01087e9eef
Make Requests Processes and create process hierarchy. Associate OpenRepository with context. (#17125)
This PR registers requests with the process manager and manages hierarchy within the processes.

Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-30 20:06:32 +00:00
Lunny Xiao
a666829a37
Move user related model into models/user (#17781)
* Move user related model into models/user

* Fix lint for windows

* Fix windows lint

* Fix windows lint

* Move some tests in models

* Merge
2021-11-24 17:49:20 +08:00
Lunny Xiao
c97d66d23c
Move repofiles from modules/repofiles to services/repository/files (#17774)
* Move repofiles from modules to services

* rename services/repository/repofiles -> services/repository/files

* Fix test

Co-authored-by: 6543 <6543@obermui.de>
2021-11-24 15:56:24 +08:00
zeripath
188fd2dd1a
Add PULL_LIMIT and PUSH_LIMIT to cron.update_mirror task (#17568) 2021-11-22 22:09:35 -05:00