Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015
The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.
That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.
This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.
And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.
`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.
<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">
(cherry picked from commit 33cc5837a655ad544b936d4d040ca36d74092588)
Conflicts:
assets/go-licenses.json
go.mod
go.sum
resolved with make tidy
- We were previously using `github.com/keybase/go-crypto`, because the
package for openpgp by Go itself is deprecated and no longer
maintained. This library provided a maintained version of the openpgp
package. However, it hasn't seen any activity for the last five years,
and I would therefore consider this also unmaintained.
- This patch switches the package to `github.com/ProtonMail/go-crypto`
which provides a maintained version of the openpgp package and was
already being used in the tests.
- Adds unit tests, I've carefully checked the callstacks to ensure the
OpenPGP-related code was covered under either a unit test or integration
tests to avoid regression, as this can easily turn into security
vulnerabilities if a regression happens here.
- Small behavior update, revocations are now checked correctly instead
of checking if they merely exist and the expiry time of a subkey is used
if one is provided (this is just cosmetic and doesn't impact security).
- One more dependency eliminated :D
- Update the `github.com/santhosh-tekuri/jsonschema` library from v5 to
v6.
- Update the migration loader function to a type, which is now required
in V6.
- `github.com/santhosh-tekuri/jsonschema/v6` was already used by gof3,
so removing the v5 library and using the v6 library reduces the binary
size of Forgejo.
- Before: 95912040 bytes
- After: 95706152 bytes
- Go's deadcode eliminator is quite simple, if you put a public function
in a package `aa/bb` that is used only by tests, it would still be built
if package `aa/bb` was imported. This means that if such functions use
libraries relevant only to tests that those libraries would still be
be built and increase the binary size of a Go binary.
- This is also the case with Forgejo, `models/migrations/base/tests.go`
contained functions exclusively used by tests which (skipping some steps
here) imports https://github.com/ClickHouse/clickhouse-go, which is
2MiB. The `code.gitea.io/gitea/models/migrations/base` package is
imported by `cmd/doctor` and thus the code of the clickhouse library is
also built and included in the Forgejo binary, although entirely unused
and not reachable.
- This patch moves the test-related functions to their own package, so
Go's deadcode eliminator knows not to build the test-related functions
and thus reduces the size of the Forgejo binary.
- It is not possible to move this to a `_test.go` file because Go does
not allow importing functions from such files, so any test helper
function must be in a non-test package and file.
- Reduction of size (built with `TAGS="sqlite sqlite_unlock_notify" make
build`):
- Before: 95912040 bytes (92M)
- After: 92306888 bytes (89M)
- Remove a unused dependency. This dependency was added to handle YAML
'frontmatter' meta, parsing them and converting them to a table or
details in the resulting HTML. As can be read in the issue that reported
the behavior of YAML frontmatter being rendered literally,
https://github.com/go-gitea/gitea/issues/5377.
- It's an unused dependency as the codebase since then moved on to do this YAML
parsing and rendering on their own, this was implemented in
812cfd0ad9.
- Adds unit tests that was related to this functionality, to proof the
codebase already handles this and to prevent regressions.
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| code.forgejo.org/f3/gof3/v3 | require | minor | `v3.3.1` -> `v3.4.0` |
---
### Configuration
📅 **Schedule**: Branch creation - "before 4am" (UTC), Automerge - "before 4am" (UTC).
🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MDkuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQwOS4xIiwidGFyZ2V0QnJhbmNoIjoiZm9yZ2VqbyIsImxhYmVscyI6WyJkZXBlbmRlbmN5LXVwZ3JhZGUiXX0=-->
Co-authored-by: Twenty Panda <twenty-panda@posteo.com>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4196
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Renovate Bot <forgejo-renovate-action@forgejo.org>
Co-committed-by: Renovate Bot <forgejo-renovate-action@forgejo.org>
replace reply with forgejos forked version
If plain text is selected as the message format in e.g. Apple Mail, the inline attachments are no longer at the end of the mail, but instead directly where they are in the mail. When parsing the mail, these inline attachments are replaced by "--". The new reply version no longer cuts the text at the first "--".
Tests for this are present in reply (7dc5750c6d).
Fixes https://codeberg.org/forgejo/forgejo/issues/3496#issuecomment-1798416
---
Additionally, I reduced the allocations for the inline attachments.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3747
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Beowulf <beowulf@beocode.eu>
Co-committed-by: Beowulf <beowulf@beocode.eu>
- It was only used to parse old U2F data to webauthn credentials. We
only used the public key and keyhandle. This functiontionality was
reworked to `parseU2FRegistration`.
- Tests are already present, `Test_RemigrateU2FCredentials`.
- Remove `gitea.com/lunny/dingtalk_webhook` as dependency, we only use
two structs which are small enough to be recreated in Forgejo and don't
need to rely on the dependency.
- Existing tests (thanks @oliverpool) prove that this has no effect.
- Update github.com/bufbuild/connect-go to
https://github.com/connectrpc/connect-go.
- This is a fork that's actively maintained and is recommend by the
original library. Looking at the recent release notes, it looks like
going in the right direction what one would expect of a library, no
strange features being added, lots of improvements.
- There's still an indirect dependency by
`code.gitea.io/actions-proto-go` on a old version of `connect-go`.
- The [rupture](https://github.com/ethantkoenig/rupture) dependency was
essentially outdated in the sense it was using old version of
dependencies.
- The usage by Forgejo was rather a small portion, so that portion is
now vendored (with its tests).
- Removes old dependencies from go.sum (less dependencies is better for
reviewing what the heck we're importing). Just to note that they were
likely not being used by Go's build process (according to
https://go.dev/ref/mod#minimal-version-selection), so it's really a
matter of formal cleaning up dependencies we don't use and therefor
don't want to download and be in our go.sum.
(cherry picked from commit aa72a5f009)
Conflicts:
go.sum
https://codeberg.org/forgejo/forgejo/pulls/2148
(cherry picked from commit fbe8d65f0b)
(cherry picked from commit e18debcb6a)
Conflicts:
go.sum
https://codeberg.org/forgejo/forgejo/pulls/2245
(cherry picked from commit 8c43c2ada8)
Update golang.org/x/crypto for CVE-2023-48795 and update other packages.
`go-git` is not updated because it needs time to figure out why some
tests fail.
Replace `github.com/gogs/cron` with `github.com/go-co-op/gocron` as the
former package is not maintained for many years.
---------
Co-authored-by: delvh <dev.lh@web.de>
Replace #10912
And there are many new tests to cover the CLI behavior
There were some concerns about the "option order in hook scripts"
(https://github.com/go-gitea/gitea/pull/10912#issuecomment-1137543314),
it's not a problem now. Because the hook script uses `/gitea hook
--config=/app.ini pre-receive` format. The "config" is a global option,
it can appear anywhere.
----
## ⚠️ BREAKING ⚠️
This PR does it best to avoid breaking anything. The major changes are:
* `gitea` itself won't accept web's options: `--install-port` / `--pid`
/ `--port` / `--quiet` / `--verbose` .... They are `web` sub-command's
options.
* Use `./gitea web --pid ....` instead
* `./gitea` can still run the `web` sub-command as shorthand, with
default options
* The sub-command's options must follow the sub-command
* Before: `./gitea --sub-opt subcmd` might equal to `./gitea subcmd
--sub-opt` (well, might not ...)
* After: only `./gitea subcmd --sub-opt` could be used
* The global options like `--config` are not affected
The package `github.com/nfnt/resize` is deprecated and archived by the
author. `github.com/oliamb/cutter` is not maintained since 2018. We
could use `golang.org/x/image/draw` instead.
Bumping `github.com/golang-jwt/jwt` from v4 to v5.
`github.com/golang-jwt/jwt` v5 is bringing some breaking changes:
- standard `Valid()` method on claims is removed. It's replaced by
`ClaimsValidator` interface implementing `Validator()` method instead,
which is called after standard validation. Gitea doesn't seem to be
using this logic.
- `jwt.Token` has a field `Valid`, so it's checked in `ParseToken`
function in `services/auth/source/oauth2/token.go`
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Co-authored-by: @awkwardbunny
This PR adds a Debian package registry. You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing. Source packages are not supported at the
moment and I did not find documentation of the architecture "all" and
how these packages should be treated.
---------
Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
This will ensure that the file always has a final newline. I'm not sure
where this bug with inconsistent final newline actually comes from, it
is likely Windows-related.
---------
Co-authored-by: delvh <dev.lh@web.de>
