githaven/services/cron/tasks_basic.go

176 lines
4.9 KiB
Go
Raw Normal View History

// Copyright 2020 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package cron
import (
"context"
"time"
"code.gitea.io/gitea/models"
git_model "code.gitea.io/gitea/models/git"
user_model "code.gitea.io/gitea/models/user"
"code.gitea.io/gitea/models/webhook"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/setting"
Refactor: Move login out of models (#16199) `models` does far too much. In particular it handles all `UserSignin`. It shouldn't be responsible for calling LDAP, SMTP or PAM for signing in. Therefore we should move this code out of `models`. This code has to depend on `models` - therefore it belongs in `services`. There is a package in `services` called `auth` and clearly this functionality belongs in there. Plan: - [x] Change `auth.Auth` to `auth.Method` - as they represent methods of authentication. - [x] Move `models.UserSignIn` into `auth` - [x] Move `models.ExternalUserLogin` - [x] Move most of the `LoginVia*` methods to `auth` or subpackages - [x] Move Resynchronize functionality to `auth` - Involved some restructuring of `models/ssh_key.go` to reduce the size of this massive file and simplify its files. - [x] Move the rest of the LDAP functionality in to the ldap subpackage - [x] Re-factor the login sources to express an interfaces `auth.Source`? - I've done this through some smaller interfaces Authenticator and Synchronizable - which would allow us to extend things in future - [x] Now LDAP is out of models - need to think about modules/auth/ldap and I think all of that functionality might just be moveable - [x] Similarly a lot Oauth2 functionality need not be in models too and should be moved to services/auth/source/oauth2 - [x] modules/auth/oauth2/oauth2.go uses xorm... This is naughty - probably need to move this into models. - [x] models/oauth2.go - mostly should be in modules/auth/oauth2 or services/auth/source/oauth2 - [x] More simplifications of login_source.go may need to be done - Allow wiring in of notify registration - *this can now easily be done - but I think we should do it in another PR* - see #16178 - More refactors...? - OpenID should probably become an auth Method but I think that can be left for another PR - Methods should also probably be cleaned up - again another PR I think. - SSPI still needs more refactors.* Rename auth.Auth auth.Method * Restructure ssh_key.go - move functions from models/user.go that relate to ssh_key to ssh_key - split ssh_key.go to try create clearer function domains for allow for future refactors here. Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-24 10:16:34 +00:00
"code.gitea.io/gitea/services/auth"
"code.gitea.io/gitea/services/migrations"
mirror_service "code.gitea.io/gitea/services/mirror"
packages_cleanup_service "code.gitea.io/gitea/services/packages/cleanup"
repo_service "code.gitea.io/gitea/services/repository"
archiver_service "code.gitea.io/gitea/services/repository/archiver"
)
func registerUpdateMirrorTask() {
type UpdateMirrorTaskConfig struct {
BaseConfig
PullLimit int
PushLimit int
}
RegisterTaskFatal("update_mirrors", &UpdateMirrorTaskConfig{
BaseConfig: BaseConfig{
Enabled: true,
RunAtStart: false,
Schedule: "@every 10m",
},
PullLimit: 50,
PushLimit: 50,
}, func(ctx context.Context, _ *user_model.User, cfg Config) error {
umtc := cfg.(*UpdateMirrorTaskConfig)
return mirror_service.Update(ctx, umtc.PullLimit, umtc.PushLimit)
})
}
func registerRepoHealthCheck() {
type RepoHealthCheckConfig struct {
BaseConfig
Timeout time.Duration
Args []string `delim:" "`
}
RegisterTaskFatal("repo_health_check", &RepoHealthCheckConfig{
BaseConfig: BaseConfig{
Enabled: true,
RunAtStart: false,
Schedule: "@midnight",
},
Timeout: 60 * time.Second,
Args: []string{},
}, func(ctx context.Context, _ *user_model.User, config Config) error {
rhcConfig := config.(*RepoHealthCheckConfig)
// the git args are set by config, they can be safe to be trusted
Refactor git command package to improve security and maintainability (#22678) This PR follows #21535 (and replace #22592) ## Review without space diff https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1 ## Purpose of this PR 1. Make git module command completely safe (risky user inputs won't be passed as argument option anymore) 2. Avoid low-level mistakes like https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918 3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg` type 4. Simplify code when using git command ## The main idea of this PR * Move the `git.CmdArg` to the `internal` package, then no other package except `git` could use it. Then developers could never do `AddArguments(git.CmdArg(userInput))` any more. * Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already trusted arguments. It's only used in a few cases, for example: use git arguments from config file, help unit test with some arguments. * Introduce `AddOptionValues` and `AddOptionFormat`, they make code more clear and simple: * Before: `AddArguments("-m").AddDynamicArguments(message)` * After: `AddOptionValues("-m", message)` * - * Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'", sig.Name, sig.Email)))` * After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)` ## FAQ ### Why these changes were not done in #21535 ? #21535 is mainly a search&replace, it did its best to not change too much logic. Making the framework better needs a lot of changes, so this separate PR is needed as the second step. ### The naming of `AddOptionXxx` According to git's manual, the `--xxx` part is called `option`. ### How can it guarantee that `internal.CmdArg` won't be not misused? Go's specification guarantees that. Trying to access other package's internal package causes compilation error. And, `golangci-lint` also denies the git/internal package. Only the `git/command.go` can use it carefully. ### There is still a `ToTrustedCmdArgs`, will it still allow developers to make mistakes and pass untrusted arguments? Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code will be very complex (see the changes for examples). Then developers and reviewers can know that something might be unreasonable. ### Why there was a `CmdArgCheck` and why it's removed? At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck` was introduced as a hacky patch. Now, almost all code could be written as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for `CmdArgCheck` anymore. ### Why many codes for `signArg == ""` is deleted? Because in the old code, `signArg` could never be empty string, it's either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just dead code. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-04 02:30:43 +00:00
return repo_service.GitFsckRepos(ctx, rhcConfig.Timeout, git.ToTrustedCmdArgs(rhcConfig.Args))
})
}
func registerCheckRepoStats() {
RegisterTaskFatal("check_repo_stats", &BaseConfig{
Enabled: true,
RunAtStart: true,
Schedule: "@midnight",
}, func(ctx context.Context, _ *user_model.User, _ Config) error {
return models.CheckRepoStats(ctx)
})
}
func registerArchiveCleanup() {
RegisterTaskFatal("archive_cleanup", &OlderThanConfig{
BaseConfig: BaseConfig{
Enabled: true,
RunAtStart: true,
Schedule: "@midnight",
},
OlderThan: 24 * time.Hour,
}, func(ctx context.Context, _ *user_model.User, config Config) error {
acConfig := config.(*OlderThanConfig)
return archiver_service.DeleteOldRepositoryArchives(ctx, acConfig.OlderThan)
})
}
func registerSyncExternalUsers() {
RegisterTaskFatal("sync_external_users", &UpdateExistingConfig{
BaseConfig: BaseConfig{
Enabled: true,
RunAtStart: false,
Schedule: "@midnight",
},
UpdateExisting: true,
}, func(ctx context.Context, _ *user_model.User, config Config) error {
realConfig := config.(*UpdateExistingConfig)
Refactor: Move login out of models (#16199) `models` does far too much. In particular it handles all `UserSignin`. It shouldn't be responsible for calling LDAP, SMTP or PAM for signing in. Therefore we should move this code out of `models`. This code has to depend on `models` - therefore it belongs in `services`. There is a package in `services` called `auth` and clearly this functionality belongs in there. Plan: - [x] Change `auth.Auth` to `auth.Method` - as they represent methods of authentication. - [x] Move `models.UserSignIn` into `auth` - [x] Move `models.ExternalUserLogin` - [x] Move most of the `LoginVia*` methods to `auth` or subpackages - [x] Move Resynchronize functionality to `auth` - Involved some restructuring of `models/ssh_key.go` to reduce the size of this massive file and simplify its files. - [x] Move the rest of the LDAP functionality in to the ldap subpackage - [x] Re-factor the login sources to express an interfaces `auth.Source`? - I've done this through some smaller interfaces Authenticator and Synchronizable - which would allow us to extend things in future - [x] Now LDAP is out of models - need to think about modules/auth/ldap and I think all of that functionality might just be moveable - [x] Similarly a lot Oauth2 functionality need not be in models too and should be moved to services/auth/source/oauth2 - [x] modules/auth/oauth2/oauth2.go uses xorm... This is naughty - probably need to move this into models. - [x] models/oauth2.go - mostly should be in modules/auth/oauth2 or services/auth/source/oauth2 - [x] More simplifications of login_source.go may need to be done - Allow wiring in of notify registration - *this can now easily be done - but I think we should do it in another PR* - see #16178 - More refactors...? - OpenID should probably become an auth Method but I think that can be left for another PR - Methods should also probably be cleaned up - again another PR I think. - SSPI still needs more refactors.* Rename auth.Auth auth.Method * Restructure ssh_key.go - move functions from models/user.go that relate to ssh_key to ssh_key - split ssh_key.go to try create clearer function domains for allow for future refactors here. Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-24 10:16:34 +00:00
return auth.SyncExternalUsers(ctx, realConfig.UpdateExisting)
})
}
func registerDeletedBranchesCleanup() {
RegisterTaskFatal("deleted_branches_cleanup", &OlderThanConfig{
BaseConfig: BaseConfig{
Enabled: true,
RunAtStart: true,
Schedule: "@midnight",
},
OlderThan: 24 * time.Hour,
}, func(ctx context.Context, _ *user_model.User, config Config) error {
realConfig := config.(*OlderThanConfig)
git_model.RemoveOldDeletedBranches(ctx, realConfig.OlderThan)
return nil
})
}
func registerUpdateMigrationPosterID() {
RegisterTaskFatal("update_migration_poster_id", &BaseConfig{
Enabled: true,
RunAtStart: true,
Schedule: "@midnight",
}, func(ctx context.Context, _ *user_model.User, _ Config) error {
return migrations.UpdateMigrationPosterID(ctx)
})
}
func registerCleanupHookTaskTable() {
RegisterTaskFatal("cleanup_hook_task_table", &CleanupHookTaskConfig{
BaseConfig: BaseConfig{
Enabled: true,
RunAtStart: false,
Schedule: "@midnight",
},
CleanupType: "OlderThan",
OlderThan: 168 * time.Hour,
NumberToKeep: 10,
}, func(ctx context.Context, _ *user_model.User, config Config) error {
realConfig := config.(*CleanupHookTaskConfig)
return webhook.CleanupHookTaskTable(ctx, webhook.ToHookTaskCleanupType(realConfig.CleanupType), realConfig.OlderThan, realConfig.NumberToKeep)
})
}
Add Package Registry (#16510) * Added package store settings. * Added models. * Added generic package registry. * Added tests. * Added NuGet package registry. * Moved service index to api file. * Added NPM package registry. * Added Maven package registry. * Added PyPI package registry. * Summary is deprecated. * Changed npm name. * Sanitize project url. * Allow only scoped packages. * Added user interface. * Changed method name. * Added missing migration file. * Set page info. * Added documentation. * Added documentation links. * Fixed wrong error message. * Lint template files. * Fixed merge errors. * Fixed unit test storage path. * Switch to json module. * Added suggestions. * Added package webhook. * Add package api. * Fixed swagger file. * Fixed enum and comments. * Fixed NuGet pagination. * Print test names. * Added api tests. * Fixed access level. * Fix User unmarshal. * Added RubyGems package registry. * Fix lint. * Implemented io.Writer. * Added support for sha256/sha512 checksum files. * Improved maven-metadata.xml support. * Added support for symbol package uploads. * Added tests. * Added overview docs. * Added npm dependencies and keywords. * Added no-packages information. * Display file size. * Display asset count. * Fixed filter alignment. * Added package icons. * Formatted instructions. * Allow anonymous package downloads. * Fixed comments. * Fixed postgres test. * Moved file. * Moved models to models/packages. * Use correct error response format per client. * Use simpler search form. * Fixed IsProd. * Restructured data model. * Prevent empty filename. * Fix swagger. * Implemented user/org registry. * Implemented UI. * Use GetUserByIDCtx. * Use table for dependencies. * make svg * Added support for unscoped npm packages. * Add support for npm dist tags. * Added tests for npm tags. * Unlink packages if repository gets deleted. * Prevent user/org delete if a packages exist. * Use package unlink in repository service. * Added support for composer packages. * Restructured package docs. * Added missing tests. * Fixed generic content page. * Fixed docs. * Fixed swagger. * Added missing type. * Fixed ambiguous column. * Organize content store by sha256 hash. * Added admin package management. * Added support for sorting. * Add support for multiple identical versions/files. * Added missing repository unlink. * Added file properties. * make fmt * lint * Added Conan package registry. * Updated docs. * Unify package names. * Added swagger enum. * Use longer TEXT column type. * Removed version composite key. * Merged package and container registry. * Removed index. * Use dedicated package router. * Moved files to new location. * Updated docs. * Fixed JOIN order. * Fixed GROUP BY statement. * Fixed GROUP BY #2. * Added symbol server support. * Added more tests. * Set NOT NULL. * Added setting to disable package registries. * Moved auth into service. * refactor * Use ctx everywhere. * Added package cleanup task. * Changed packages path. * Added container registry. * Refactoring * Updated comparison. * Fix swagger. * Fixed table order. * Use token auth for npm routes. * Enabled ReverseProxy auth. * Added packages link for orgs. * Fixed anonymous org access. * Enable copy button for setup instructions. * Merge error * Added suggestions. * Fixed merge. * Handle "generic". * Added link for TODO. * Added suggestions. * Changed temporary buffer filename. * Added suggestions. * Apply suggestions from code review Co-authored-by: Thomas Boerger <thomas@webhippie.de> * Update docs/content/doc/packages/nuget.en-us.md Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Thomas Boerger <thomas@webhippie.de>
2022-03-30 08:42:47 +00:00
func registerCleanupPackages() {
RegisterTaskFatal("cleanup_packages", &OlderThanConfig{
BaseConfig: BaseConfig{
Enabled: true,
RunAtStart: true,
Schedule: "@midnight",
},
OlderThan: 24 * time.Hour,
}, func(ctx context.Context, _ *user_model.User, config Config) error {
realConfig := config.(*OlderThanConfig)
return packages_cleanup_service.Cleanup(ctx, realConfig.OlderThan)
Add Package Registry (#16510) * Added package store settings. * Added models. * Added generic package registry. * Added tests. * Added NuGet package registry. * Moved service index to api file. * Added NPM package registry. * Added Maven package registry. * Added PyPI package registry. * Summary is deprecated. * Changed npm name. * Sanitize project url. * Allow only scoped packages. * Added user interface. * Changed method name. * Added missing migration file. * Set page info. * Added documentation. * Added documentation links. * Fixed wrong error message. * Lint template files. * Fixed merge errors. * Fixed unit test storage path. * Switch to json module. * Added suggestions. * Added package webhook. * Add package api. * Fixed swagger file. * Fixed enum and comments. * Fixed NuGet pagination. * Print test names. * Added api tests. * Fixed access level. * Fix User unmarshal. * Added RubyGems package registry. * Fix lint. * Implemented io.Writer. * Added support for sha256/sha512 checksum files. * Improved maven-metadata.xml support. * Added support for symbol package uploads. * Added tests. * Added overview docs. * Added npm dependencies and keywords. * Added no-packages information. * Display file size. * Display asset count. * Fixed filter alignment. * Added package icons. * Formatted instructions. * Allow anonymous package downloads. * Fixed comments. * Fixed postgres test. * Moved file. * Moved models to models/packages. * Use correct error response format per client. * Use simpler search form. * Fixed IsProd. * Restructured data model. * Prevent empty filename. * Fix swagger. * Implemented user/org registry. * Implemented UI. * Use GetUserByIDCtx. * Use table for dependencies. * make svg * Added support for unscoped npm packages. * Add support for npm dist tags. * Added tests for npm tags. * Unlink packages if repository gets deleted. * Prevent user/org delete if a packages exist. * Use package unlink in repository service. * Added support for composer packages. * Restructured package docs. * Added missing tests. * Fixed generic content page. * Fixed docs. * Fixed swagger. * Added missing type. * Fixed ambiguous column. * Organize content store by sha256 hash. * Added admin package management. * Added support for sorting. * Add support for multiple identical versions/files. * Added missing repository unlink. * Added file properties. * make fmt * lint * Added Conan package registry. * Updated docs. * Unify package names. * Added swagger enum. * Use longer TEXT column type. * Removed version composite key. * Merged package and container registry. * Removed index. * Use dedicated package router. * Moved files to new location. * Updated docs. * Fixed JOIN order. * Fixed GROUP BY statement. * Fixed GROUP BY #2. * Added symbol server support. * Added more tests. * Set NOT NULL. * Added setting to disable package registries. * Moved auth into service. * refactor * Use ctx everywhere. * Added package cleanup task. * Changed packages path. * Added container registry. * Refactoring * Updated comparison. * Fix swagger. * Fixed table order. * Use token auth for npm routes. * Enabled ReverseProxy auth. * Added packages link for orgs. * Fixed anonymous org access. * Enable copy button for setup instructions. * Merge error * Added suggestions. * Fixed merge. * Handle "generic". * Added link for TODO. * Added suggestions. * Changed temporary buffer filename. * Added suggestions. * Apply suggestions from code review Co-authored-by: Thomas Boerger <thomas@webhippie.de> * Update docs/content/doc/packages/nuget.en-us.md Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Thomas Boerger <thomas@webhippie.de>
2022-03-30 08:42:47 +00:00
})
}
func initBasicTasks() {
if setting.Mirror.Enabled {
registerUpdateMirrorTask()
}
registerRepoHealthCheck()
registerCheckRepoStats()
registerArchiveCleanup()
registerSyncExternalUsers()
registerDeletedBranchesCleanup()
if !setting.Repository.DisableMigrations {
registerUpdateMigrationPosterID()
}
registerCleanupHookTaskTable()
Add Package Registry (#16510) * Added package store settings. * Added models. * Added generic package registry. * Added tests. * Added NuGet package registry. * Moved service index to api file. * Added NPM package registry. * Added Maven package registry. * Added PyPI package registry. * Summary is deprecated. * Changed npm name. * Sanitize project url. * Allow only scoped packages. * Added user interface. * Changed method name. * Added missing migration file. * Set page info. * Added documentation. * Added documentation links. * Fixed wrong error message. * Lint template files. * Fixed merge errors. * Fixed unit test storage path. * Switch to json module. * Added suggestions. * Added package webhook. * Add package api. * Fixed swagger file. * Fixed enum and comments. * Fixed NuGet pagination. * Print test names. * Added api tests. * Fixed access level. * Fix User unmarshal. * Added RubyGems package registry. * Fix lint. * Implemented io.Writer. * Added support for sha256/sha512 checksum files. * Improved maven-metadata.xml support. * Added support for symbol package uploads. * Added tests. * Added overview docs. * Added npm dependencies and keywords. * Added no-packages information. * Display file size. * Display asset count. * Fixed filter alignment. * Added package icons. * Formatted instructions. * Allow anonymous package downloads. * Fixed comments. * Fixed postgres test. * Moved file. * Moved models to models/packages. * Use correct error response format per client. * Use simpler search form. * Fixed IsProd. * Restructured data model. * Prevent empty filename. * Fix swagger. * Implemented user/org registry. * Implemented UI. * Use GetUserByIDCtx. * Use table for dependencies. * make svg * Added support for unscoped npm packages. * Add support for npm dist tags. * Added tests for npm tags. * Unlink packages if repository gets deleted. * Prevent user/org delete if a packages exist. * Use package unlink in repository service. * Added support for composer packages. * Restructured package docs. * Added missing tests. * Fixed generic content page. * Fixed docs. * Fixed swagger. * Added missing type. * Fixed ambiguous column. * Organize content store by sha256 hash. * Added admin package management. * Added support for sorting. * Add support for multiple identical versions/files. * Added missing repository unlink. * Added file properties. * make fmt * lint * Added Conan package registry. * Updated docs. * Unify package names. * Added swagger enum. * Use longer TEXT column type. * Removed version composite key. * Merged package and container registry. * Removed index. * Use dedicated package router. * Moved files to new location. * Updated docs. * Fixed JOIN order. * Fixed GROUP BY statement. * Fixed GROUP BY #2. * Added symbol server support. * Added more tests. * Set NOT NULL. * Added setting to disable package registries. * Moved auth into service. * refactor * Use ctx everywhere. * Added package cleanup task. * Changed packages path. * Added container registry. * Refactoring * Updated comparison. * Fix swagger. * Fixed table order. * Use token auth for npm routes. * Enabled ReverseProxy auth. * Added packages link for orgs. * Fixed anonymous org access. * Enable copy button for setup instructions. * Merge error * Added suggestions. * Fixed merge. * Handle "generic". * Added link for TODO. * Added suggestions. * Changed temporary buffer filename. * Added suggestions. * Apply suggestions from code review Co-authored-by: Thomas Boerger <thomas@webhippie.de> * Update docs/content/doc/packages/nuget.en-us.md Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Thomas Boerger <thomas@webhippie.de>
2022-03-30 08:42:47 +00:00
if setting.Packages.Enabled {
registerCleanupPackages()
}
}