Commit Graph

23 Commits

Author SHA1 Message Date
zeripath 7aa29720f0
Improve SyncMirrors logging (#19045) (#19050)
Backport #19045

Yet another issue has come up where the logging from SyncMirrors does not provide
enough context. This PR adds more context to these logging events.

Related #19038

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-03-10 16:06:35 +01:00
wxiaoguang 385dc6a992
Allow admin to associate missing LFS objects for repositories (#18143)
This PR reworked the Find pointer files feature in Settings -> LFS page.

When a LFS object is missing from database but exists in LFS content store, admin can associate it to the repository by clicking the Associate button.

This PR is not perfect (because the LFS module itself should be improved too), it's just a nice-to-have feature to help users recover their LFS repositories (eg: database was lost / table was truncated)
2022-01-01 17:05:31 +08:00
Lunny Xiao 719bddcd76
Move repository model into models/repo (#17933)
* Some refactors related repository model

* Move more methods out of repository

* Move repository into models/repo

* Fix test

* Fix test

* some improvements

* Remove unnecessary function
2021-12-10 09:27:50 +08:00
Lunny Xiao a666829a37
Move user related model into models/user (#17781)
* Move user related model into models/user

* Fix lint for windows

* Fix windows lint

* Fix windows lint

* Move some tests in models

* Merge
2021-11-24 17:49:20 +08:00
Lunny Xiao d710af6669
Remove NewSession method from db.Engine interface (#17577)
* Remove NewSession method from db.Engine interface

* Fix bug

* Some improvements

* Fix bug

* Fix test

* Use XXXBean instead of XXXExample
2021-11-21 23:41:00 +08:00
zeripath 9302eba971
DBContext is just a Context (#17100)
* DBContext is just a Context

This PR removes some of the specialness from the DBContext and makes it context
This allows us to simplify the GetEngine code to wrap around any context in future
and means that we can change our loadRepo(e Engine) functions to simply take contexts.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix unit tests

Signed-off-by: Andrew Thornton <art27@cantab.net>

* another place that needs to set the initial context

Signed-off-by: Andrew Thornton <art27@cantab.net>

* avoid race

Signed-off-by: Andrew Thornton <art27@cantab.net>

* change attachment error

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-09-23 23:45:36 +08:00
Lunny Xiao a4bfef265d
Move db related basic functions to models/db (#17075)
* Move db related basic functions to models/db

* Fix lint

* Fix lint

* Fix test

* Fix lint

* Fix lint

* revert unnecessary change

* Fix test

* Fix wrong replace string

* Use *Context

* Correct committer spelling and fix wrong replaced words

Co-authored-by: zeripath <art27@cantab.net>
2021-09-19 19:49:59 +08:00
6543 9c4601bdf8
Code Formats, Nits & Unused Func/Var deletions (#15286)
* _ to unused func options

* rm useless brakets

* rm trifial non used models functions

* rm dead code

* rm dead global vars

* fix routers/api/v1/repo/issue.go

* dont overload import module
2021-04-09 09:40:34 +02:00
KN4CK3R c03e488e14
Add LFS Migration and Mirror (#14726)
* Implemented LFS client.

* Implemented scanning for pointer files.

* Implemented downloading of lfs files.

* Moved model-dependent code into services.

* Removed models dependency. Added TryReadPointerFromBuffer.

* Migrated code from service to module.

* Centralised storage creation.

* Removed dependency from models.

* Moved ContentStore into modules.

* Share structs between server and client.

* Moved method to services.

* Implemented lfs download on clone.

* Implemented LFS sync on clone and mirror update.

* Added form fields.

* Updated templates.

* Fixed condition.

* Use alternate endpoint.

* Added missing methods.

* Fixed typo and make linter happy.

* Detached pointer parser from gogit dependency.

* Fixed TestGetLFSRange test.

* Added context to support cancellation.

* Use ReadFull to probably read more data.

* Removed duplicated code from models.

* Moved scan implementation into pointer_scanner_nogogit.

* Changed method name.

* Added comments.

* Added more/specific log/error messages.

* Embedded lfs.Pointer into models.LFSMetaObject.

* Moved code from models to module.

* Moved code from models to module.

* Moved code from models to module.

* Reduced pointer usage.

* Embedded type.

* Use promoted fields.

* Fixed unexpected eof.

* Added unit tests.

* Implemented migration of local file paths.

* Show an error on invalid LFS endpoints.

* Hide settings if not used.

* Added LFS info to mirror struct.

* Fixed comment.

* Check LFS endpoint.

* Manage LFS settings from mirror page.

* Fixed selector.

* Adjusted selector.

* Added more tests.

* Added local filesystem migration test.

* Fixed typo.

* Reset settings.

* Added special windows path handling.

* Added unit test for HTTPClient.

* Added unit test for BasicTransferAdapter.

* Moved into util package.

* Test if LFS endpoint is allowed.

* Added support for git://

* Just use a static placeholder as the displayed url may be invalid.

* Reverted to original code.

* Added "Advanced Settings".

* Updated wording.

* Added discovery info link.

* Implemented suggestion.

* Fixed missing format parameter.

* Added Pointer.IsValid().

* Always remove model on error.

* Added suggestions.

* Use channel instead of array.

* Update routers/repo/migrate.go

* fmt

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
2021-04-08 18:25:57 -04:00
Bo-Yi Wu 167b0f46ef
chore(models): rewrite code format. (#14754)
* chore: rewrite format.

* chore: update format

Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>

* chore: update format

Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>

* chore: Adjacent parameters with the same type should be grouped together

* chore: update format.
2021-03-14 19:52:12 +01:00
Lunny Xiao 7a5465fc56
LFS support to be stored on minio (#12518)
* LFS support to be stored on minio

* Fix test

* Fix lint

* Fix lint

* Fix check

* Fix test

* Update documents and add migration for LFS

* Fix some bugs
2020-09-08 23:45:10 +08:00
John Olheiser baadb51445
Add gitea-vet (#10948)
* Add copyright

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add gitea-vet and fix non-compliance

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Combine tools.go into build.go and clean up

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Remove extra GO111MODULE=on

Signed-off-by: jolheiser <john.olheiser@gmail.com>
2020-04-05 07:20:50 +01:00
Manush Dodunekov 1751d5fcf2 Restricted users (#6274)
* Restricted users (#4334): initial implementation

* Add User.IsRestricted & UI to edit it

* Pass user object instead of user id to places where IsRestricted flag matters

* Restricted users: maintain access rows for all referenced repos (incl public)

* Take logged in user & IsRestricted flag into account in org/repo listings, searches and accesses

* Add basic repo access tests for restricted users

Signed-off-by: Manush Dodunekov <manush@stendahls.se>

* Mention restricted users in the faq

Signed-off-by: Manush Dodunekov <manush@stendahls.se>

* Revert unnecessary change `.isUserPartOfOrg` -> `.IsUserPartOfOrg`

Signed-off-by: Manush Dodunekov <manush@stendahls.se>

* Remove unnecessary `org.IsOrganization()` call

Signed-off-by: Manush Dodunekov <manush@stendahls.se>

* Revert to an `int64` keyed `accessMap`

* Add type `userAccess`
* Add convenience func updateUserAccess()
* Turn accessMap into a `map[int64]userAccess`

Signed-off-by: Manush Dodunekov <manush@stendahls.se>

* or even better: `map[int64]*userAccess`

* updateUserAccess(): use tighter syntax as suggested by lafriks

* even tighter

* Avoid extra loop

* Don't disclose limited orgs to unauthenticated users

* Don't assume block only applies to orgs

* Use an array of `VisibleType` for filtering

* fix yet another thinko

* Ok - no need for u

* Revert "Ok - no need for u"

This reverts commit 5c3e886aab.

Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-01-13 18:33:46 +01:00
zeripath 5e6a008fba
Add basic repository lfs management (#7199)
This PR adds basic repository LFS management UI including the ability to find all possible pointers within the repository. Locks are not managed at present but would be addable through some simple additions.

* Add basic repository lfs management
* add auto-associate function
* Add functionality to find commits with this lfs file
* Add link to find commits on the lfs file view
* Adjust commit view to state the likely branch causing the commit
* Only read Oid from database
2019-10-28 18:31:55 +00:00
Lunny Xiao 85202d4784
Display ui time with customize time location (#7792)
* display ui time with customize time location

* fix lint

* rename UILocation to DefaultUILocation

* move time related functions to modules/timeutil

* fix tests

* fix tests

* fix build

* fix swagger
2019-08-15 22:46:21 +08:00
zeripath 296814e887 Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702)
* Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile

* Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview

* Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile

* Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles

* Move branch checkout functions out of repo_editor.go as they are no longer used there

* BUGFIX: The default permissions should be 100644

    This is a change from the previous code but is more in keeping
    with the default behaviour of git.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Standardise cleanUploadFilename to more closely match git

See verify_path in: 7f4e641693/read-cache.c (L951)

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Redirect on bad paths

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Refactor to move the uploading functions out to a module

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add LFS support

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update upload.go attribution header

Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution.

* Delete upload files after session committed

* Ensure that GIT_AUTHOR_NAME etc. are valid for git

see #5774

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add in test cases per @lafriks comment

* Add space between gitea and github imports

Signed-off-by: Andrew Thornton <art27@cantab.net>

* more examples in TestCleanUploadName

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix formatting

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Set the SSH_ORIGINAL_COMMAND to ensure hooks are run

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Switch off SSH_ORIGINAL_COMMAND

Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 15:07:31 +02:00
zeripath 7d434376f1 Pooled and buffered gzip implementation (#5722)
* Pooled and buffered gzip implementation

* Add test for gzip

* Add integration test

* Ensure lfs check within transaction

The previous code made it possible for a race condition to occur whereby a LFSMetaObject could be checked into the database twice. We should check if the LFSMetaObject is within the database and insert it if not in one transaction.

* Try to avoid primary key problem in postgres

The integration tests are being affected by
https://github.com/go-testfixtures/testfixtures/issues/39 if we set the
primary key high enough, keep a count of this and remove at the end of
each test we shouldn't be affected by this.
2019-01-23 10:56:51 +02:00
Lunny Xiao f2e20c81b6 Refactor struct's time to remove unnecessary memory usage (#3142)
* refactor struct's time to remove unnecessary memory usage

* use AsTimePtr simple code

* fix tests

* fix time compare

* fix template on gpg

* use AddDuration instead of Add
2017-12-11 06:37:04 +02:00
Lauris BH 82fc50f4ee Fix Git LFS object/repo link storage in database and small refactoring (#2803) 2017-10-30 20:11:56 +08:00
Lunny Xiao a8717e5e3a Use AfterLoad instead of AfterSet on Structs (#2628)
* use AfterLoad instead of AfterSet on Structs

* fix the comments on AfterLoad

* fix the comments on action AfterLoad
2017-10-01 19:52:35 +03:00
Lunny Xiao 005900baea Use created & updated instead BeforeInsert & BeforeUpdate (#2482)
* use created & updated instead BeforeInsert & BeforeUpdate

* fix vendor checksum

* only show generated SQL when development mode

* remove extra update column updated_unix

* remove trace config
2017-09-13 08:18:22 +03:00
Lauris BH 6db387a21e Refactor session close as xorm already does everything needed internally (#2020) 2017-06-21 08:57:05 +08:00
Fabian Zaremba 2e7ccecfe6 Git LFS support v2 (#122)
* Import github.com/git-lfs/lfs-test-server as lfs module base

Imported commit is 3968aac269a77b73924649b9412ae03f7ccd3198

Removed:

Dockerfile CONTRIBUTING.md mgmt* script/ vendor/ kvlogger.go
.dockerignore .gitignore README.md

* Remove config, add JWT support from github.com/mgit-at/lfs-test-server

Imported commit f0cdcc5a01599c5a955dc1bbf683bb4acecdba83

* Add LFS settings

* Add LFS meta object model

* Add LFS routes and initialization

* Import github.com/dgrijalva/jwt-go into vendor/

* Adapt LFS module: handlers, routing, meta store

* Move LFS routes to /user/repo/info/lfs/*

* Add request header checks to LFS BatchHandler / PostHandler

* Implement LFS basic authentication

* Rework JWT secret generation / load

* Implement LFS SSH token authentication with JWT

Specification: https://github.com/github/git-lfs/tree/master/docs/api

* Integrate LFS settings into install process

* Remove LFS objects when repository is deleted

Only removes objects from content store when deleted repo is the only
referencing repository

* Make LFS module stateless

Fixes bug where LFS would not work after installation without
restarting Gitea

* Change 500 'Internal Server Error' to 400 'Bad Request'

* Change sql query to xorm call

* Remove unneeded type from LFS module

* Change internal imports to code.gitea.io/gitea/

* Add Gitea authors copyright

* Change basic auth realm to "gitea-lfs"

* Add unique indexes to LFS model

* Use xorm count function in LFS check on repository delete

* Return io.ReadCloser from content store and close after usage

* Add LFS info to runWeb()

* Export LFS content store base path

* LFS file download from UI

* Work around git-lfs client issue with unauthenticated requests

Returning a dummy Authorization header for unauthenticated requests
lets git-lfs client skip asking for auth credentials
See: https://github.com/github/git-lfs/issues/1088

* Fix unauthenticated UI downloads from public repositories

* Authentication check order, Finish LFS file view logic

* Ignore LFS hooks if installed for current OS user

Fixes Gitea UI actions for repositories tracking LFS files.
Checks for minimum needed git version by parsing the semantic version
string.

* Hide LFS metafile diff from commit view, marking as binary

* Show LFS notice if file in commit view is tracked

* Add notbefore/nbf JWT claim

* Correct lint suggestions - comments for structs and functions

- Add comments to LFS model
- Function comment for GetRandomBytesAsBase64
- LFS server function comments and lint variable suggestion

* Move secret generation code out of conditional

Ensures no LFS code may run with an empty secret

* Do not hand out JWT tokens if LFS server support is disabled
2016-12-26 09:16:37 +08:00