For each assigned project, the worker sequentially calls
Sync_NetworkHalf and Sync_LocalHalf, respecting --local-only and
--network-only flags. To prevent scrambled progress bars, all stderr
output from the checkout phase is captured (shown with --verbose).
Result objects now carry status and timing information from the worker
for state updates.
Bug: 421935613
Change-Id: I398602e08a375e974a8914e5fa48ffae673dda9b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/483301
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Introduce the parallel orchestration framework for `repo sync
--interleaved`.
The new logic respects project dependencies by processing them in
hierarchical levels. Projects sharing a git object directory are grouped
and processed serially. Also reuse the familiar fetch progress bar UX.
Bug: 421935613
Change-Id: Ia388a231fa96b3220e343f952f07021bc9817d19
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/483281
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Upload gets a FileNotFoundError if not using superproject because it
tries to access the superproject's repo_id before checking if
superproject was actually enabled.
Reorder the logic to check use_superproject first.
Change-Id: I65cd2adab481e799dd7bb75e1a83553ad6e34d8d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/484401
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Prepare for an interleaved fetch and checkout mode for `repo sync`. The
goal of the new mode is to significantly speed up syncs by running fetch
and checkout operations in parallel for different projects, rather than
waiting for all fetches to complete before starting any checkouts.
Bug: 421935613
Change-Id: I8c66d1e790c7bba6280e409b95238c5e4e61a9c8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/482821
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
This will make it easier to add more formats without exploding the
common --xxx space and checking a large set of boolean flags.
Also fill out the test coverage while we're here.
Bug: b/412725063
Change-Id: I754013dc6cb3445f8a0979cefec599d55dafdcff
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/471941
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This change should be a noop from a functional point of view.
Exceptions inheriting directly from BaseException (KeyboardInterrupt,
SystemExit) are not caught by "except Exception", they will instead
continue raising upwards the stack, so there is no need to explicitly
catch and re-raise them.
Change-Id: Ic10764af4a6c05d1162f8b21651e7864ed742286
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/469601
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
In sha1 mode, when depth is enabled, syncing the revision from
upstream may not work because some servers only allow fetching
named refs. Fetching a specific sha1 may result in an error like
'server does not allow request for unadvertised object'. In this
case, attempt a full sync with depth disabled.
Bug: 410825502
Change-Id: If51bcf18b877cd9491706f5bc3d6fd13c0c3d4f3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/468282
Commit-Queue: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Warn users if the effective job count specified via `-j`,
`--jobs-network`, or `--jobs-checkout` exceeds a threshold
(currently 100). This encourages users to use more reasonable
values.
Bug: 406868778
Bug: 254914814
Change-Id: I116e2bbaf3dc824c04d1b2fbe52cf9ca5be77b9a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/466801
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
If you try running this from a subdir, then most of the tests fail
because they assume they're running from the top of the source tree.
Change all the tests to actually run there.
For example: cd docs && ../run_tests
Change-Id: I92e17476393a108e56b58e049193b9fd72c5b7ba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/464841
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
repo attempts to create /etc/.repo_gitconfig.json file, and fails if
root file system is mounted read-only. Removing non-existing file on
read-only filesystem results in EROFS instead of ENOENT.
Bug: 401018409
Change-Id: I64edc0567fb88649f3fd8cacb65a8780744640d4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/458821
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Egor Duda <egor.duda@gmail.com>
Commit-Queue: Egor Duda <egor.duda@gmail.com>
If the repo index is stale, reset --keep will refuse to reset workspace.
An index can be stale if there are any modifications to file node,
including mtime, atime, ownership changes, etc.
Bug: b/375423099
Change-Id: Ibef03d9d8d2babbb107041707281687342ab7a77
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/460022
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@google.com>
If tags are omitted from the remote pack list, they must be present in
local pack. However, local packs don't have promisor objects, meaning
that all blobs must be available locally, and therefore all missing
blobs will be downloaded during rev-list phase. Git downloads those
sequentially, by invokving fetch operation (rev-list/fetch).
Instead of downloading tags' blobs, instruct Git to include all tags in
remote rev-list operation. This change was tested with `git fsck --all`.
R=yiwzhang@google.com
Bug: b/392732561
Change-Id: Id94a40aebbe4f084c952329583d559d296db1a11
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/451422
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Change the bare checkout directory for submodules from 'subprojects'
to 'modules'. Git expects bare submodule checkouts to be in the
'modules' directory. If old subproject directories are found, they
will be migrated to the new modules directory. This change is the
first step in ensuring Git can understand repo's submodules to some
extent.
Change-Id: I385029f1bb55d040616d970d6ffb4bb856692520
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/444881
Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Currently, repo ignores the XDG path for the git config file, and
creates a new one in the user's home directory. This commit changes the
behavior to prefer the XDG path if it exists, which matches git behavior
and avoids littering the home directory.
Bug: 40012443
Change-Id: Icd3ec6db6b0832f47417bbe98ff9461306b51297
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/448385
Tested-by: lmaor xenix <25misha52@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
When repo is initializing a git repository, it calls `git read-tree`.
During such operation, git is restoring workspace based on the current
index. However, some things can go wrong: a user can run out of disk
space, or, in case of partial clone, user may no longer reach the remote
host. That will leave affected repository in a bad state with partially
checked out workspace. The follow up repo sync won't try to fix such
state.
This change removes .git symlink, which will force the next `repo sync`
to redo Git repository setup.
Bug: b/363171216
Bug: b/390161127
Change-Id: I57db4b6cae0ef21826dc7cede4d3bf02cfc3d955
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447801
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
When a repository is partially cloned, no longer needed blobs are never
removed. To reclaim some of disk space, allow user to pass --repack
which affects only repositories with filter=blob:none and if projects
are not shared.
Change-Id: I0608172c9eff82fb8a6b6ef703eb109fedb7a6cc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447722
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@google.com>
When projects are removed from manifest, they are only removed from
worktree and not from .repo/projects and .repo/project-objects. Keeping
data under .repo can be desired if user expects deleted projects to be
restored (e.g. checking out a release branch).
Android has ongoing effort to remove many stale projects and this change
allows users to easily free-up their disk space.
Bug: b/344018971
Bug: 40013312
Change-Id: Id23c7524a88082ee6db908f9fd69dcd5d0c4f681
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445921
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
If repo sync is invoked outside the repo root, and the latest manifest
removes symlinks, repo incorrectly tries to remove symlink - it starts
from `cwd` instead of the repo root.
Bug: b/113935847
Bug: 40010423
Change-Id: Ia50ea70a376e38c94389880f020c80da3c3f453c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445901
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
