1936 Commits

Author SHA1 Message Date
Mike Frysinger
59b81c84de launcher: change collections.namedtuple to typing.NamedTuple
Since we require Python 3.6 now in the launcher, switch to NamedTuple
so we get better documentation & typing information.

Change-Id: Ic58fdc07db02fc49166eccbbc3e527f474973424
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/463721
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2025-03-28 19:13:49 -07:00
Mike Frysinger
507d463600 tox: sync black settings with run_tests
We updated run_tests to use black-25, so update tox too.

Change-Id: I7ee6471fbc78825bd2dbc8c1f8dab9dc10460852
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/463601
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2025-03-27 14:20:01 -07:00
Mike Frysinger
cd391e77d0 black: update to v25
Requires a little reformatting in the tree.

Change-Id: Iaa40fe0dfca372c49c04cc26edccb5f7b0c2a8ad
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/462883
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2025-03-25 11:20:35 -07:00
Mike Frysinger
8310436be0 run_tests: move test filtering to pytest markers
Move the test disable logic even closer to the exact test that's
disabled.  This way people updating tests have a better chance of
seeing they'll get reduced coverage in the CQ.

Change-Id: I57c1a073a844019798b27e14d742fd32925d9ae8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/462882
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2025-03-25 11:19:49 -07:00
Mike Frysinger
d5087392ed run_tests: move CQ test skips here
Our recipes have been disabling a bunch of tests.  To increase
visibility, and to make it easier to test changes, move that
logic to this script.

Change-Id: I3894f047715177c0f1d27a2fe4c3490972dab204
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/462881
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2025-03-25 10:08:54 -07:00
Mike Frysinger
91f428058d run_tests: run all tests all the time
Using a generator w/all() causes the code to exit on the first error.
We really want to see all errors all the time, so use sum() instead.

Change-Id: Ib1adb8de199db9fe727d4b49c890b4d5061e9e6b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/462901
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
2025-03-25 10:07:42 -07:00
Mike Frysinger
243df2042e launcher: change RunResult to subprocess.CompletedProcess
Since we require Python 3.6 now in the launcher, swap out our custom
RunResult class for the standard subprocess one.

Change-Id: Idd8598df37c0a952d3ef828df6e250cab03c6589
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/462341
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2025-03-24 11:49:00 -07:00
Albert Akmukhametov
4b94e773ef Sync: Fix full submodule sync while shallow specified
Git allows to clone submodules as shallow clone [1]. On the other
hand, when repo synchronize a projcet with submodules inside, it
ignores the shallow parameter.

When a project contains submodules, project.py parses the .gitmodules
file for URL and path. This parsing does not consider the shallow
option. Consequently, this parameter is not propgated to newly
created Project instance for that submodule.

[1] https://git-scm.com/docs/gitmodules#Documentation/gitmodules.txt-submoduleltnamegtshallow

Change-Id: I54fc9c69ae1b8e3cda2801202e3f0c7693b718d2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/454261
Tested-by: Albert Akmukhametov <alb.02057@gmail.com>
Commit-Queue: Albert Akmukhametov <alb.02057@gmail.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Никита Сказкоподателев (Nask) <skazkopodatelev@gmail.com>
v2.53
2025-03-13 09:12:45 -07:00
Josip Sokcevic
fc901b92bb sync: Refresh index before updating repo
If the repo index is stale, reset --keep will refuse to reset workspace.
An index can be stale if there are any modifications to file node,
including mtime, atime, ownership changes, etc.

Bug: b/375423099
Change-Id: Ibef03d9d8d2babbb107041707281687342ab7a77
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/460022
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@google.com>
2025-03-13 08:24:35 -07:00
Josip Sokcevic
8d5f032611 gc: Add tags to remote pack list
If tags are omitted from the remote pack list, they must be present in
local pack. However, local packs don't have promisor objects, meaning
that all blobs must be available locally, and therefore all missing
blobs will be downloaded during rev-list phase. Git downloads those
sequentially, by invokving fetch operation (rev-list/fetch).

Instead of downloading tags' blobs, instruct Git to include all tags in
remote rev-list operation. This change was tested with `git fsck --all`.

R=yiwzhang@google.com

Bug: b/392732561
Change-Id: Id94a40aebbe4f084c952329583d559d296db1a11
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/451422
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
v2.52
2025-02-05 12:36:27 -08:00
Kaushik Lingarkar
99eca45eb2 Activate submodules
This change moves further towards ensuring Git can understand repo's
submodules. 'submodule init' is used to make the submodules active[1].

[1] https://git-scm.com/docs/gitsubmodules#_active_submodules

Change-Id: I0c20ff1991101fc5be171e566d8fb644aab47200
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/446182
Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Mike Frysinger <vapier@google.com>
2025-02-04 08:07:49 -08:00
Kaushik Lingarkar
66685f07ec Use 'gitfile' in submodule checkouts
This change takes another step towards ensuring Git can understand
repo's submodules to some extent. Replace the old '.git' symlink with
gitfile[1] pointing to the bare checkout of the submodule. This is
required for Git's 'recurse submodules' opts to work with repo's
submodules as '.git' is expected to be writable by Git when recursing
over submodules.

[1] https://git-scm.com/docs/gitrepository-layout#_description

Change-Id: I52d15451768ee7bd6db289f4d2b3be5907370d42
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/446181
Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
2025-02-04 08:07:49 -08:00
Kaushik Lingarkar
cf9a2a2a76 Update internal filesystem layout for submodules
Change the bare checkout directory for submodules from 'subprojects'
to 'modules'. Git expects bare submodule checkouts to be in the
'modules' directory. If old subproject directories are found, they
will be migrated to the new modules directory. This change is the
first step in ensuring Git can understand repo's submodules to some
extent.

Change-Id: I385029f1bb55d040616d970d6ffb4bb856692520
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/444881
Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2025-02-04 08:07:49 -08:00
Josip Sokcevic
5ae8292fea Revert "sync: skip network half on repo upgrade"
This reverts commit 61224d01fa29bcf54dd6d521e09e09a8c0da77fe.

Reason for revert: the manifest will be updated during in the
post-upgrade process, and that can result in a missing object in
LocalHalf, since NetworkHalf is not skipped.

Bug: b/392979411
Change-Id: I8a46e5b54093ed78285c8b30f000bb08a8244179
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/450181
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@google.com>
2025-01-31 12:01:49 -08:00
Mike Frysinger
dfdf577e98 docs: smart-sync: split out & expand details
The existing documentation on smart-sync behavior is a bit light on
details, and out of date wrt what the code actually does.  Start a
dedicated document and fill it out more.

Change-Id: I1a8a3ac6edf9291d72182ad55db865035d9b683e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/450002
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2025-01-30 19:17:24 -08:00
Mike Frysinger
747ec83f58 run_tests: update to python 3.11 & pytest 8.3.4
Change-Id: Iffe45d85a54dc380cdd37bbbbe64b058eacad0a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/449901
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2025-01-30 10:54:54 -08:00
flexagoon
1711bc23c0 git_config: prefer XDG config location
Currently, repo ignores the XDG path for the git config file, and
creates a new one in the user's home directory. This commit changes the
behavior to prefer the XDG path if it exists, which matches git behavior
and avoids littering the home directory.

Bug: 40012443
Change-Id: Icd3ec6db6b0832f47417bbe98ff9461306b51297
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/448385
Tested-by: lmaor xenix <25misha52@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2025-01-23 23:47:06 -08:00
Josip Sokcevic
db111d3924 sync: Recover from errors during read-tree
When repo is initializing a git repository, it calls `git read-tree`.
During such operation, git is restoring workspace based on the current
index. However, some things can go wrong: a user can run out of disk
space, or, in case of partial clone, user may no longer reach the remote
host. That will leave affected repository in a bad state with partially
checked out workspace. The follow up repo sync won't try to fix such
state.

This change removes .git symlink, which will force the next `repo sync`
to redo Git repository setup.

Bug: b/363171216
Bug: b/390161127
Change-Id: I57db4b6cae0ef21826dc7cede4d3bf02cfc3d955
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447801
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
v2.51
2025-01-16 09:19:45 -08:00
Josip Sokcevic
3405446a4e gc: Add repack option
When a repository is partially cloned, no longer needed blobs are never
removed. To reclaim some of disk space, allow user to pass --repack
which affects only repositories with filter=blob:none and if projects
are not shared.

Change-Id: I0608172c9eff82fb8a6b6ef703eb109fedb7a6cc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447722
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@google.com>
2025-01-14 15:17:34 -08:00
Josip Sokcevic
41a27eb854 gc: extract deletion from Execute method
Change-Id: Icef4f28fbdb9658892611def7589f5eba43c952c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447721
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
2025-01-14 12:33:45 -08:00
Josip Sokcevic
d93fe60e89 sync: Handle KeyboardInterrupt during checkout
KeyboardInterrupt is handled during NetworkHalf. This patch handles
KeyboardInterrupt during LocalHalf.

Bug: b/372069163
Change-Id: I26847f7ca3cdf1fe57b265b4f6b18cc8102d2921
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447401
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2025-01-08 13:36:52 -08:00
Josip Sokcevic
61224d01fa sync: skip network half on repo upgrade
When repo upgrades itself, it will restart itself and rerun sync
command. At that point, we know that network half is already done and we
can just proceed with local half.

R=ddoman@google.com

Bug: b/377567091
Change-Id: I77205b1f2df19891597347d55283a617de3c6634
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/446201
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
2024-12-18 11:49:17 -08:00
Josip Sokcevic
13d6588bf6 gc: Introduce new command to remove old projects
When projects are removed from manifest, they are only removed from
worktree and not from .repo/projects and .repo/project-objects. Keeping
data under .repo can be desired if user expects deleted projects to be
restored (e.g. checking out a release branch).

Android has ongoing effort to remove many stale projects and this change
allows users to easily free-up their disk space.

Bug: b/344018971
Bug: 40013312
Change-Id: Id23c7524a88082ee6db908f9fd69dcd5d0c4f681
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445921
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
v2.50.1
2024-12-18 09:23:49 -08:00
Josip Sokcevic
9500aca754 sync: Delete symlinks relative to client topdir
If repo sync is invoked outside the repo root, and the latest manifest
removes symlinks, repo incorrectly tries to remove symlink - it starts
from `cwd` instead of the repo root.

Bug: b/113935847
Bug: 40010423
Change-Id: Ia50ea70a376e38c94389880f020c80da3c3f453c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445901
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
v2.50 v2.50.0
2024-12-16 10:23:40 -08:00
Fredrik de Groot
e8a7b9d596 Add smoke test for subcmd forall
After some refactoring earlier, the forall command was
broken briefly, returning after only one run instead
of after all projects.

This test, albeit simple in nature, would have caught that.

Due to the somewhat demanding nature of forall,
a lot more setup was needed than expected but seems
to do its job now so hopefully it catches similar stuff
in the future.

Change-Id: I51e161ff0e7e31a65401211c376f319bda504532
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445461
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Fredrik de Groot <fredrik.de.groot@haleytek.com>
2024-12-11 00:30:15 -08:00
Josip Sokcevic
cf411b3f03 Remove gitc support from repo
gitc is no longer available.

Change-Id: I0cbfdf936832f2cdd4876104ae3cc5a6e26154e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/444841
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
2024-12-03 22:27:56 +00:00
Josip Sokcevic
1feecbd91e branches: Escape percent signs in branch names
If a branch name contains a percent sign, it will be interpreted as a placeholder and color.py will fail to format it.

To avoid this, escape the percent signs prior to calling Coloring
method.

Bug: b/379090488
Change-Id: Id019c776bbf8cbed5c101f2773606f1d32c9e057
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443801
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-12-03 19:02:20 +00:00
Peter Kjellerstedt
616e314902 sync: Do not fail to sync a manifest with no projects
Since commit 454fdaf1191c87e5c770ab865a911e10e600e178 (v2.48), syncing a
manifest without any projects would result in:

  Repo command failed: RepoUnhandledExceptionError
          Number of processes must be at least 1

Bug: 377546300
Change-Id: Iaa2f6a3ac64542ad65a19c0eef449f53c09cae67
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443442
Reviewed-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Tested-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
2024-11-26 10:16:03 +00:00
Josip Sokcevic
fafd1ec23e Fix event log command event hierarchy.
command should be cmd_name, to match what git is emitting. This also
fixes arguments, so that only relevant arguments are passed instead
of the entire sys.args, which will contain wrapper information

Change-Id: Id436accfff511292ec2c56798fffb2306dda38fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443741
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
2024-11-22 18:39:41 +00:00
Josip Sokcevic
b1613d741e Make repo installation work without .git
Some tools like jj and cog will not have .git. This change
makes it possible to run all repo commands in such setups.

Change-Id: I7f3845dc970fbaa731c31e0aa48355a4b56ed3a6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/442821
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-11-18 19:36:14 +00:00
Kuang-che Wu
ab2d321104 sync: fix connection error on macOS
With a large number of sync workers, the sync process may fail on
macOS due to connection errors. The root cause is that multiple
workers may attempt to connect to the multiprocessing manager server
at the same time when handling the first job. This can lead to
connection failures if there are too many pending connections, exceeding
the socket listening backlog.

Bug: 377538810
Change-Id: I1924d318d076ca3be61d75daa37bfa8d7dc23ed7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441541
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
v2.49.3
2024-11-06 16:33:17 +00:00
Josip Sokcevic
aada468916 upload: Return correct tuple values in _ProcessResults
Incorrect tuple values were returned with http://go/grev/440221 -
instead of returning (Project, ReviewableBranch), _ProcessResults was
returning (int, ReviewableBranch).

R=jojwang@google.com

Bug: 376731172
Change-Id: I75205f42fd23f5ee6bd8d0c15b18066189b42bd9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441121
Reviewed-by: Sam Saccone <samccone@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
v2.49.2
2024-10-31 21:18:53 +00:00
Allen Webb
1d5098617e worktree: Do not try to fix relative paths
--worktree was broken with incorrect paths in the .git files
whenever the local copy of git populated gitdir with relative paths
instead of absoulte paths.

Bug: 376251410
Change-Id: Id32dc1576315218967de2a9bfe43bf7a5a0e7aa6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440801
Commit-Queue: Allen Webb <allenwebb@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Allen Webb <allenwebb@google.com>
v2.49.1
2024-10-30 17:03:57 +00:00
Josip Sokcevic
e219c78fe5 forall: Fix returning results early
rc should be returned only after all results are processed.

R=jojwang@google.com

Bug: b/376454189
Change-Id: I8200b9954240dd3e8e9f2ab82494779a3cb38627
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440901
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
2024-10-30 16:11:04 +00:00
joehsu
f9f4df62e0 Use full name of the revision when checking dest-branch
The manifest usually doesn't sepecify the revision with the full name
(e.g. refs/heads/REV).
However, when checking if the name of the merge branch, full name is
used on the merge branch.

The CL use full name of revision when comparing it with the merge
branch.

Bug: b/370919047
Test: repo upload on a project with `dest-branch` set
Change-Id: Ib6fa2f7246beb5bae0a26a70048a7ac03b6c5a2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438401
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Joe Hsu <joehsu@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
v2.49
2024-10-28 23:47:08 +00:00
Fredrik de Groot
ebdf0409d2 Add REPO_SKIP_SELF_UPDATE check in sync
The command _PostRepoFetch will try to self update
during repo sync. That is beneficial but adds
version uncertainty, fail potential and slow downs
in non-interactive scenarios.

Conditionally skip the update if env variable
REPO_SKIP_SELF_UPDATE is defined.

A call to selfupdate works as before, meaning even
with the variable set, it will run the update.

Change-Id: Iab0ef55dc3d3db3cbf1ba1f506c57fbb58a504c3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439967
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2024-10-28 17:46:25 +00:00
Fredrik de Groot
303bd963d5 manifest: add optional base check on remove and extend
This adds an optional, built-in checker for
guarding against patches hanging on wrong
base revisions, which is useful if a lower layer of
the manifest changes after a patch was done.

When adding a patch with a new revision using
extend-project or remove-project/project:

          C---D---E patches in project bla
         /
    A---B project bla in manifest state 1

<extend-project name="bla" revision="E" base-rev="B">

If project bla gets updated, in a new snap ID
or by a supplier or similar, to a new state:

          C---D---E patches in project bla
         /
    A---B---F---G project bla in manifest state 2

Parsing will fail because revision of bla is now G,
giving the choice to create a new patch branch
from G and updating base-rev, or keeping previous
branch for some reason and only updating base-rev.

Intended for use in a layered manifest with
hashed revisions. Named refs like branches and tags
also work fine when comparing, but will be misleading
if a branch is used as base-rev.

Change-Id: Ic6211550a7d3cc9656057f6a2087c505b40cad2b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436777
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-10-28 16:55:10 +00:00
Josip Sokcevic
ae384f8623 [event_log] Stop leaking semaphore resources
With the global state and fork, we are left with uncleaned resources.
Isolate mulitprocessing.Value in a function so we stop the leak.

Bug: 353656374
Change-Id: If50bb544bda12b72f00c02bc1d2c0d19de000b88
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440261
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
2024-10-24 16:58:17 +00:00
Kuang-che Wu
70a4e643e6 progress: always show done message
The done message was omitted if the task is shorter than 0.5s. This
might confuse users.

Bug: b/371638995
Change-Id: I3fdd2cd8daea16d34fba88457d09397fff71af15
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440222
Tested-by: Kuang-che Wu <kcwu@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2024-10-24 16:21:28 +00:00
Kuang-che Wu
8da4861b38 subcmds: reduce multiprocessing serialization overhead
Follow the same approach as 39ffd9977e to reduce serialization overhead.

Below benchmarks are tested with 2.7k projects on my workstation
(warm cache). git tracing is disabled for benchmark.

(seconds)              | v2.48 | v2.48 | this CL | this CL
	               |       |  -j32 |         |    -j32
-----------------------------------------------------------
with clean tree state:
branches (none)        |   5.6 |   5.9 |    1.0  |    0.9
status (clean)         |  21.3 |   9.4 |   19.4  |    4.7
diff (none)            |   7.6 |   7.2 |    5.7  |    2.2
prune (none)           |   5.7 |   6.1 |    1.3  |    1.2
abandon (none)         |  19.4 |  18.6 |    0.9  |    0.8
upload (none)          |  19.7 |  18.7 |    0.9  |    0.8
forall -c true         |   7.5 |   7.6 |    0.6  |    0.6
forall -c "git log -1" |  11.3 |  11.1 |    0.6  |    0.6

with branches:
start BRANCH --all     |  21.9 |  20.3 |   13.6  |    2.6
checkout BRANCH        |  29.1 |  27.8 |    1.1  |    1.0
branches (2)           |  28.0 |  28.6 |    1.5  |    1.3
abandon BRANCH         |  29.2 |  27.5 |    9.7  |    2.2

Bug: b/371638995
Change-Id: I53989a3d1e43063587b3f52f852b1c2c56b49412
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440221
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Kuang-che Wu <kcwu@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>
2024-10-23 23:34:34 +00:00
Kuang-che Wu
39ffd9977e sync: reduce multiprocessing serialization overhead
Background:
 - Manifest object is large (for projects like Android) in terms of
   serialization cost and size (more than 1mb).
 - Lots of Project objects usually share only a few manifest objects.

Before this CL, Project objects were passed to workers via function
parameters. Function parameters are pickled separately (in chunk). In
other words, manifests are serialized again and again. The major
serialization overhead of repo sync was
  O(manifest_size * projects / chunksize)

This CL uses following tricks to reduce serialization overhead.
 - All projects are pickled in one invocation. Because Project objects
   share manifests, pickle library remembers which objects are already
   seen and avoid the serialization cost.
 - Pass the Project objects to workers at worker intialization time.
   And pass project index as function parameters instead. The number of
   workers is much smaller than the number of projects.
 - Worker init state are shared on Linux (fork based). So it requires
   zero serialization for Project objects.

On Linux (fork based), the serialization overhead is
  O(projects)  --- one int per project
On Windows (spawn based), the serialization overhead is
  O(manifest_size * min(workers, projects))

Moreover, use chunksize=1 to avoid the chance that some workers are idle
while other workers still have more than one job in their chunk queue.

Using 2.7k projects as the baseline, originally "repo sync" no-op
sync takes 31s for fetch and 25s for checkout on my Linux workstation.
With this CL, it takes 12s for fetch and 1s for checkout.

Bug: b/371638995
Change-Id: Ifa22072ea54eacb4a5c525c050d84de371e87caa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439921
Tested-by: Kuang-che Wu <kcwu@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>
2024-10-23 02:58:45 +00:00
Kaushik Lingarkar
584863fb5e Fix incremental syncs for prjs with submodules
When performing an incremental sync (re-running repo init with an
updated manifest revision) with --fetch-submodules or sync-s=true,
there is an attempt to get a list of all projects (including
submodules) before projects are actually fetched. However, we can
only list submodules of a project if we have already fetched its
revision. Instead of throwing an error when we don't have the
revision, assume there are no submodules for that project. In the
sync cmd, we already update the list of projects to include
submodules after fetching superprojects.

Change-Id: I48bc68c48b5b10117356b18f5375d17f9a89ec05
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439761
Commit-Queue: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Tested-by: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org>
2024-10-18 03:55:10 +00:00
Josip Sokcevic
454fdaf119 sync: Always use WORKER_BATCH_SIZE
With 551285fa35ccd0836513e9cf64ee8d3372e5e3f4, the comment about number
of workers no longer stands - dict is shared among multiprocesses and
real time information is available.

Using 2.7k projects as the baseline, using chunk size of 4 takes close
to 5 minutes. A chunk size of 32 takes this down to 40s - a reduction of
rougly 8 times which matches the increase.

R=gavinmak@google.com

Bug: b/371638995
Change-Id: Ida5fd8f7abc44b3b82c02aa0f7f7ae01dff5eb07
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438523
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
v2.48
2024-10-07 18:44:19 +00:00
Josip Sokcevic
f7f9dd4deb project: Handle git sso auth failures as repo exit
If a user is not authenticated, repo continues execution and it will
likely result in more of the same errors being printed. A user is also
likely to SIGTERM the process resulting in more errors.

This change stops repo sync if any of repositories can't be fetched to
Git authentcation using sso helper. We could extend this to all Git
authentication

Change-Id: I9e471e063450c0a51f25a5e7f12a83064dfb170c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438522
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-10-03 20:47:50 +00:00
Josip Sokcevic
70ee4dd313 superproject: Remove notice about beta
It's been the default for Android for over a year now, and it's no
longer useful notice.

Change-Id: I53c6f1e7cee8c1b2f408e67d3a6732db3b272bee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438521
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-10-03 20:37:18 +00:00
Josip Sokcevic
cfe3095e50 project: run fetch --refetch on unable to not parse commit
Similarly to e59e2ae757623e64f625a9cdadf1c2010ef82b34, handle missing
gc'ed commits by running `git fetch --refetch`.

R=jojwang@google.com

Bug: b/360889369
Bug: b/371000949
Change-Id: I108b870b855d3b9f23665afa134c6e35f7cd2830
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438461
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-10-03 17:40:37 +00:00
Josip Sokcevic
621de7ed12 Disable git terminal prompt during fetch/clone
git fetch operation may prompt user to enter username and password.
This won't be visible to user when repo sync operation since stdout and
stderr are redirected. If that happens, user may think repo is doing
work and likely won't realize it's stuck on user's input.

This patch disables prompt for clone and fetch operations, and repo will
fail fast.

R=gavinmak@google.com

Bug: b/368644181
Change-Id: I2efa88ae66067587a00678eda155d861034b9127
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438001
Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2024-09-26 22:10:36 +00:00
Kaushik Lingarkar
d7ebdf56be init: add --manifest-upstream-branch
When a sha1 is provided to '--manifest-branch', the ref which
is expected to contain that sha1 can be provided using the new
'--manifest-upstream-branch' option. This is useful with
'--current-branch' to avoid having to sync all heads and tags,
or with a commit that comes from a non-head/tag ref (like a
Gerrit change ref).

Change-Id: I46a3e255ca69ed9e809039e58b0c163e02af94ef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436717
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Tested-by: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
2024-09-26 00:52:28 +00:00
Kaushik Lingarkar
fabab4e245 man: regenerate man pages
Change-Id: Icf697eda7d5dcdc87854ad6adf607353c7ba5ac2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/437941
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2024-09-25 20:57:42 +00:00
Brian Norris
b577444a90 project: Copy and link files even with local branches
In the winding maze that constitutes Sync_LocalHalf(), there are paths
in which we don't copy-and-link files. Examples include something like:

  cd some/project/
  repo start head .
  # do some work, make some commit, upload that commit to Gerrit

  [[ ... in the meantime, someone addes a <linkfile ...> for
     some/project/ in the manifest ... ]]

  cd some/project/
  git pull --rebase
  repo sync

In this case, we never hit a `repo rebase` case, which might have saved
us. Instead, the developer is left confused why some/project/ never had
its <linkfile>s created.

Notably, this opens up one more corner case in which <linkfile ... /> or
<copyfile ... /> could potentially clobber existing work in the
destination directory, but there are existing cases where that's true,
and frankly, those seem like bigger holes than this new one.

Change-Id: I394b0e4529023a8ee319dc25d03d513a19251a4a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/437421
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Brian Norris <briannorris@google.com>
Commit-Queue: Brian Norris <briannorris@google.com>
2024-09-19 00:11:52 +00:00