Commit Graph

757 Commits

Author SHA1 Message Date
Jason Chang
1d3b4fbeec sync: Track new/existing project count
New vs existing project may be a useful measure for analyzing
sync performance.

Bug: b/287105597
Change-Id: Ibea3e90c9fe3d16fd8b863bcae22b21963a6771a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/377574
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
2023-06-23 20:08:58 +00:00
Gavin Mak
945c006f40 sync: Update sync progress even when _sync_dict is empty
By chance, _sync_dict can be empty even though repo sync is still
working. In that case, the progress message shows incorrect info. Handle this case and fix a bug where "0 jobs" can show.

Bug: http://b/284465096
Change-Id: If915d953ba60e7cf84a6fb2d137fd6ed82abd3cc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375494
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2023-05-30 20:25:00 +00:00
Josip Sokcevic
71122f941f sync: Handle race condition when reading active jobs
It's possible that number of jobs is more than 0 when we
check length, but in the meantime number of jobs drops to
0. In that case, we are working with float(inf) which
causes other problems

Bug: 284383869
Change-Id: I5d070d1be428f8395df7fde8ca84866db46f2100
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375134
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2023-05-26 15:50:20 +00:00
Jason Chang
17833322d9 Add envar to replace shallow clones with partial
An investigation go/git-repo-shallow shows a number of problems
when doing a shallow git fetch/clone. This change introduces an
environment variable REPO_ALLOW_SHALLOW. When this environment variable
is set to 1 during a repo init or repo sync all shallow git fetch
commands are replaced with partial fetch commands. Any shallow
repository needing update is unshallowed. This behavior continues until
a subsequent repo sync command is run with REPO_ALLOW_SHALLOW set to 1.

Bug: b/274340522
Change-Id: I1c3188270629359e52449788897d9d4988ebf280
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374754
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
2023-05-25 22:37:04 +00:00
Gavin Mak
04cba4add5 sync: Show number of running fetch jobs
Last of the recent `repo sync` UX changes. Show number of fetch jobs eg:
"Fetching:  3% (8/251) 0:03 | 8 jobs | 0:01 chromiumos/overlays/chrom.."

Bug: https://crbug.com/gerrit/11293
Change-Id: I1b3dcf3e56ae6731c6c6cb73cfce069b2f374b69
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374920
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
2023-05-25 17:26:22 +00:00
Josip Sokcevic
3eacfdf309 upload: use f-string
Change-Id: I91b99a7147c7c3cb5485d5406316c8ffd79f9272
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374914
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2023-05-25 17:12:18 +00:00
Gavin Mak
551285fa35 sync: Show elapsed time for the longest syncing project
"Last synced: X" is printed only after a project finishes syncing.
Replace that with a message that shows the longest actively syncing
project.

Bug: https://crbug.com/gerrit/11293
Change-Id: I84c7873539d84999772cd554f426b44921521e85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/372674
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2023-05-18 18:10:24 +00:00
Josip Sokcevic
131fc96381 [git_trace2] Add logs for critical cmds
Trace logs emitted from repo are not useful on error for many critical
commands. This change adds errors for critical commands to trace logs.

Change-Id: Ideb9358bee31e540bd84a94327a09ff9b0246a77
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/373814
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2023-05-17 18:06:14 +00:00
Gavin Mak
041f97725a sync: Fix how sync times for shared projects are recorded
https://gerrit.googlesource.com/git-repo/+/d947858325ae70ff9c0b2f463a9e8c4ffd00002a introduced a moving average of fetch times in 2012.

The code does not handle shared projects, and averages times based on project names which is incorrect.

Change-Id: I9926122cdb1ecf201887a81e96f5f816d3c2f72a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/373574
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2023-05-10 21:14:57 +00:00
Gavin Mak
edcaa94ca8 sync: Display total elapsed fetch time
Give users an indication that `repo sync` isn't stuck if taking a long
time to fetch.

Bug: https://crbug.com/gerrit/11293
Change-Id: Iccdaec918f86c9cc2db5dc12f9e3eef7ad0bcbda
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/371414
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2023-05-02 20:51:46 +00:00
Aravind Vasudevan
e74d9046ee Update abandon to support multiple branches
This change updates `repo abandon` command to take multiple space-separated branchnames as parameters.

Bug: https://crbug.com/gerrit/13354
Change-Id: I00ad7a79872c0e4161f8183843835f25cd515605
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/365524
Tested-by: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
2023-03-24 07:39:28 +00:00
Gavin Mak
ea2e330e43 Format codebase with black and check formatting in CQ
Apply rules set by https://gerrit-review.googlesource.com/c/git-repo/+/362954/ across the codebase and fix any lingering errors caught
by flake8. Also check black formatting in run_tests (and CQ).

Bug: b/267675342
Change-Id: I972d77649dac351150dcfeb1cd1ad0ea2efc1956
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/363474
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2023-03-22 17:46:28 +00:00
Joanna Wang
7fa149b47a upload: Skip upload if merge branch doesn't match project revision and
dest_branch.

- This still prevents the case mentioned here:
https://gerrit-review.googlesource.com/c/50300
while also supporting dest_branch.
- Update _GetMergeBranch to get merge branches for any branch, not just
the one we happen to run `repo upload` in. (e.g. for uploading multiple
branches)

Bug: b/27955930
Change-Id: Ia8ee1d6a83a783c984bb2eb308bb11b3a721a95d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/360794
Commit-Queue: Joanna Wang <jojwang@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Joanna Wang <jojwang@google.com>
2023-02-28 14:21:17 +00:00
Josip Sokcevic
48067714ec sync: Remove unused variable
Change-Id: I44ab990c89ab4da82c424bae95e463cabb12fd50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357136
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2023-02-02 18:11:27 +00:00
Josip Sokcevic
69427da8c9 Handle KeyboardInterrupt during repo sync
If interrupt signal is sent to repo process while sync is running, repo
prints stack trace for each concurrent job that is currently running
with no useful information.

Instead, this change captures KeyboardInterrupt in each process and
prints one line about current project that is being processed.

Change-Id: Ieca760ed862341939396b8186ae04128d769cd56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357135
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
2023-02-01 23:41:11 +00:00
Josip Sokcevic
dccf38e34f Update sync progress
repo sync progress bar is misleading. Many bug reports mentioned that
repo is stuck at the repo that is currently displayed in the progress
bar. Repo sync actually shows what repository is the last processed.
This change makes that obvious.

Change-Id: I962bf0bc65af7ac0ed98db86e9144f07d9e1f96f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357134
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
2023-02-01 23:38:52 +00:00
Michael Kelly
7eab0eedf2 sync: Silence 'not found in manifest' message
This can potentially show up when sync'ing projects with submodules
that are not declared in the manifest as well as the internal
'.repo/repo' project, which is likely not desirable from a user
standpoint.

Change-Id: I93d7fcd6e3fd1818357ea4537882a864dea9942c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/355920
Reviewed-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
2023-01-31 21:52:16 +00:00
Gavin Mak
c3d61ec252 init: Silence the "rm -r .repo and try again" message if quiet
Bug: b/258532367
Change-Id: I53a23aa0b237b0bb5f7e58464936f8c9b0db1311
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/355915
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2023-01-06 16:01:52 +00:00
Sergiy Belozorov
78e82ec78e Fix flake8 warnings for some files
Change-Id: If67f8660cfb0479f0e710b3566285ef401fcf077
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/355969
Tested-by: Sergiy Belozorov <sergiyb@chromium.org>
Reviewed-by: Mike Frysinger <vapier@google.com>
2023-01-05 18:43:12 +00:00
Daniel Kutik
e641281d14 Use print with flush=True instead of stdout.flush
Resolves multiple TODOs. Since we are requiring Python 3,
we move to using print function with flush=True instead of
using sys.stdout.flush().

Change-Id: I54db0344ec78ac81a8d6c6c7e43ee7d301f42f02
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354701
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-12-13 16:23:28 +00:00
Daniel Kutik
035f22abec pylint: remove unused imports
Removed unused imports accross multiple files.

Change-Id: Ib5ae4cebf9660e7339b11e3fa592d99f8d51e8d8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354700
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-12-13 16:23:19 +00:00
David Greenaway
d98f393524 upload: Allow user to configure unusual commit threshold
Add a per-remote option `uploadwarningthreshold` allowing the user to
override how many commits can be uploaded prior to a warning being
displayed.

Change-Id: Ia7e1b2c7de89a0bf9ca1c24cc83dc595b3667437
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354375
Tested-by: David Greenaway <dgreenaway@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-12-12 22:19:57 +00:00
Mike Frysinger
f159ce0f9e sync: fix manifest sync-j handling
Since --jobs defaults to 0, not None, we never pull the value out
of the manifest.  Treat values of 0 and None the same to fix.

Bug: http://b/239712300
Bug: http://b/260908907
Change-Id: I9b1026682072366616825fd72f90bd90c10a252f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354254
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
2022-12-08 15:06:24 +00:00
Karsten Tausche
802cd0c601 sync: Fix undefined variable in _FetchOne
If syncing in _FetchOne fails with GitError, sync_result does not get
set. There's already a separate local variable for success; do the same
for remote_fetched instead of referring to the conditionally defined
named tuple.

This bug is originally caused by a combination of ad8aa697 "sync: only
print error.GitError, don't raise that exception." and 1eddca84 "sync:
use namedtuples for internal return values".

Change-Id: I0f9dbafb97f8268044e5a56a6f92cf29bc23ca6a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354176
Tested-by: Karsten Tausche <karsten@fairphone.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-12-08 06:29:00 +00:00
LaMont Jones
100a214315 sync: finish marking REPO_AUTO_GC=1 as deprecated.
The wrong revision of the change was submitted as
d793553804.

Change-Id: I6f3e4993cf40c30ccf0d69020177db8fe5f76b8c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353934
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
2022-12-05 18:11:24 +00:00
LaMont Jones
43549d8d08 sync: cleanup output when not doing GC
Do not use a progress bar when not doing GC, and restrict activity in
that case to only repairing preciousObject state.

This also includes additional cleanup based on review comments from
previous changes.

Change-Id: I48581c9d25da358bc7ae15f40e98d55bec142331
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353514
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-12-02 22:50:23 +00:00
LaMont Jones
55b7125d6a Revert "sync: save any cruft after calling git gc."
This bug-cacher related code is no longer needed.

This reverts commit 891e8f72ce.

Change-Id: Ia94a2690ff149427fdcafacd39f5008cd60827d5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353774
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-12-02 22:40:06 +00:00
LaMont Jones
d793553804 sync: mark REPO_AUTO_GC=1 as deprecated.
REPO_AUTO_GC was introduced as a way for users to restore the previous
default behavior, since the default changed at the same time as the
option was added.  As such, it should be marked as deprecated, and
removed entirely in a future release.

Change-Id: Ib73d98fbea693e7057cc4587928c225a9e4beab2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353734
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-12-02 22:33:11 +00:00
LaMont Jones
bee4efb874 subcmds: display correct path multitree messages
Correct usage of project.relpath for multi manifest workspaces.

Change-Id: Idc32873552fcdae6eec7b03dde2b2f31134b72fd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/347534
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-11-15 21:13:06 +00:00
LaMont Jones
b240d28bc0 upload: track projects by path, rather than name
Since the same project can be checked out in multiple paths, we need to
track the "to be uploaded" projects by path, rather than project name.

Bug: crbug.com/gerrit/16260
Test: manual
Change-Id: Ic3dc81bb8acb34886baa6299e90a49c7ba372957
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/351054
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-11-14 21:58:10 +00:00
LaMont Jones
5ed8c63942 sync: REPO_AUTO_GC=1 to restore old behavior.
Add an environment variable to restore previous behavior, since the
older version of repo does not support `--auto-gc`.

Change-Id: I874dfb8fc3533a97b8adfd52125eb3d1d75e2f3c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/351194
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-11-10 00:44:33 +00:00
LaMont Jones
7efab539f0 sync: no garbage collection by default
Adds --auto-gc and --no-auto-gc (default) options to control sync's
behavior around calling `git gc`.

Bug: b/184882274
Change-Id: I4d6ca3b233d79566f27e876ab2d79f238ebc12a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344535
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-11-08 19:54:20 +00:00
Woody Lin
859d3d9580 GitcInit: fix gitc-init failure
Aligns argument usage of refactored GitcManifest (8c1e9cbef
"manifest_xml: refactor manifest parsing from client management") to fix
the `repo gitc-init` error: `fatal: manifest_file must be abspath`.

Change-Id: I1728032cce3f39ed1077bbb7ef714410c2c49e1a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350374
Tested-by: Woody Lin <woodylin@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2022-11-04 17:30:40 +00:00
LaMont Jones
fa8d939c8f sync: clear preciousObjects when set in error.
If this is a project that is not using object sharing (there is only one
copy of the remote project) then clear preciousObjects.

To override this for a project, run:

  git config --replace-all repo.preservePreciousObjects true

Change-Id: If3ea061c631c5ecd44ead84f68576012e2c7405c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350235
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-11-03 23:01:16 +00:00
Joanna Wang
a6c52f566a Set tracing to always on and save to .repo/TRACE_FILE.
- add `--trace_to_stderr` option so stderr will include trace outputs and any other errors that get sent to stderr
- while TRACE_FILE will only include trace outputs

piggy-backing on: https://gerrit-review.googlesource.com/c/git-repo/+/349154

Change-Id: I3895a84de4b2784f17fac4325521cd5e72e645e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350114
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Joanna Wang <jojwang@google.com>
2022-11-03 21:07:07 +00:00
LaMont Jones
b750b48f50 init: add --manifest-depth for shallow manifest clone
People rarely care about the history of the manifest repo.  Add a
parameter to specify depth for the manifest.

For now, make the default behavior the same as the current behavior.  At
a future date, the default will be changed to 1.  People who need the
full history should begin passing --manifest-depth=0 to preserve the
behavior when the default changes.

We can't reuse the existing --depth option because that applies to
all projects we clone, not just the manifest repo.

Bug: https://crbug.com/gerrit/16193, https://crbug.com/gerrit/16358
Change-Id: I9130fed3eaed656435c778a85cfe9d04e3a4a6a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349814
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2022-10-27 21:59:09 +00:00
LaMont Jones
6c8b894d8d Revert "init: change --depth default to 1 for manifest repo"
This reverts commit 076d54652e.

Reason for revert: crbug.com/gerrit/16358

Change-Id: I2970eb50677cca69786f71edffe4aa5271cf139f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349834
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2022-10-27 21:52:02 +00:00
LaMont Jones
b6cfa09500 sync: uninitialized variable on mirror sync failure
When repo sync fails, if the workspace is a mirror, an uninitialized
variable is referenced.

Bug: crbug.com/gerrit/16356
Change-Id: I1dba9f92319b9cbfd18460327560a395c88a089f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349654
Reviewed-by: Sam Saccone <samccone@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-10-26 23:13:02 +00:00
LaMont Jones
78dcd3799b sync: do not require python 3.9
Use pre-3.9 syntax for NamedTuple, so that users do not need to have
python 3.9 or later installed.

Bug: b/255632143, crbug.com/gerrit/16355
Test: manually verified with python 3.8
Change-Id: I488d2d5267ed98d5c55c233cc789e629f1911c9d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349395
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2022-10-25 22:46:47 +00:00
LaMont Jones
acc4c857a0 sync: only use --cruft when git supports it.
git gc --cruft was added in 2.37.0.

Bug: https://crbug.com/gerrit/16270
Change-Id: I71e46741e33472a92f16d6f11c51a23e1e55d869
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/346577
Reviewed-by: Emily Shaffer <emilyshaffer@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-09-22 19:18:48 +00:00
LaMont Jones
1eddca8476 sync: use namedtuples for internal return values
Replace tuple returns with namedtuples, to simplify adding new fields.

Extend the Sync_NetworkHalf return value to:
 - success: True if successful (the former return value)
 - remote_fetched: True if we called `git fetch`

Change-Id: If63c24c2f849523f77fa19c05bbf23a5e9a20ba9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344534
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-09-19 22:03:18 +00:00
LaMont Jones
aefa4d3a29 sync: incorporate review feedback.
This incorporates feedback from
https://gerrit-review.googlesource.com/c/git-repo/+/345114

Change-Id: I04433d6435b967858f1ffb355217d90bc48c1e5d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/345894
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-09-19 22:03:18 +00:00
Shashank Devaraj
4ba29c42ca diffmanifests: Handle Missing Projects in Repo Workspace
By default there are 4 categories in the diffmanifests
api puts the diffs in to - added, removed, changed and unreachable

Example of command - repo diffmanifests 1.xml 2.xml

added - list down the projects present in second manifest but not in
first
removed - list down the projects present in first but not in
second
changed - list down the changes and the differences for each project
unreachable - when it encounters revision value in a project is incorrect

But, when there are projects present in both manifests and could not
find in local workspace where we have cloned the repo(because of
different/subset manifest xml) - this will create unhandled exception

Now we have added a 5th category called 'missing' - where in such
cases it will handle the scenario and print the log for user

Example:
added projects :
        project_2 at revision e6c8a59832c05dc4b6a68cee6bc0feb832181725

removed projects :
        project_1 at revision e6c8a59832c05dc4b6a68cee6bc0feb832181725

changed projects :
        project_3 changed from 3bb890e1286f04e84d505e5db48e0ada89892331 to e434b3736f11537c67590fefadfe4495895e9785

missing projects :
        project_4

Change-Id: I244e8389bff7e95664c29d3dcb61e22308e3a573
Signed-off-by: Shashank Devaraj <shashankkarthik@gmail.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344774
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-09-15 17:42:08 +00:00
LaMont Jones
891e8f72ce sync: save any cruft after calling git gc.
This is ENABLED BY DEFAULT due to data corruption potential.  To disable
it, set REPO_BACKUP_OBJECTS=0 in the environment.

While the workspace will grow over time, this provides a recovery path
for an issue where objects are erroneously deleted from the workspace,
resulting in lost work.  Once the root cause is determined, we will be
able to stop saving backups again.

Backups are kept in .git/objects/.repo/pack.bak

Bug: https://crbug.com/gerrit/16247
Change-Id: Ib8b5c9b4bf0dfa9e29606e0f5c881d65996b2a40
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/345114
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2022-09-09 19:04:30 +00:00
LaMont Jones
4112c07688 sync: Correctly sync multi manifest workspaces
When actually fetching the manifests, start at the correct (sub)
manifest.

Bug: https://crbug.com/gerrit/16198
Change-Id: I39fdd726f1917ef4277a0b7c83663c8f49167466
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343914
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-08-24 19:35:38 +00:00
Mike Frysinger
3d27c71dd9 init: hide identify spam when reinitializing
We don't want to keep showing the user config notice when reinitializing
existing checkouts, so hide it.

Change-Id: Id40610bd683396cbff7e1aefc092c8b77c397783
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343536
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-08-22 21:01:35 +00:00
Mike Frysinger
488d54d4ee init: show a notice when reinitializing
Make it clear to users when we're reinitializing an existing checkout
in case they weren't expecting it.

Bug: https://crbug.com/gerrit/12396
Change-Id: I22e89ae041a8e7b147c9d06a82f1302dd5807ae0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343535
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-08-22 21:01:10 +00:00
Mike Frysinger
5a5cfce1b2 stage: add missing flush before project prompt
Bug: https://crbug.com/gerrit/13223
Change-Id: Ib279d86a52e1035e02d6f7d8f053c3a43e721032
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343555
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-08-22 20:48:29 +00:00
Mike Frysinger
e6d4b84060 upload: respect --yes with large upload confirmation
If the user passes in --yes, don't prompt them to confirm large uploads.

Bug: https://crbug.com/gerrit/14085
Change-Id: Ic801b21be80ba181801531acd4af5057ec10c11c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343554
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-08-22 20:48:27 +00:00
Mike Frysinger
8a54a7eac3 cherry-pick: tighten up output
If stdout or stderr are empty, don't print empty lines.
Also trim any trailing lines so we don't show excess ones.

Change-Id: I65fcea191e1f725be03c064cb510578632466013
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343516
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-08-22 19:38:32 +00:00
Mike Frysinger
076d54652e init: change --depth default to 1 for manifest repo
People rarely care about the history of the manifest repo.  Change
the default to 1 to speed up initial setup.  If people really want
the full history, they can pass --manifest-depth=0.

We can't reuse the existing --depth option because that applies to
all projects we clone, not just the manifest repo.

Bug: https://crbug.com/gerrit/16193
Change-Id: Ideab1712e9ffc743b9bae97903e074d7d429b134
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343435
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-08-18 19:12:21 +00:00
Mike Frysinger
7fa8eedd8f upload: add --push-options tips & doc link
Change-Id: Iee38a80974c53231d1e9f04f7f85b2d0bac96dbb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/342354
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-08-18 16:40:40 +00:00
Mike Frysinger
dede564c3d project: simplify GetRemote a bit
We almost always use self.remote.name when calling self.GetRemote,
so make that the default to simplify the code a bit.

Change-Id: Ifdf6e1370d6b8963b44e6d384b0fac8fa5c4f2ba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343184
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-08-17 16:54:21 +00:00
William Escande
ac76fd3e3a upload: Add ready flag to remove wip
The `--wip` allow to bulk push changed as work-in-progress. This CL
intend to allow the opposite opperation by removing the wip mark on the
CL and set it to be ready to review

Change-Id: If0743c5b14829f77be2def5a8547060d06a5648c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/342214
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: William Escande <wescande@google.com>
2022-08-03 20:17:06 +00:00
Mike Frysinger
5951e3043f sync: handle smartsync HTML responses better
If the server responds with an HTML page, we should show that to the
user instead of crashing with XML errors.

Bug: https://crbug.com/gerrit/15936
Change-Id: I52e6b781c3bb6a6c9f6ecbe2e0907044876cdc8d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337519
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-07-28 17:44:21 +00:00
LaMont Jones
48ea25c6a7 sync: start clearing git settings that disabled pruning
For projects that no longer share their per-project objects directly, we
no longer have to disable the git settings that disable pruning.  See
commit "project: stop directly sharing objects/ between shared projects"
for more details.

Bug: https://crbug.com/gerrit/15553
Change-Id: Ica0e83c3002716424c2bc9111b3b3d3a76c30973
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337535
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-07-25 17:13:20 +00:00
Mike Frysinger
355f4398d8 sync: rework --jobs to provide better defaults
For --jobs-network, the logic is now:
* If the user specifies --jobs-network, use that.
* Else, if the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to 1.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.

For --jobs-checkout, the logic is now:
* If the user specifies --jobs-checkout, use that.
* Else, if the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to DEFAULT_LOCAL_JOBS which is based on user's ncpus.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.

For garbage collecting, the logic is now:
* If the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to the user's ncpus.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.

Having to factor in the manifest settings makes this more complicated
which is why we delay processing of defaults until after we've synced
the manifest projects.

Bug: http://b/239712300
Change-Id: Id27cda63c76c156f1d63f6a20cb2c4ceeb3d547c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/341394
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-07-25 15:36:43 +00:00
Mateus Azis
bddc964d93 Fix the printed path of the ".repo" dir after a fresh init.
Apparently, manifest.topdir already contains a trailing slash in some
cases, so a simple string concatenation may or not lead to double
slashes. It is safer to use os.path.join. See
https://screenshot.googleplex.com/6pSY3QewAeCdAqk

Change-Id: I2411452296b7e78fc975787b675273a48d6b3d85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/341574
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mateus Azis <azis@google.com>
2022-07-25 15:22:14 +00:00
LaMont Jones
8501d4602a status, diff: display correct path for multi-manifest
Display the project path relative to the outermost manifest by default,
and relative to the sub manifest only when --this-manifest-only is
specified.

For project-related diagnostic messages, use the outermost manifest for
messages.

Change-Id: I4537d7dd412a2c182e77d6720e95c1b0ef70eb0e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340754
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-07-14 16:00:18 +00:00
Martin Geisler
9fb64ae29c upload: add ‘--ignore-untracked-files’ option
This option will suppress the

    Uncommitted changes in ... (did you forget to amend?)

prompt when there are untracked (unknown) files in the working copy.
The prompt is still shown if tracked files are modified.

Change-Id: Ia3fcc82989b7fad09b69214eda31e2d0dfc14600
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340456
Tested-by: Martin Geisler <mgeisler@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-07-11 17:57:43 +00:00
LaMont Jones
ff6b1dae1e Only sync superproject if it will be used.
If the user says `--no-use-superproject`, then do not bother syncing the
superproject.

Also add/update docstrings and comments throughout.

Change-Id: I9cdad706130501bab9a22d3099a1dae605e9c194
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/338975
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-06-08 16:49:08 +00:00
LaMont Jones
bdcba7dc36 sync: add multi-manifest support
With this change, partial syncs (sync with a project list) are again
supported.

If the updated manifest includes new sub manifests, download them
inheriting options from the parent manifestProject.

Change-Id: Id952f85df2e26d34e38b251973be26434443ff56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334819
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-05-26 00:03:37 +00:00
Mike Frysinger
3a0a145b0e upload: move label validation to core function
This way we know we don't need to encode the labels.

Change-Id: Ib83ed8f4ed05f00b9d2d06a9dd3f304e4443430e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337518
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-05-21 19:19:44 +00:00
LaMont Jones
501733c2ab manifest: add submanifest.default_groups attribute
When the user does not specify any manifest groups, this allows the
parent manifest to indicate which manifest groups should be used for
syncing the submanifest.

Change-Id: I88806ed35013d13dd2ab3cd245fcd4f9061112c4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335474
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-29 18:42:23 +00:00
LaMont Jones
0de4fc3001 project: Add missing imports
Some imports were missed when moving manifestProject to project.py

Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=15861
Change-Id: Id8fffeaa3f88f344a13b5ab44e5403c7edd98f31
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335554
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
2022-04-21 18:44:26 +00:00
LaMont Jones
a46047a822 sync: refactor use of self.manifest
We need to iterate over multiple manifests, and generally use the
outer_client.manifest for multi-manifest support.  This refactors the
use of self.manifest into a chosen manifest.

Change-Id: I992f21d610c929675e99555ece9c38df4b635839
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334699
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-14 22:24:04 +00:00
LaMont Jones
5fa912b0d1 Stop passing optparse.Values to git_superproject
Make git_superproject independent of the command line by passing
the specific value instead of requiring the caller to have an
optparse.Values object to pass in.

Flag --use-superproject and --archive as incompatible in subcmds/init.py

Change-Id: Ied7c874b312e151038df903c8af4328f070f387c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335135
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-14 22:23:16 +00:00
LaMont Jones
d8de29c447 forall: fix multi-manifest variables.
- REPO_PATH is relative to the root of the client. REPO_OUTERPATH is not
  needed.
- REPO_INNERPATH is relative to the sub manifest root.
- REPO_OUTERPATH is the path for the sub manifest root relative to the
  root of the client.

Change-Id: I031692891cfef2634d1358584d27a6a4df735c20
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334899
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-04-14 14:31:47 +00:00
LaMont Jones
d56e2eb421 manifest_xml: use Superproject to hold XML content
Always create Superproject when there is a <superproject> tag, and have
it hold the XML content, similar to how other manifest elements are
handled.

This also adds SetQuiet and SetPrintMessages to Superproject
consistent with manifest.SetUseLocalManifests.

Change-Id: I522bf3da542006575799f0640c67f7052704f266
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334641
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-04-12 15:46:23 +00:00
Daniel Andersson
d52ca421d5 sync: respect sync-c manifest option
The documentation states that a `sync-c` attribute in the manifest file
can set a default for whether only the current branch should be fetched
or all branches. This seems to have been broken for some time.

Commit 7356114 introduced the `--no-current-branch` CLI option and
relied on getting `None` via `optparse` if neither `--current-branch`
nor `--no-current-branch` was set to distinguish it from a boolean
value. If `None` was received, it would read the value from the manifest
option `sync-c`. The parsing went through the utility function
`_GetCurrentBranchOnly` which returned `True` if `--current-branch` had
been given on the command-line, or fell back on the "superproject"
setting, which would either return `True` or `None`. This would
incorrectly make `repo` fall back to the manifest setting even if the
user had given `--no-current-branch` if no superproject was requested --
the manifest became "too powerful":

Command-line         Using superproject  → `current_branch_only`
------------         ------------------  -----------------------
                     No                  From manifest
                     Yes                 True
--current-branch     No                  True
--current-branch     Yes                 True
--no-current-branch  No                  From manifest ← wrong
--no-current-branch  Yes                 True

In commit 0cb6e92 the superproject configuration value reading changed
from something that could return `None` to something that always
returned a boolean. If it returned `False`, this would then incorrectly
make `repo` ignore the manifest option even if neither
`--current-branch` nor `--no-current-branch` had been given. The
manifest default became useless:

Command-line         Using superproject  → `current_branch_only`
------------         ------------------  -----------------------
                     No                  False ← wrong
                     Yes                 True
--current-branch     No                  True
--current-branch     Yes                 True
--no-current-branch  No                  False
--no-current-branch  Yes                 True

By swapping the order in which the command-line option target and the
superproject setting is evaluated, things should work as documented:

Command-line         Using superproject  → `current_branch_only`
------------         ------------------  -----------------------
                     No                  From manifest
                     Yes                 True
--current-branch     No                  True
--current-branch     Yes                 True
--no-current-branch  No                  False
--no-current-branch  Yes                 True

Change-Id: I933c232d2fbecc6b9bdc364ebac181798bce9175
Tested-by: Daniel Andersson <daniel.r.andersson@volvocars.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334270
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-08 21:06:37 +00:00
LaMont Jones
a2ff20dd20 manifest_xml: Add Load and Unload methods
- do not call the internal method from subcmds/sync.py.
- use the correct default groups for submanifests.
- only sync the superproject when we are told to.

Change-Id: I81e4025058f1ee564732b9e17aecc522f6b5f626
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334639
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-04-08 19:52:04 +00:00
LaMont Jones
55ee304304 Fix sub manifest handling
Also fixes some typos

Change-Id: Id2ba5834ba3a74ed3f29c36d2c0030737dc63e35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334579
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-06 21:04:46 +00:00
LaMont Jones
409407a731 init: add multi-manifest support
This moves more of the manifest project handling into ManifestProject.

Change-Id: Iecdafbec18cccdfd8e625753c3bd1bcddf2b227f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334520
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-06 17:02:40 +00:00
LaMont Jones
d82be3e672 Move manifest config logic into ManifestProject
Use ManifestProject properties for config values.

Change-Id: Ib4ad90b0d9a089916e35615b8058942e6d01dc04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334519
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-06 16:59:45 +00:00
LaMont Jones
9b03f15e8e project: add ManifestProject.Sync()
Move the logic to sync a ManifestProject out of subcmds/init.py

Change-Id: Ia9d00f3da1dc3c5dada84c4d19cf9802c2346cb0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334140
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-01 15:48:04 +00:00
LaMont Jones
cc879a97c3 Add multi-manifest support with <submanifest> element
To be addressed in another change:
 - a partial `repo sync` (with a list of projects/paths to sync)
   requires `--this-tree-only`.

Change-Id: I6c7400bf001540e9d7694fa70934f8f204cb5f57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322657
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-02-17 21:57:55 +00:00
XD Trol
630876f9e4 init: add an option --enable-git-lfs-filter
It was reported that git-lfs did not work with git-repo. Specifically,
`git read-tree -u` run by `repo sync` would fail git-lfs's smudge
filter. See https://github.com/github/git-lfs/issues/1422.

In fact, by the time `git read-tree -u` is run, the repository is not
bare. It is just that, the working directory is not the same as the
.git directory. git-lfs's filter should work. No one seems to have
delved into that issue.

Today, with newer versions of git-repo and git-lfs, that issue will
not reproduce. Tested with
- git 2.33, git-lfs 2.13 on macOS
- git 2.17, git-lfs 2.3 on ubuntu

So, it seems fine to add an option --enable-git-lfs-filter, default to
false, and stat that it may not work with older versions of git and
git-lfs in the help doc.

Bug: https://crbug.com/gerrit/14516
Change-Id: I8d21854eeeea541e072f63d6b10ad1253b1a9826
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328359
Tested-by: XD Trol <milestonejxd@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-01-26 01:47:20 +00:00
Mike Frysinger
4aa8584ec6 init: make bad --repo-rev settings more clear
If the user passes a bad --repo-rev setting in a new checkout, add a
tip to the error message that their option is probably bad instead of
just saying "unable to resolve".

If the user has already initialized a checkout, we'd display a raw
traceback which would confuse them.  Swallow that and also include
the --repo-rev tip.

Bug: https://crbug.com/gerrit/15610
Change-Id: I5d72513c7b37bf9bb5d19862fcdfaf0d1f44e886
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328820
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-25 18:54:42 +00:00
LaMont Jones
a535ae4418 branches: Fix "not in" handling
If the branch is current, or present in less than half of the projects,
list which projects it is *in*.

Otherwise, correctly detect which projects (by relpath) it is not in.

Previously, the "not in" path would incorrectly list all projects.

Change-Id: Ia153856f577035a51f538b7bf5d3135b70c69d52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328199
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2022-01-21 17:20:21 +00:00
Raman Tenneti
cf0ba48649 sync: With --mirror option, don't display no-use-superproject... message.
+ Display 'Defaulting to no-use-superproject because there is no working tree.'
  message if --use-superproject option is used and we are not using
  superproject because manifest is either a mirror or is an archive.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL.

$ repo init -u https://android.googlesource.com/mirror/manifest --mirror

$ repo_dev sync
Receiving objects: 100% (3/3), done.eiving objects:  33% (1/3)

$ repo_dev sync --use-superproject
Defaulting to no-use-superproject because there is no working tree.
Fetching:  0% (0/2158) warming up

Bug: https://crbug.com/gerrit/15368
Change-Id: I16b87ee9623315dbc3100b612b1decdaab7ac1dc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/325797
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-12-07 16:46:41 +00:00
Raman Tenneti
4a478edb44 init, sync: fixed flake8 warnings.
Tested:
+ run_tests
+ flake8 subcmds/init.py
+ flake8 subcmds/sync.py

Change-Id: Ie337481d8a210bfc49b0745f75c05a308a0e74d3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/324155
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-11-18 16:22:40 +00:00
Raman Tenneti
6bd89aa657 superproject: Inherit --no-use-superproject with --mirror option.
init.py
+ Similar to opt.archive, gave an error if --mirror option is
  used with --use-superproject.

sync.py
+ Defaulted to --no-use-superproject if manifest is a mirror or
  archive (similar to error at line# 1067).

Tested:
+ run_tests
+ flake8 (will fix known errors in another CL).

$ repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest --use-superproject --mirror
Usage: repo init [options] [manifest url]

main.py: error: --mirror and --use-superproject cannot be used together.

+ repo init and repo sync with --mirror and without --mirror
  options.
  $ repo_dev init -u https://android.googlesource.com/platform/manifest
  $ repo_dev sync
    ...superproject.git: Initial setup for superproject completed.

+ With --mirror option, verfied there are no exceptions in git_superproject.py

Bug: [google internal] b/206537893
Change-Id: I059f20e76f0ab36f0587f29779bb53ede4663bd4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323955
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-11-18 01:27:41 +00:00
Mike Frysinger
fdeb20f43f sync: link the internal-fs-layout doc into checkouts
This should make it easy to discover for people poking around .repo/.

Change-Id: Ie5051551f25127c0592df5e36efba7bb2263e5d4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323701
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-11-15 01:39:53 +00:00
Mike Frysinger
3001d6a426 help: fix grammar in help text
Bug: https://crbug.com/gerrit/14838
Change-Id: Ic5000921ba9a1baa086153630ebbb429e3d17642
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323515
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-11-12 17:30:32 +00:00
Mike Frysinger
0531a623e1 sync: make --prune the default
If a remote deletes a ref, and it points to an object that doesn't
exist locally, we can get into a bad state, and the only way for the
user to recover is to run `repo sync --prune` (and to know that is
the option they need).  The error message is not helpful:

fatal: bad object refs/remotes/cros/firmware-zork-13421.B-master
error: https://chromium.googlesource.com/chromiumos/platform/ec did not send all necessary objects

This situation can also come up when the remote renames refs in a
UNIX FS incompatible way.  For example, replacing refs/heads/foo
with refs/heads/foo/bar.

Also add a --no-prune option for users to disable the behavior.

Bug: https://issuetracker.google.com/203366450
Change-Id: Icf45d838a10938feb091d29800f7e49240830ec3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322956
Reviewed-by: Andrew Lamb <andrewlamb@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-11-05 20:13:30 +00:00
Mike Frysinger
2273f46cb3 sync: fix --tags option
This has been broken since it was added where --tags was actually
the same as --no-tags.  Oddly, it was copied from init where the
logic is correct.

Bug: https://crbug.com/gerrit/12401
Change-Id: I15b89da1a655176a11bebc22573b25c728055328
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322955
Reviewed-by: Andrew Lamb <andrewlamb@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-11-05 20:13:02 +00:00
Raman Tenneti
6251729cb4 superproject: added 'implies -c' in the help of --use-superproject option.
sync.py: deleted unused import errno.

Tested:
$ ./run_tests
$ flake8 repo subcmds/sync.py

Bug: https://crbug.com/gerrit/15208
Change-Id: I2bb3098f5602ded3861e000100766041ad93b53d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322555
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-11-01 22:15:53 +00:00
Jack Neus
198838599c fetch: Fix stderr handling for gsutil
Previously gsutil stderr was getting piped into stdout, which
yields bad results if there are non-fatal warnings in stderr.

Additionally, we should fail outright if gsutil fails (by adding
`check = True`) rather than fail later on when we try to sync to
a manifest that is in fact just a stderr dump.

BUG=none
TEST=manual runs with bad gs urls

Change-Id: Id71791d0c3f180bd0601ef2c783a8e8e4afa8f59
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/321935
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-26 22:18:28 +00:00
Jack Neus
03ff276cd7 sync: properly handle standalone manifests for sync command
sync should not attempt to sync the manifest project if it was
created from a standalone manifest. The current work around is to
run sync with --nmu.

BUG=none
TEST=manual runs

Change-Id: I2e121af0badf9642143e77c7af89d1c2d993b0f3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/321195
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-15 17:20:00 +00:00
Allen Webb
4ee4a45d03 subcmds/sync: Use pack-refs instead of gc for redundant gitdirs.
Previously `git gc` was being run on every gitdir even when they shared
the same objects. Instead only call it once and use pack-refs for the
gitdirs that were not gc'ed.

Bug: https://crbug.com/gerrit/15113
Test: repo sync -j # and check that git pack-refs is called
Change-Id: Icff37ab3ec78cfb44391d8cc7f2d875991532320
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/320275
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-14 12:27:12 +00:00
Jack Neus
0f6f16ed17 repo: more arg checking for --standalone-manifest re-inits
`repo init` doesn't do anything on re-init when the checkout has
been initialized using --standalone manifest. Rather than let the
tool run through its existing flows (which happen to noop), check
the args and explicitly quit if a bare `repo init` is run on a
standalone checkout.

BUG=none
TEST=manual tests

Change-Id: Ie4346ef6df1282ec3e3f8045a08138c93653fece
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/320735
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-11 18:58:11 +00:00
Jack Neus
76491590b8 repo: fix bug with --standalone-manifest
We were accidentally always setting manifest.standlone in config,
which was messing up behavior for standard use cases.

BUG=gerrit:15160
TEST=manual runs

Change-Id: Ic80f084ae97de5721aced3bb52d3ea9115f8d833
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/320715
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-11 18:57:57 +00:00
Allen Webb
669efd0fd7 subcmds/sync: Disable autoDetach for git gc.
gc.autoDetach is enabled by default which makes 'git gc --auto' return
immediately and run in background. This can lead to a pile up of
operations all using large amounts of memory at the same time. To avoid
this set gc.autoDetach to false so that the garbage collect task waits
for instances to finish before spawning more.

Bug: https://crbug.com/gerrit/15113
Test: repo sync -j # and check the number of 'git gc' processes
Change-Id: Ic0815156ba3db03972968f33f6f9f51e4928f23b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319835
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-05 14:12:01 +00:00
Mike Frysinger
9d96f58f5f make file removal a bit more robust
Some of the file removal calls are subject to race conditions (if
something else deletes the file), so extend our remove API to have
an option to ignore ENOENT errors.  Then update a bunch of random
call sites to use this new functionality.

Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195
Reviewed-by: Sean McAllister <smcallis@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-09-28 16:06:50 +00:00
Jack Neus
c474c9cba1 repo: Add support for standalone manifests
Added --standalone_manifest to repo tool. If set, the
manifest is downloaded directly from the appropriate source
(currently, we only support GS) and used instead of creating
a manifest git checkout. The manifests.git repo is still created to
keep track of various config but is marked as being for a standalone
manifest so that the repo tool doesn't try to run networked git
commands in it.

BUG=b:192664812
TEST=existing tests (no coverage), manual runs

Change-Id: I84378cbc7f8e515eabeccdde9665efc8cd2a9d21
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312942
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-28 15:40:46 +00:00
Raman Tenneti
6448a4f2af sync: Log repo sync state events as 'data' events.
git_trace2_event_log.py:
+ Added LogDataConfigEvents method to log 'data' events.
  Sync's current_sync_state and previous_sync_state are logged
  as 'data' events in the current log.

  It logs are key/value in the |config| argument. Each key is
  prefixed with |prefix| argument.

  The following are sample events that are logged during repo sync.

   {"event":"data",
   "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
   "thread":"MainThread",
   "time":"2021-09-14T18:16:19.935846Z",
   "key":"previous_sync_state/repo.syncstate.main.synctime",
   "value":"2021-09-14T17:27:11.573717Z"}

   {"event":"data",
   "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
   "thread":"MainThread",
   "time":"2021-09-14T18:16:19.955546Z",
   "key":"current_sync_state/repo.syncstate.main.synctime",
   "value":"2021-09-14T18:16:19.935979Z"}

tests/test_git_trace2_event_log.py:
+ Added unit tests

sync.py:
+ Changed logging calls to LogDataConfigEvents.

Tested:
$ ./run_tests

Tested it by running the following command multiple times.
$ repo_dev sync -j 20
  repo sync has finished successfully

  Verified config data is looged in trace2 event logs.

Bug: [google internal] b/199758376
Change-Id: I75fd830e90c1811ec28510538c99a2632b104e85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317823
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-09-14 21:36:12 +00:00
Raman Tenneti
b55769a5c9 superproject: print messages if the manifest has superproject tag.
1) If the manifest has superproject tag (git_master, etc), then
   display error/warning messages (as it is doing today)
2) If the manifest doesn't have superproject tag (nest, chromeos
   manifests), then don't display any error/warning messages about
   superrproject (behave as though user has specified
   --no-use-superproject).
3) Print error/warning messages if --use-superproject passed as
   argument to repo sync.
4) No change in behavior for the repo init command.

git_superproject.py:
+ Fixed typo in _WriteManifestFile method name
+ Superproject accepts print_message  as an argument and it defaults
  to True. All messages that are printed to stderr are controlled by
  this flag. If it is True, then messages get printed.
+ Added PrintMessages function which return true if either
  --use-superproject is specified on the command line or if the
  manifest has a superproject tag.

sync.py:
+ Displays the warning message if PrintMessgages are enabled and
  passes that as argument to superproject object.
+ Added 'hassuperprojecttag' trace2 log entry for analysis. We can
  find users/branches that are using superproject, but the manifest is
  missing the superproject tag.

Tested:
$ ./run_tests

+ Verified printing of messages with and without superproject tag, with
  with --use-superproject option.

+ aosp-master
  $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
  $ repo_dev sync

+ A manifest without superproject tag.
  $ repo_dev init -m $(pwd)/manifest_7482982.xml
  $ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml

Bug: [google internal] b/196411099
Change-Id: I92166dcad15a4129fab82edcf869e7c8db3efd4b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314982
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-08-13 20:07:40 +00:00
Raman Tenneti
7954de13b7 sync: Added logging of repo sync state and config options for analysis.
git_config.py:
+ Added SyncAnalysisState class, which saves the following data
  into the config object.
  ++ sys.argv, options, superproject's logging data.
  ++ repo.*, branch.* and remote.* parameters from config object.
  ++ current time as synctime.
  ++ Version number of the object.
+ All the keys for the above data are prepended with 'repo.syncstate.'
+ Added GetSyncAnalysisStateData and UpdateSyncAnalysisState methods
  to GitConfig object to save/get the above data.

git_trace2_event_log.py:
+ Added LogConfigEvents method with code from DefParamRepoEvents
  to log events.

sync.py:
+ superproject_logging_data is a dictionary that collects all the
  superproject data that is to be logged as trace2 event.
+ Sync at the end logs the previously saved syncstate.* parameters
  as previous_sync_state. Then it calls config's UpdateSyncAnalysisState
  to save and log all the current options, superproject logged data.

docs/internal-fs-layout.md:
+ Added doc string explaining [repo.syncstate ...] sections of
  .repo/manifests.git/config file.

test_git_config.py:
+ Added unit test for the new methods of GitConfig object.

Tested:
$ ./run_tests

$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest

Tested it by running the following command multiple times.
$ repo_dev sync -j 20
  repo sync has finished successfully

  Verified config file has [syncstate ...] data saved.

Bug: [google internal] b/188573450
Change-Id: I1f914ce50f3382111b72940ca56de7c41b53d460
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313123
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2021-07-29 19:20:57 +00:00
Raman Tenneti
ae86a46022 superproject: Skip updating of superproject when -l is used with sync.
Skip updating the superproject when -l is present and use the existing
superproject, if available (this would make sync -l work as it's
intended to do), and fall back to sync without superproject when not
(this would catch the case when superproject is enabled by automatic
rollout).

Tested:
$ repo sync -j 20 -n
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/android/src/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Fetching: 100% (1032/1032), done in 41.184s
...

$ repo_dev sync -j 20 -l
prebuilts/asuite/: discarding 1 commits
prebuilts/runtime/: discarding 1 commits
...
repo sync has finished successfully.

+ With superproject-override.xml and test it.

  $ ls -l .repo/exp-superproject/
  total 176
  drwxr-xr-x 7 rtenneti primarygroup   4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
  -rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 superproject_override.xml
  rtenneti@rtenneti:~/work/android/src/aosp$ repo_dev sync -j 20 -l
  ...
  repo sync has finished successfully.

+ Rename the file superproject-override.xml and test it.
  $ ls -l .repo/exp-superproject/
  total 176
  drwxr-xr-x 7 rtenneti primarygroup   4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
  -rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 temp.xml

  $ repo_dev sync -j 20 -l
  Checking out:  1% (12/1031) platform/external/rust/crates/fallible-streaming-iteexternal/linux-kselftest/: discarding 1 commits
  prebuilts/remoteexecution-client/: discarding 1 commits
  Checking out: 51% (536/1031) platform/prebuilts/gcc/darwin-x86/aarch64/....
  ....
  Checking out: 100% (1031/1031), done in 5.478s
  repo sync has finished successfully.

Bug: [google internal] b/184368268
Change-Id: I3aba5872e4f7c299977b92c2a39847ef28698c5a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312962
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-07-28 16:12:53 +00:00
Mike Frysinger
56345c345b repo: refactor help output handling
Currently we have the behavior:
* `repo`: Equivalent to `repo help` -- only shows common subcommands
  (with short description), and then exits 0.
* `repo --help`: Shows repo's core options, lists all commands (no
  specific info), and then exits 0.

The first case is not behaving well:
* If you run `repo` without a specific subcommand, that's an error,
  so we should be exiting 1 instead.
* Showing only subcommands and no actual option summary makes it seem
  like repo itself doesn't take any options.  This confuses users.

Let's rework things a bit.  Now we have the behavior:
* `repo`: Shows repo's core options, lists all commands (no specific
  info), and then exits 1.
* `repo --help`: Shows repo's core options, shows common subcommands
  (with short description), and then exits 0.
* `repo --help-all`: Shows repo's core options, shows all subcommands
  (with short description), and then exits 0.

Basically we swap the behavior of `repo` and `repo --help`, and fix
the exit status when the subcommand is missing.

The addition of --help-all is mostly for the man pages.  We were
relying on `repo help --all` to generate the repo(1) man page, but
that too omitted the core repo options.  Now the man page includes
all the core repo options and provides a summary of all commands.

Change-Id: I1f99b99d5b8af2591f96a078d0647a3d76d6b0fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312908
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-07-28 05:38:34 +00:00