Partial syncs are not supported and can lead to strange behavior like
deleting files. Explicitly warn users on partial sync.
Bug: b/286126621, b/271507654
Change-Id: I471f78ac5942eb855bc34c80af47aa561dfa61e8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/382154
Reviewed-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Per discussion in go/repo-error-update updated aggregated and exit
errors for sync command.
Aggregated errors are errors that result in eventual command failure.
Exit errors are errors that result in immediate command failure.
Also updated main.py to log aggregated and exit errors to git sessions
log
Bug: b/293344017
Change-Id: I77a21f14da32fe2e68c16841feb22de72e86a251
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/379614
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Commit-Queue: Jason Chang <jasonnc@google.com>
In order to better analyze and track repo errors, repo command failures
need to be tied to specific errors in repo source code.
Additionally a new GitCommandError was added to differentiate between
general git related errors to failed git commands. Git commands that opt
into verification will raise a GitCommandError if the command failed.
The first step in this process is a general error refactoring
Bug: b/293344017
Change-Id: I46944b1825ce892757c8dd3f7e2fab7e460760c0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/380994
Commit-Queue: Jason Chang <jasonnc@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Save the latest time any project is fetched and checked out. This will
be used to detect partial checkouts.
Bug: b/286126621
Change-Id: I53b264dc70ba168d506076dbd693ef79a696b61d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/380514
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
New vs existing project may be a useful measure for analyzing
sync performance.
Bug: b/287105597
Change-Id: Ibea3e90c9fe3d16fd8b863bcae22b21963a6771a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/377574
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
By chance, _sync_dict can be empty even though repo sync is still
working. In that case, the progress message shows incorrect info. Handle this case and fix a bug where "0 jobs" can show.
Bug: http://b/284465096
Change-Id: If915d953ba60e7cf84a6fb2d137fd6ed82abd3cc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375494
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
It's possible that number of jobs is more than 0 when we
check length, but in the meantime number of jobs drops to
0. In that case, we are working with float(inf) which
causes other problems
Bug: 284383869
Change-Id: I5d070d1be428f8395df7fde8ca84866db46f2100
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375134
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
An investigation go/git-repo-shallow shows a number of problems
when doing a shallow git fetch/clone. This change introduces an
environment variable REPO_ALLOW_SHALLOW. When this environment variable
is set to 1 during a repo init or repo sync all shallow git fetch
commands are replaced with partial fetch commands. Any shallow
repository needing update is unshallowed. This behavior continues until
a subsequent repo sync command is run with REPO_ALLOW_SHALLOW set to 1.
Bug: b/274340522
Change-Id: I1c3188270629359e52449788897d9d4988ebf280
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374754
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Last of the recent `repo sync` UX changes. Show number of fetch jobs eg:
"Fetching: 3% (8/251) 0:03 | 8 jobs | 0:01 chromiumos/overlays/chrom.."
Bug: https://crbug.com/gerrit/11293
Change-Id: I1b3dcf3e56ae6731c6c6cb73cfce069b2f374b69
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374920
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Change-Id: I91b99a7147c7c3cb5485d5406316c8ffd79f9272
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374914
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
"Last synced: X" is printed only after a project finishes syncing.
Replace that with a message that shows the longest actively syncing
project.
Bug: https://crbug.com/gerrit/11293
Change-Id: I84c7873539d84999772cd554f426b44921521e85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/372674
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Trace logs emitted from repo are not useful on error for many critical
commands. This change adds errors for critical commands to trace logs.
Change-Id: Ideb9358bee31e540bd84a94327a09ff9b0246a77
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/373814
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Give users an indication that `repo sync` isn't stuck if taking a long
time to fetch.
Bug: https://crbug.com/gerrit/11293
Change-Id: Iccdaec918f86c9cc2db5dc12f9e3eef7ad0bcbda
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/371414
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Apply rules set by https://gerrit-review.googlesource.com/c/git-repo/+/362954/ across the codebase and fix any lingering errors caught
by flake8. Also check black formatting in run_tests (and CQ).
Bug: b/267675342
Change-Id: I972d77649dac351150dcfeb1cd1ad0ea2efc1956
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/363474
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
dest_branch.
- This still prevents the case mentioned here:
https://gerrit-review.googlesource.com/c/50300
while also supporting dest_branch.
- Update _GetMergeBranch to get merge branches for any branch, not just
the one we happen to run `repo upload` in. (e.g. for uploading multiple
branches)
Bug: b/27955930
Change-Id: Ia8ee1d6a83a783c984bb2eb308bb11b3a721a95d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/360794
Commit-Queue: Joanna Wang <jojwang@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Joanna Wang <jojwang@google.com>
If interrupt signal is sent to repo process while sync is running, repo
prints stack trace for each concurrent job that is currently running
with no useful information.
Instead, this change captures KeyboardInterrupt in each process and
prints one line about current project that is being processed.
Change-Id: Ieca760ed862341939396b8186ae04128d769cd56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357135
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
repo sync progress bar is misleading. Many bug reports mentioned that
repo is stuck at the repo that is currently displayed in the progress
bar. Repo sync actually shows what repository is the last processed.
This change makes that obvious.
Change-Id: I962bf0bc65af7ac0ed98db86e9144f07d9e1f96f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357134
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
This can potentially show up when sync'ing projects with submodules
that are not declared in the manifest as well as the internal
'.repo/repo' project, which is likely not desirable from a user
standpoint.
Change-Id: I93d7fcd6e3fd1818357ea4537882a864dea9942c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/355920
Reviewed-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
Resolves multiple TODOs. Since we are requiring Python 3,
we move to using print function with flush=True instead of
using sys.stdout.flush().
Change-Id: I54db0344ec78ac81a8d6c6c7e43ee7d301f42f02
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354701
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Add a per-remote option `uploadwarningthreshold` allowing the user to
override how many commits can be uploaded prior to a warning being
displayed.
Change-Id: Ia7e1b2c7de89a0bf9ca1c24cc83dc595b3667437
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354375
Tested-by: David Greenaway <dgreenaway@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Since --jobs defaults to 0, not None, we never pull the value out
of the manifest. Treat values of 0 and None the same to fix.
Bug: http://b/239712300
Bug: http://b/260908907
Change-Id: I9b1026682072366616825fd72f90bd90c10a252f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354254
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
If syncing in _FetchOne fails with GitError, sync_result does not get
set. There's already a separate local variable for success; do the same
for remote_fetched instead of referring to the conditionally defined
named tuple.
This bug is originally caused by a combination of ad8aa697 "sync: only
print error.GitError, don't raise that exception." and 1eddca84 "sync:
use namedtuples for internal return values".
Change-Id: I0f9dbafb97f8268044e5a56a6f92cf29bc23ca6a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354176
Tested-by: Karsten Tausche <karsten@fairphone.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
The wrong revision of the change was submitted as
d793553804.
Change-Id: I6f3e4993cf40c30ccf0d69020177db8fe5f76b8c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353934
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
Do not use a progress bar when not doing GC, and restrict activity in
that case to only repairing preciousObject state.
This also includes additional cleanup based on review comments from
previous changes.
Change-Id: I48581c9d25da358bc7ae15f40e98d55bec142331
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353514
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This bug-cacher related code is no longer needed.
This reverts commit 891e8f72ce.
Change-Id: Ia94a2690ff149427fdcafacd39f5008cd60827d5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353774
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
REPO_AUTO_GC was introduced as a way for users to restore the previous
default behavior, since the default changed at the same time as the
option was added. As such, it should be marked as deprecated, and
removed entirely in a future release.
Change-Id: Ib73d98fbea693e7057cc4587928c225a9e4beab2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353734
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Correct usage of project.relpath for multi manifest workspaces.
Change-Id: Idc32873552fcdae6eec7b03dde2b2f31134b72fd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/347534
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Since the same project can be checked out in multiple paths, we need to
track the "to be uploaded" projects by path, rather than project name.
Bug: crbug.com/gerrit/16260
Test: manual
Change-Id: Ic3dc81bb8acb34886baa6299e90a49c7ba372957
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/351054
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Add an environment variable to restore previous behavior, since the
older version of repo does not support `--auto-gc`.
Change-Id: I874dfb8fc3533a97b8adfd52125eb3d1d75e2f3c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/351194
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Adds --auto-gc and --no-auto-gc (default) options to control sync's
behavior around calling `git gc`.
Bug: b/184882274
Change-Id: I4d6ca3b233d79566f27e876ab2d79f238ebc12a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344535
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Aligns argument usage of refactored GitcManifest (8c1e9cbef
"manifest_xml: refactor manifest parsing from client management") to fix
the `repo gitc-init` error: `fatal: manifest_file must be abspath`.
Change-Id: I1728032cce3f39ed1077bbb7ef714410c2c49e1a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350374
Tested-by: Woody Lin <woodylin@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
If this is a project that is not using object sharing (there is only one
copy of the remote project) then clear preciousObjects.
To override this for a project, run:
git config --replace-all repo.preservePreciousObjects true
Change-Id: If3ea061c631c5ecd44ead84f68576012e2c7405c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350235
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
People rarely care about the history of the manifest repo. Add a
parameter to specify depth for the manifest.
For now, make the default behavior the same as the current behavior. At
a future date, the default will be changed to 1. People who need the
full history should begin passing --manifest-depth=0 to preserve the
behavior when the default changes.
We can't reuse the existing --depth option because that applies to
all projects we clone, not just the manifest repo.
Bug: https://crbug.com/gerrit/16193, https://crbug.com/gerrit/16358
Change-Id: I9130fed3eaed656435c778a85cfe9d04e3a4a6a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349814
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
This reverts commit 076d54652e.
Reason for revert: crbug.com/gerrit/16358
Change-Id: I2970eb50677cca69786f71edffe4aa5271cf139f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349834
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
When repo sync fails, if the workspace is a mirror, an uninitialized
variable is referenced.
Bug: crbug.com/gerrit/16356
Change-Id: I1dba9f92319b9cbfd18460327560a395c88a089f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349654
Reviewed-by: Sam Saccone <samccone@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Use pre-3.9 syntax for NamedTuple, so that users do not need to have
python 3.9 or later installed.
Bug: b/255632143, crbug.com/gerrit/16355
Test: manually verified with python 3.8
Change-Id: I488d2d5267ed98d5c55c233cc789e629f1911c9d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349395
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Replace tuple returns with namedtuples, to simplify adding new fields.
Extend the Sync_NetworkHalf return value to:
- success: True if successful (the former return value)
- remote_fetched: True if we called `git fetch`
Change-Id: If63c24c2f849523f77fa19c05bbf23a5e9a20ba9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344534
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
By default there are 4 categories in the diffmanifests
api puts the diffs in to - added, removed, changed and unreachable
Example of command - repo diffmanifests 1.xml 2.xml
added - list down the projects present in second manifest but not in
first
removed - list down the projects present in first but not in
second
changed - list down the changes and the differences for each project
unreachable - when it encounters revision value in a project is incorrect
But, when there are projects present in both manifests and could not
find in local workspace where we have cloned the repo(because of
different/subset manifest xml) - this will create unhandled exception
Now we have added a 5th category called 'missing' - where in such
cases it will handle the scenario and print the log for user
Example:
added projects :
project_2 at revision e6c8a59832c05dc4b6a68cee6bc0feb832181725
removed projects :
project_1 at revision e6c8a59832c05dc4b6a68cee6bc0feb832181725
changed projects :
project_3 changed from 3bb890e1286f04e84d505e5db48e0ada89892331 to e434b3736f11537c67590fefadfe4495895e9785
missing projects :
project_4
Change-Id: I244e8389bff7e95664c29d3dcb61e22308e3a573
Signed-off-by: Shashank Devaraj <shashankkarthik@gmail.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344774
Reviewed-by: Mike Frysinger <vapier@google.com>
This is ENABLED BY DEFAULT due to data corruption potential. To disable
it, set REPO_BACKUP_OBJECTS=0 in the environment.
While the workspace will grow over time, this provides a recovery path
for an issue where objects are erroneously deleted from the workspace,
resulting in lost work. Once the root cause is determined, we will be
able to stop saving backups again.
Backups are kept in .git/objects/.repo/pack.bak
Bug: https://crbug.com/gerrit/16247
Change-Id: Ib8b5c9b4bf0dfa9e29606e0f5c881d65996b2a40
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/345114
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
