Previously repo would abort a sync if there were published changes not
merged upstream. The --rebase option allows the modification of
published commits.
This is a copy of http://go/grev/369694 with the merge conflicts
resolved.
Bug: 40014610
Change-Id: Idac8199400346327b530abea33f1ed794e5bb4c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/435838
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Jeroen Dhollander <jeroendh@google.com>
Commit-Queue: Jeroen Dhollander <jeroendh@google.com>
The current logic to create checkout layers doesn't work in all cases.
For example, let's assume there are three projects: "foo", "foo/bar" and
"foo-bar". Sorting lexicographical order is incorrect as foo-bar would
be placed between foo and foo/bar, breaking layering logic.
Instead, we split filepaths based using path delimiter (always /) and
then use lexicographical sort.
BUG=b:325119758
TEST=./run_tests, manual sync on chromiumos repository
Change-Id: I76924c3cc6ba2bb860d7a3e48406a6bba8f58c10
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/412338
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: George Engelbrecht <engeg@google.com>
In some cases (e.g. in a CI system), it's desirable to be able to
instruct repo to force checkout. This flag passes --force flag to `git
checkout` operations.
Bug: b/327624021
Change-Id: I579edda546fb8147c4e1a267e2605fcf6e597421
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/411518
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: George Engelbrecht <engeg@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
If a repo manifest is updated so that project B is placed within a
project A, and if project A had content in new B's location in the old
checkout, then repo sync could break depending on checkout order, since
B can't be checked out before A.
This change introduces checkout levels which enforces right sequence of
checkouts while still allowing for parallel checkout. In an example
above, A will always be checked out first before B.
BUG=b:325119758
TEST=./run_tests, manual sync on ChromeOS repository
Change-Id: Ib3b5e4d2639ca56620a1e4c6bf76d7b1ab805250
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/410421
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Greg Edelston <gredelston@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Prior to this change RepoChangedException would be caught and re-rasied
as a different exception. This would prevent RepoChangedException
handler from running in main.py
Bug: b/323232806
Change-Id: I9055ff95d439d6ff225206c5bf1755cc718bcfcc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/407144
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
UpdateManifestError inherits from RepoExitError which inherits
from BaseRepoError. None of them takes project as kwargs
causing the error like "UpdateManifestError() takes no keyword
arguments" in b/317183321
[1]: https://gerrit.googlesource.com/git-repo/+/449b23b698d7d4b13909667a49a0698eb495eeaa/error.py#144
Bug: b/317183321
Change-Id: I64c3dc502027f9dda56a0824f2712364b4502934
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/398997
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Tested-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Jason Chang <jasonnc@google.com>
There are still some verbose messages (e.g. "remote: ...") when doing
repo sync after a couple days. Let's hide them behind verbose flag.
Bug: N/A
Test: repo sync
Change-Id: I1408472c95ed80d9555adfe8f92211245c03cf41
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/400855
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Tomasz Wasilczyk <twasilczyk@google.com>
Commit-Queue: Tomasz Wasilczyk <twasilczyk@google.com>
Most times a repo sync after some time (week+) results in a bunch of
messages, which are not very useful for average user:
- discarding 1 commits
- Deleting obsolete checkout.
Bug: N/A
Test: repo sync
Change-Id: I881eab61f9f261e98f3656c09e73ddd159ce288c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/397038
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Tomasz Wasilczyk <twasilczyk@google.com>
In the case of a project being removed from the manifest, and in the
path in which the project used to exist, and symlink is place to another
project repo will start to warn about partial syncs when a partial sync
did not occur.
Repro steps:
1) Create a manifest with two projects. Project a -> a/ and project b -> b/
2) Run `repo sync`
3) Remove project b from the manifest.
4) Use `link` in the manifest to link all of Project a to b/
Bug: 314161804
Change-Id: I4a4ac4f70a7038bc7e0c4e0e51ae9fc942411a34
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/395640
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Matt Schulte <matsch@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
- Bump minimum version to Python 3.6.
- Use f-strings in a lot of places.
Change-Id: I2aa70197230fcec2eff8e7c8eb754f20c08075bb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/389034
Tested-by: Jason R. Coombs <jaraco@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Jason R. Coombs <jaraco@google.com>
Found via pylint:
W0231: __init__ method from base class 'Transport'
is not called (super-init-not-called)
Just fixed for code correctness and to avoid potential future bugs.
Change-Id: Ie1e723c2afe65d026d70ac01a16ee7a40c149834
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/390676
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Commit-Queue: Daniel Kutik <daniel.kutik@lavawerk.com>
When a new shared project is added to manifest, there's a short window
where objects can be deleted that are used by other projects.
To close that window, set preciousObjects during git init. For
non-shared projects, repo should correct the state in the same execution
instance.
Bug: 288102993
Change-Id: I366f524535ac58c820d51a88599ae2108df9ab48
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/390234
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
In Python 3, these exceptions were merged into OSError, so switch
everything over to that.
Change-Id: If876a28b692de5aa5c62a3bdc8c000793ce52c63
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/390376
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If a KeyboardInterrupt is encountered before an error is aggregated then
the context surrounding the interrupt is lost. This change aggregates
errors as soon as possible for the sync command
Bug: b/293344017
Change-Id: Iac14f9d59723cc9dedbb960f14fdc1fa5b348ea3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/384974
Tested-by: Jason Chang <jasonnc@google.com>
Commit-Queue: Jason Chang <jasonnc@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Google Python style guide says to import modules.
Clean up all our stdlib imports. Leave the repo ones alone
for now as that's a much bigger shave.
Change-Id: Ida42fc2ae78b86e6b7a6cbc98f94ca04b295f8cc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/383714
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The repo project is fetched at most once a day and should be ignored
when checking if the tree is partially synced.
Bug: b/286126621, b/271507654
Change-Id: I684ed1669c3b3b9605162f8cc9d57185bb3dfe8e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/383494
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Partial syncs are not supported and can lead to strange behavior like
deleting files. Explicitly warn users on partial sync.
Bug: b/286126621, b/271507654
Change-Id: I471f78ac5942eb855bc34c80af47aa561dfa61e8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/382154
Reviewed-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Per discussion in go/repo-error-update updated aggregated and exit
errors for sync command.
Aggregated errors are errors that result in eventual command failure.
Exit errors are errors that result in immediate command failure.
Also updated main.py to log aggregated and exit errors to git sessions
log
Bug: b/293344017
Change-Id: I77a21f14da32fe2e68c16841feb22de72e86a251
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/379614
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Commit-Queue: Jason Chang <jasonnc@google.com>
Save the latest time any project is fetched and checked out. This will
be used to detect partial checkouts.
Bug: b/286126621
Change-Id: I53b264dc70ba168d506076dbd693ef79a696b61d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/380514
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
New vs existing project may be a useful measure for analyzing
sync performance.
Bug: b/287105597
Change-Id: Ibea3e90c9fe3d16fd8b863bcae22b21963a6771a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/377574
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
By chance, _sync_dict can be empty even though repo sync is still
working. In that case, the progress message shows incorrect info. Handle this case and fix a bug where "0 jobs" can show.
Bug: http://b/284465096
Change-Id: If915d953ba60e7cf84a6fb2d137fd6ed82abd3cc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375494
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
It's possible that number of jobs is more than 0 when we
check length, but in the meantime number of jobs drops to
0. In that case, we are working with float(inf) which
causes other problems
Bug: 284383869
Change-Id: I5d070d1be428f8395df7fde8ca84866db46f2100
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375134
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
An investigation go/git-repo-shallow shows a number of problems
when doing a shallow git fetch/clone. This change introduces an
environment variable REPO_ALLOW_SHALLOW. When this environment variable
is set to 1 during a repo init or repo sync all shallow git fetch
commands are replaced with partial fetch commands. Any shallow
repository needing update is unshallowed. This behavior continues until
a subsequent repo sync command is run with REPO_ALLOW_SHALLOW set to 1.
Bug: b/274340522
Change-Id: I1c3188270629359e52449788897d9d4988ebf280
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374754
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Last of the recent `repo sync` UX changes. Show number of fetch jobs eg:
"Fetching: 3% (8/251) 0:03 | 8 jobs | 0:01 chromiumos/overlays/chrom.."
Bug: https://crbug.com/gerrit/11293
Change-Id: I1b3dcf3e56ae6731c6c6cb73cfce069b2f374b69
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374920
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
"Last synced: X" is printed only after a project finishes syncing.
Replace that with a message that shows the longest actively syncing
project.
Bug: https://crbug.com/gerrit/11293
Change-Id: I84c7873539d84999772cd554f426b44921521e85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/372674
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Trace logs emitted from repo are not useful on error for many critical
commands. This change adds errors for critical commands to trace logs.
Change-Id: Ideb9358bee31e540bd84a94327a09ff9b0246a77
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/373814
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Give users an indication that `repo sync` isn't stuck if taking a long
time to fetch.
Bug: https://crbug.com/gerrit/11293
Change-Id: Iccdaec918f86c9cc2db5dc12f9e3eef7ad0bcbda
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/371414
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Apply rules set by https://gerrit-review.googlesource.com/c/git-repo/+/362954/ across the codebase and fix any lingering errors caught
by flake8. Also check black formatting in run_tests (and CQ).
Bug: b/267675342
Change-Id: I972d77649dac351150dcfeb1cd1ad0ea2efc1956
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/363474
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
If interrupt signal is sent to repo process while sync is running, repo
prints stack trace for each concurrent job that is currently running
with no useful information.
Instead, this change captures KeyboardInterrupt in each process and
prints one line about current project that is being processed.
Change-Id: Ieca760ed862341939396b8186ae04128d769cd56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357135
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
repo sync progress bar is misleading. Many bug reports mentioned that
repo is stuck at the repo that is currently displayed in the progress
bar. Repo sync actually shows what repository is the last processed.
This change makes that obvious.
Change-Id: I962bf0bc65af7ac0ed98db86e9144f07d9e1f96f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357134
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
This can potentially show up when sync'ing projects with submodules
that are not declared in the manifest as well as the internal
'.repo/repo' project, which is likely not desirable from a user
standpoint.
Change-Id: I93d7fcd6e3fd1818357ea4537882a864dea9942c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/355920
Reviewed-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
Since --jobs defaults to 0, not None, we never pull the value out
of the manifest. Treat values of 0 and None the same to fix.
Bug: http://b/239712300
Bug: http://b/260908907
Change-Id: I9b1026682072366616825fd72f90bd90c10a252f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354254
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
If syncing in _FetchOne fails with GitError, sync_result does not get
set. There's already a separate local variable for success; do the same
for remote_fetched instead of referring to the conditionally defined
named tuple.
This bug is originally caused by a combination of ad8aa697 "sync: only
print error.GitError, don't raise that exception." and 1eddca84 "sync:
use namedtuples for internal return values".
Change-Id: I0f9dbafb97f8268044e5a56a6f92cf29bc23ca6a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354176
Tested-by: Karsten Tausche <karsten@fairphone.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
The wrong revision of the change was submitted as
d793553804.
Change-Id: I6f3e4993cf40c30ccf0d69020177db8fe5f76b8c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353934
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
