Bug: 302680231
Change-Id: Icd01dd2ce62d737a4acb114e729189cd31f6bde9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/388234
Tested-by: Jason R. Coombs <jaraco@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Jason R. Coombs <jaraco@google.com>
Sometimes it is expected that a GitCommand executed in repo fails. In
such cases indicate in trace logs that the error was expected.
Bug: b/293344017
Change-Id: If137fae9ef9769258246f5b4494e070345db4a71
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/387714
Commit-Queue: Jason Chang <jasonnc@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
tox uses virtualenv under its hood for managing virtual environments.
Virtualenv 20.22.0 dropped support for Python <= 3.6.
Since we want to test against Python 3.6 we need to make sure we use
a version of virtualenv earlier than 20.22.0.
This error was not stopping any tests from passing but was printed
multiple times to stderr when executing the py36 target:
Error processing line 1 of [...]/.tox/py36/[...]/_virtualenv.pth:
Traceback (most recent call last):
File "/usr/lib/python3.6/site.py", line 168, in addpackage
exec(line)
File "<string>", line 1, in <module>
File "[...]/.tox/py36/[...]/_virtualenv.py", line 3
from __future__ import annotations
^
SyntaxError: future feature annotations is not defined
Source: https://tox.wiki/en/latest/faq.html#testing-end-of-life-python-versions
Change-Id: I27bd8200987ecf745108ee8c7561a365f542102a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/387694
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Commit-Queue: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
DeprecationWarning: datetime.datetime.utcnow() is deprecated and
scheduled for removal in a future version. Use timezone-aware objects to
represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
Change-Id: Ia2c46fb87c544d98cc2dd68a829f67d4770b479c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/386615
Tested-by: Łukasz Patron <priv.luk@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Łukasz Patron <priv.luk@gmail.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
We don't accept PRs via GH, so add a job to automatically close them
with an explanation for how to submit.
Change-Id: I5cc3176549a04ff23b04dae1110cd27a58ba1fd3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/386134
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
tests/test_subcmds_sync.py::LocalSyncState::test_prune_removed_projects
was failing in Python 3.6 and 3.7 due to topdir not being set with the
following error message:
TypeError: expected str, bytes or os.PathLike object, not MagicMock
topdir is accessed from within PruneRemovedProjects().
Test: tox with Python 3.6 to 3.11
Change-Id: I7ba5144df0a0126c01776384e2178136c3510091
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/382816
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Daniel Kutik <daniel.kutik@lavawerk.com>
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
tests/test_git_superproject.py::SuperprojectTestCase::test_Fetch was
failing in Python 3.6 and 3.7 due to attribute args only being
introduced in Python 3.8. Falling back on old way of accessing
the arguments.
Test: tox with Python 3.6 to 3.11
Change-Id: Iae1934a7bce8cbd6b4519e4dbc92d94e21b43435
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/382818
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Daniel Kutik <daniel.kutik@lavawerk.com>
When comparing 2 manifests, most of the time is
spent getting the relevant commit id as it relies
on _allrefs which ends up loading all git references.
However, the value from `revisionIs` (when it is valid)
could be used directly leading to a huge performance improvement
(from 180+ seconds to less than 0.01 sec which is more
than 25000 times faster for manifests with 700+ projects).
Bug: 295282548
Change-Id: I5881aa4b2326cc17bbb4ee91d23293111f76ad7e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/385834
Tested-by: Sylvain Desodt <sylvain.desodt@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Sylvain Desodt <sylvain.desodt@gmail.com>
The logic in projectsDiff performs various operations which
suggest that a set is more appropriate than a list:
- membership lookup ("in")
- removal
Also, sorting can be performed on the the remaining elements at the
end (which will usually involve a much smaller number of elements).
(The performance gain is invisible in comparison to the time being
spent performing git operations).
Cosmetic chance:
- the definition of 'fromProj' is moved to be used in more places
- the values in diff["added"] are added with a single call to extend
Change-Id: I5ed22ba73b50650ca2d3a49a1ae81f02be3b3055
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/383434
Tested-by: Sylvain Desodt <sylvain.desodt@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Sylvain Desodt <sylvain.desodt@gmail.com>
If a KeyboardInterrupt is encountered before an error is aggregated then
the context surrounding the interrupt is lost. This change aggregates
errors as soon as possible for the sync command
Bug: b/293344017
Change-Id: Iac14f9d59723cc9dedbb960f14fdc1fa5b348ea3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/384974
Tested-by: Jason Chang <jasonnc@google.com>
Commit-Queue: Jason Chang <jasonnc@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Black will only check .py files when given a dir and --check, so list
our few standalone programs explicitly. This causes the repo launcher
to be reformatted since it was missed in the previous mass reformat.
Bug: b/267675342
Change-Id: Ic90a7f5d84fc02e9fccb05945310fd067e2ed764
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/385034
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
A previous change captured stderr when uploading git projects. This
change ensures stderr is sent to stderr.
Bug: b/297097597
Change-Id: I8314e1017d2a42b7b655fe43ce3c312d397894ca
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/384134
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Jason Chang <jasonnc@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
a previous introduced isort, which causes tox
runs to fail for all python versions. adding
isort as dependency resolve these issues.
Change-Id: If3faf78e6928e6e5111b2ef2359351459832431f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/384175
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Google Python style guide says to import modules.
Clean up all our stdlib imports. Leave the repo ones alone
for now as that's a much bigger shave.
Change-Id: Ida42fc2ae78b86e6b7a6cbc98f94ca04b295f8cc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/383714
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The repo project is fetched at most once a day and should be ignored
when checking if the tree is partially synced.
Bug: b/286126621, b/271507654
Change-Id: I684ed1669c3b3b9605162f8cc9d57185bb3dfe8e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/383494
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
There was a bug in error handeling code that caused an uncaught
exception to be raised.
Bug: b/296316540
Change-Id: I49c72f29c00f26ba60de552f958bc6eddf841162
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/383254
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Jason Chang <jasonnc@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Python 3.11 was released almost a year ago.
Test: tox -epy311
Change-Id: I447637a1e97038a596373d7612c9000c0c738ec9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/382838
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Daniel Kutik <daniel.kutik@lavawerk.com>
Excluding these two folders to avoid countless lint warnings
caused by dependencies in these two folders.
Change-Id: I2403b23f88cebb5941a4f9b5ac6cc34d107fd2f1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/382837
Commit-Queue: Daniel Kutik <daniel.kutik@lavawerk.com>
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Partial syncs are not supported and can lead to strange behavior like
deleting files. Explicitly warn users on partial sync.
Bug: b/286126621, b/271507654
Change-Id: I471f78ac5942eb855bc34c80af47aa561dfa61e8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/382154
Reviewed-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Not including it causes flaky behavior in some Chromium builders
because Chromium's custom Python build used by vpython relies on
argv[0] to find its own internal files.
Bug: https://crbug.com/1468522
Change-Id: I5c32ebe71c9b684d6ee50dbd8c3d6fcd51ca309b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/381974
Reviewed-by: Chenlin Fan <fancl@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Prior to this change there is no way to distinguish between git sessions logs
generated from repo source v.s. from git.
Bug: b/294446468
Change-Id: I309f59e146c30cb08a0637e8d0b9c5d9efd5cada
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/381794
Commit-Queue: Jason Chang <jasonnc@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Per discussion in go/repo-error-update updated aggregated and exit
errors for sync command.
Aggregated errors are errors that result in eventual command failure.
Exit errors are errors that result in immediate command failure.
Also updated main.py to log aggregated and exit errors to git sessions
log
Bug: b/293344017
Change-Id: I77a21f14da32fe2e68c16841feb22de72e86a251
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/379614
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Commit-Queue: Jason Chang <jasonnc@google.com>
In order to better analyze and track repo errors, repo command failures
need to be tied to specific errors in repo source code.
Additionally a new GitCommandError was added to differentiate between
general git related errors to failed git commands. Git commands that opt
into verification will raise a GitCommandError if the command failed.
The first step in this process is a general error refactoring
Bug: b/293344017
Change-Id: I46944b1825ce892757c8dd3f7e2fab7e460760c0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/380994
Commit-Queue: Jason Chang <jasonnc@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Save the latest time any project is fetched and checked out. This will
be used to detect partial checkouts.
Bug: b/286126621
Change-Id: I53b264dc70ba168d506076dbd693ef79a696b61d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/380514
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Fixed a couple of bugs in ExitEvent logging:
- log exitcode 130 on KeyboardInterrupt
- log exitcode 1 on unhandled Exception
- log errorevent with specific reason for exit
Before this CL an exitcode of 0 would be logged, and it would be
difficult to determine the cause of non-zero exit codes
Bug: b/287105597
Change-Id: I2d34f180581f9fbd77a1c78c966ebed065223af6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/377834
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
New vs existing project may be a useful measure for analyzing
sync performance.
Bug: b/287105597
Change-Id: Ibea3e90c9fe3d16fd8b863bcae22b21963a6771a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/377574
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
A something.xml that gets included by two different
files, that both remove and add same shared project
to two different locations, would not work
prior to this change.
Reason is that remove killed all name keys, even
though reuse of same repo in different locations
is allowed.
Solve by adding optional attrib path to
<remove-project name="foo" path="only_this_path" />
and tweak remove-project.
Behaves as before without path, and deletes
more selectively when remove path is supplied.
As secondary feature, a project can now also be removed
by only using path, assuming a matching project name
can be found.
Change-Id: I502d9f949f5d858ddc1503846b170473f76dc8e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375694
Tested-by: Fredrik de Groot <fredrik.de.groot@aptiv.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
By chance, _sync_dict can be empty even though repo sync is still
working. In that case, the progress message shows incorrect info. Handle this case and fix a bug where "0 jobs" can show.
Bug: http://b/284465096
Change-Id: If915d953ba60e7cf84a6fb2d137fd6ed82abd3cc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375494
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
It's possible that number of jobs is more than 0 when we
check length, but in the meantime number of jobs drops to
0. In that case, we are working with float(inf) which
causes other problems
Bug: 284383869
Change-Id: I5d070d1be428f8395df7fde8ca84866db46f2100
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375134
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This matches the git behavior. From [1],
> When the `LESS` environment variable is unset, Git sets it to `FRX`
> (if `LESS` environment variable is set, Git does not change it at
> all).
The default $LESS is changed from FRSX to FRX since git 2.1.0 [2]. This
change also updates the default $LESS for repo.
[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-corepager
[2] b3275838d9
Bug: https://crbug.com/gerrit/16973
Change-Id: I64ccaa7b034fdb6a92c10025e47f5d07e85e6451
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374894
Reviewed-by: Chih-Hsuan Yen <x5f4qvj3w3ge2tiq@chyen.cc>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Chih-Hsuan Yen <x5f4qvj3w3ge2tiq@chyen.cc>
An investigation go/git-repo-shallow shows a number of problems
when doing a shallow git fetch/clone. This change introduces an
environment variable REPO_ALLOW_SHALLOW. When this environment variable
is set to 1 during a repo init or repo sync all shallow git fetch
commands are replaced with partial fetch commands. Any shallow
repository needing update is unshallowed. This behavior continues until
a subsequent repo sync command is run with REPO_ALLOW_SHALLOW set to 1.
Bug: b/274340522
Change-Id: I1c3188270629359e52449788897d9d4988ebf280
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374754
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
