Cut out some more standalone code from Execute to make this func a
bit more manageable. The manifest project update is pretty simple
and standalone, but still takes up a good chunk of what's left.
Change-Id: Idc2442d9def495eccd0a49cda203c44aef16f129
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236614
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We've rolled quite a number of fixes since the last update, including
a lot of Python 3 improvements. Lets bump the wrapper version for it.
Change-Id: I6c6c04c3c8241bf8e8bcf26603549ae4595fede8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/237812
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When displaying progress bars, we use \r to reset the cursor to the
start of the line before showing the new update. This assumes the
new line will fully erase whatever was displayed there previously.
The "done" codepath tries to handle this by including a few extra
spaces at the end of the message to "white out" what was there.
Lets replace that hack with the standard ECMA escape sequence that
clears the current line completely. This is the CSI "erase in line"
sequence that the terminal will use to delete all content. The \r
is still needed to move the cursor to the start of the line. Using
this sequence should be OK since we're already assuming the terminal
is ECMA compliant with our use of coloring sequences. We also put
the \r after the CSI sequence on the off chance the terminal can't
process it and displays a few bytes of garbage.
The other improvement is to the syncbuffer API. When it dumps its
status information, it almost always comes after a progress bar
update which leads to confusing comingled output. Something like:
Fetching projects: 100% (2/2) error: src/platform2/: branch ...
Since the progress bar is "throw away", have the syncbuffer reset
the current output to the start of the line before showing whatever
messages it has queued.
Bug: https://crbug.com/gerrit/11293
Change-Id: I6544d073fe993d98ee7e91fca5e501ba5fecfe4c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236615
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We use subprocess a lot in the wrapper, but we don't always read
or write the streams directly. When we do, make sure we convert
to/from bytes before trying to use the content.
Change-Id: I318bcc8e7427998348e359f60c3b49e151ffbdae
Reported-by: Michael Scott <mike@foundries.io>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236612
Reviewed-by: Michael Scott <mike@foundries.io>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Michael Scott <mike@foundries.io>
The smart sync logic takes up about 45% of the overall Execute func
and is about 100 lines of code. The only effect it has on the rest
of the code is to set the manifest_name variable. Since this func
is already quite huge, split the smart sync logic out.
Change-Id: Id861849b0011ab47387d74e92c2ac15afcc938ba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234835
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This makes it way easier to recover from forgetting to run repo start
before committing: just run `repo start -b new-branch`, instead of
all that tedious mucking around with reflogs.
Change-Id: I56d49dce5d027e28fbba0507ac10cd763ccfc36d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232712
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
In commit d9e5cf0e ("sync: invert --force-broken with --fail-fast") the
force-broken option has been deprecated. Accidentally the option has
been changed from Boolean to Value. This breaks all users of repo with:
main.py: error: -f option requires an argument
This is easy to avoid by keeping the type.
Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Change-Id: Ia8b589cf41ac756d10c61e17ec8d76ba8f7031f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/235043
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
A common pattern in our subcommands is to verify the arguments &
options before executing things. For some subcommands, that check
stage is quite long which makes the execution function even bigger.
Lets split that logic out of the execute phase so it's easier to
manage these.
This is most noticeable in the sync subcommand whose Execute func
is quite large, and the option checking makes up ~15% of it.
The manifest command's Execute can be simplified significantly as
the optparse configuration always sets output_file to a string.
Change-Id: I7097847ff040e831345e63de6b467ee17609990e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234834
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This can help debug issues by tracing all the repo python code with
the standard trace module.
Change-Id: Ibb7f4496ab6c7f9e130238ddf3a07c831952697a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234833
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
There is a standard Python "trace" module, so having a local trace.py
prevents us being able to import that. Rename the module to avoid.
Change-Id: I23e29ec95a2204bb168a641323d05e76968d9b57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234832
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This is useful when you want to scan all the possibilities of repo
at once. Like when you're searching for different option names.
Change-Id: I225dfb94d2be78229905b744ecf57eb2829bb52d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232894
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
People seem to not expect the sync process to halt immediately if an
error is encountered. It's also basically guaranteed to leave their
tree in an incomplete state. Lets invert the default behavior so we
attempt to sync (both fetch & checkout) all projects. If an error is
hit, we still exit(1) and show it at the end.
If people want the sync to abort quickly, they can use the new option
--fail-fast.
Bug: https://crbug.com/gerrit/11293
Change-Id: I49dd6c4dc8fd5cce8aa905ee169ff3cbe230eb3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234812
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This makes the code a bit easier to read by doing all the project
independent settings first instead of repeating it for every for
loop iteration.
Change-Id: I4ff21296e444627beba2f4b86561069f5e9a0d73
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233554
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Callers don't actually see -1 (they'll usually see 255, but the exact
answer here is complicated). Just switch to 1 as that's the standard
value tools use to indicate an error.
Change-Id: Ib712db1924bc3e5f7920bafd7bb5fb61f3bda44f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233553
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The partial clone rework (commit 745be2ede1
"Add support for partial clone") changed the behavior when a single repo
hit a failure: it would always call sys.exit() immediately. This isn't
even necessary as we already pass down an error event object which the
workers set and the parent checks. Just delete the exit entirely.
Change-Id: Id72d8642aefa2bde24e1a438dbe102c3e3cabf48
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233552
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The GitCommand Wait helper takes care of decoding bytes to strings
for us. That means we don't have to decode stdout ourselves which
is what our local rev list, ls-remote, and generic get_attr helpers
were doing.
If we don't use Wait though to capture the output but instead go
directly to the subprocess stdout, we do have to handle decoding
ourselves. This is what the diff helpers were doing.
Bug: https://crbug.com/gerrit/10418
Change-Id: I057ca245af3ff18d6b4a074e3900887f06a5617d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233076
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When syncing in parallel, this exception is hard to trace back to
a specific repo as the relevant log line could have been pushed
out by other repos syncing code.
Change-Id: I382efeec7651e85622aa51e351134aef0148267f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233075
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Nasser Grainawi <nasser@codeaurora.org>
Tested-by: Mike Frysinger <vapier@google.com>
While we don't (yet) explicitly enforce all of these, make sure
we document the expected behavior so we can all agree on it.
Bug: https://crbug.com/gerrit/11218
Change-Id: Ife8298702fa445ac055ef43c6d62706a9cb199ce
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232893
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
A warning is emitted
DeprecationWarning: the imp module is deprecated in favour of
importlib; see the module's documentation for alternative uses
Change-Id: I6c5a9e024a9a904e02a24331f615548be3fe5f8e
Signed-off-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/230984
Reviewed-by: Jonathan Nieder <jrn@google.com>
As we convert repo to support Python 3, the version of Python that we
use might not be the version that repo hooks users have written for.
Since repo upgrades are not immediate, and not easily under direct
control of end users (relative to the projects maintaining the hook
code), allow hook authors to declare the version of Python that they
want to use.
Now repo will read the shebang from the hook script and compare it
against the version of Python repo itself is running under. If they
differ, we'll try to execute a separate instance of Python and have
it load & execute the hook. If things are compatible, then we still
use the inprocess execution logic that we have today.
This allows repo hook users to upgrade on their own schedule (they
could even upgrade to Python 3 ahead of us) without having to worry
about their supported version being exactly in sync with repo's.
Bug: https://crbug.com/gerrit/10418
Change-Id: I97c7c96b64fb2ee465c39b90e9bdcc76394a146a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228432
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
A new option, --partial-clone is added to 'repo init' which tells repo
to utilize git's partial clone functionality, which reduces disk and
bandwidth usage when downloading by omitting blob downloads initially.
Different from restricting clone-depth, the user will have full access
to change history, etc., as the objects are downloaded on demand.
Change-Id: I60326744875eac16521a007bd7d5481112a98749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
This should help people get some bearings in the codebase.
Change-Id: I951238fe617a3ecb04a47ead3809ec72c8fbf5a1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231232
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We're relying on sys.stdout.write() to flush its buffer which isn't
guaranteed, and is not the case in Python 3. Change to use print()
everywhere to be standard, and utilize the end= keyword to get the
EOL semantics we need.
We can't use print's flush= keyword as that's only in Python 3.
Leave behind a TODO to clean it up when we can drop Python 2.
Bug: https://crbug.com/gerrit/10418
Change-Id: I562128c7f1e6d154f4a6ecdf33a70fa2811dc2af
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/230392
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This makes it very easy for people to run all our unittests with just
`./run_tests`. There doesn't seem to be any other way currently to
quickly invoke any of the tests.
Change-Id: I1f9a3745fa397a1e797bd64065c2ba7f338de4a1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/227613
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This fix exception with python3 with stack-trace:
error: Cannot fetch platform_external_grpc-grpc-java.git (UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 640: invalid start byte)
[...]
File "[...]project.py", line 2255, in _IsValidBundle
if f.read(16) == '# v2 git bundle\n':
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
Even if we ask 16 characters, python buffered decoder will try to decode more in the buffer
The patch works for python2 and python3, and open the file in byte mode so that decoding is not attemped
Signed-off-by: Pierre Tardy <tardyp@gmail.com>
Change-Id: I837ae3c5cd724b34670fc2a84e853431f482b20d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/224642
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We've been truncating the git version info in the user agent to the
first three components. So given an example `git --version` like
"2.22.0.510.g264f2c817a-goog", we were cutting it down to "2.22.0".
For user-agent purposes, we usually want that full string, so use
the original full value instead.
Bug: https://crbug.com/gerrit/11144
Change-Id: I8ffe3186bdaac96164c34ac835a54bb3fc85527e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231056
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Since ParseGitVersion can call `git --version` automatically, we don't
need this duplicate version() helper anymore. The only other user is
the `repo version` code, so convert that to version_tuple().full.
Bug: https://crbug.com/gerrit/11144
Change-Id: I9d77822fc39f4ba28884d9183359169cabf5f17d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231055
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
All code that calls ParseGitVersion needs to run `git --version`
itself and parse the output before passing it in. To avoid that
duplication, allow ParseGitVersion to run `git --version` itself
if ver_str=None.
Bug: https://crbug.com/gerrit/11144
Change-Id: Ie07793ca57a40c0231af808df04a576118d5eea3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231054
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We were returning an e.g. tuple(1,2,3), but that strips off the full
version string which we might want in some places e.g. '1.2.3-rc3'.
Change the return value to a namedtuple so we can pass back up the
full version string. For code doing a compare with three elements
(all code today), things still work fine as the namedtuple will DTRT
in this scenario.
Bug: https://crbug.com/gerrit/11144
Change-Id: Ib897b5df308116ad1550b0cf18f49afeb662423e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231053
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
%prog represents the full subcommand ("repo" + subcommand name), not a
Windows-style environment variable for "repo". The current help output
shows
repo forall% forall ...
Correct the variable usage so it shows "repo forall ..." instead.
Change-Id: I1fea55572428cc922ddf24ace1168a3d8f82dad0
This takes a single argument (the error message), not multiple
arguments that get formatted implicitly.
Change-Id: Idfbc913ea9f93820edb7e955e9e4f57618c8cd1b
Currently we read the binary stream from the subprocess code directly
before waiting for it to finish, but there's no need to do so as we
aren't streaming the output to the user. This also means we pass up
binary data to the caller as we don't go through GitCommand's internal
logic which decodes the stream as utf-8.
Simplify the code by calling Wait first, then splitting the entire
captured output in one line.
Bug: https://crbug.com/gerrit/10418
Change-Id: I7a57904be8cb546a229980fb79c829fc3df31e7d
Python 3 returns bytes by default with urlopen. Adjust our code to
handle that scenario and decode as necessary.
Bug: https://crbug.com/gerrit/10418
Change-Id: Icf4cd80e7ef92d71a3eefbc6113f1ba11c32eebc
Neither of the fields here expect floats so make sure we use integer
division when calculating things.
Bug: https://crbug.com/gerrit/10418
Change-Id: Ibda068b16a7bba7ff3efba442c4bbff4415caa6e
This module uses print() so make sure we import the print function.
It doesn't really impact the current code due to the simple way it
is calling print, but we should be sane to avoid future issues.
Change-Id: I0b15344678c1dcb71207faa333c239b3fced1d62
We avoided this future import because Python 2.4 & 2.5 did not
support it. We've dropped support for Python 2.6 at this point,
and those versions are long dead. Since this workaround adds a
bit of complexity to the codebase, drop it. Considering we are
not running any actual tests against older versions, there's no
sense in trying to support them anymore.
Change-Id: Icda874861e8a8eb4fa07c624a9e7c5ee2a0da401
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.
Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
Use the `raise` statement directly.
Switch to using .items() instead of .iteritems(). Python 3 doesn't
have .iteritems() as .items() is a generator, and these are small
enough that the Python 2 overhead should be negligible.
We have to run .keys() through list() in a few places as Python 3
uses a generator and we sometimes want to iterate more than once.
That's why we don't change all .keys() or .items() calls -- most
are in places where generators are fine.
Bug: https://crbug.com/gerrit/10418
Change-Id: I469899d9b77ffd77ccabb831bc4b217407fefe6f
Standard utilities exit normally/zero when users explicitly request
--help, and they write to stdout. Exiting non-zero & using stderr
is meant for incorrect tool usage instead. We're already doing this
for `repo help <init|gitc-init>` calls, so lets fix `repo help` and
`repo --help|-h` to match.
Change-Id: Ia4f352b431c91eefef70dcafc11f00209ee69809
This isn't executable (+x), nor does it have a main func or code
that would run if it were. It's simply an imported module like
most others in here. Drop the shebang to avoid confusion.
Change-Id: I5e2881eb1de5e809a3fa9e8f49220ed797034fb1
