Currently we don't have a way for the checked out repo version to
declare the version of tools it needs before we start running it.
For somethings, like git, it's not a big deal as it can handle all
the asserts itself. But for things like Python, it's impossible
to reliably check before executing.
We're in this state now:
- we've been allowing Python 3.4, so the launcher accepts it
- the repo codebase starts using Python 3.6 features
- launcher tries to import us but hits syntax errors
- user is left confused and assuming new repo is broken because
they're seeing syntax errors
This scenario is playing out with old launchers that still accept
Python 2, and will continue to play out as time goes on and we want
to require newer versions of Python 3.
Lets create a JSON file to declare all these system requirements.
That file format is extremely stable, so loading & parsing from
even ancient versions of Python shouldn't be a problem. Then the
launcher can read these settings and check the system state before
attempting to execute any code. If the tools are too old, it can
clearly diagnose & display information to the user as to the real
problem (and not emit tracebacks or syntax errors).
We have a couple of different tool version checks already (git,
python, ssh) and can harmonize them in a single place.
This also allows us to assert a reverse dependency if the need
ever comes up: force the user to upgrade their `repo` launcher
before we'll let them run us. Even though the launcher warns
whenever a newer release is available, some users seem to ignore
that, or they don't use repo that often (on the scale of years),
and their upgrade jump is so dramatic that they fall back into
the syntax error pit.
Hopefully by the end of the year we can assume enough people
have upgraded their launcher such that we can delete all of the
duplicate version checks in the codebase. But until then, we'll
keep them to maintain coverage.
Change-Id: I5c12bbffdfd0a8ce978f39aa7f4674026fe9f4f8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293003
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We don't support Python 2 anymore, so stop allowing it to fallback.
If we try to run the latest version with Python 2, it just hits
syntax errors which confuses people. Dump a clear error message
that their system is too old and give up.
Bug: https://crbug.com/gerrit/13795
Change-Id: I38c243cf09502f670cddad72c2d0148f736515e0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292443
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The codebase still supports Python 3.5, so allow use of that instead
of requiring Python 3.6+. Supporting this mode well is a bit tricky
as we want to first scan for newer versions before falling back to
older ones. And we have to avoid infinite loops in the process.
Bug: https://crbug.com/gerrit/13795
Change-Id: I47949a173899bfa9ab20d3fefa1a97bf002659f6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We don't use javadoc in this project, so clean up the few places
that slipped in with the gitc code.
Change-Id: Ia365fb2d1e3188ad16b2f65b1a3b7e8466001946
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291262
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
In the .repo discovery loop
while curdir != '/' and curdir != olddir:
... break if we found .repo ...
olddir = curdir
curdir = os.path.dirname(curdir)
the "while" condition is meant to avoid searching forever if we do not
find .repo before reaching the top-level directory of the filesystem.
For that purpose, the first half of the condition is redundant; once
we reach "/", the parent directory will be "/" again and the curdir !=
olddir check would suffice to terminate the search. Simplify by
removing the redundant first half of the check.
Noticed by code inspection. The first half of the check was retained
when introducing the second half in df14a70c ("Make path references OS
independent", 2011-01-09), in an excess of caution.
This also improves consistency a little: if I start with curdir =
'/home/me', then with the redundant check in place we search
/home/me
/home
before hitting / and giving up. On Windows, if I start with
'c:/users/me', then we search
c:/users/me
c:/users
c:/
before hitting a repetition and giving up. Fortunately it is not
common for people to set up repo clients at the top level of
filesystems, but consistently following the latter behavior should
make debugging a little easier in case it comes up.
Link: https://gerrit-review.googlesource.com/id/Ib9e830e3b9adfb1c4e56f3bcfba4746c401fb84f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/286002
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jonathan Nieder <jrn@google.com>
We've already released 2.7, and the next tag is 2.8, so this should
be pulled up to 2.8 so it'll stay in sync.
Change-Id: Id47bdbdb8050b29ea36442ac2149dd948648237f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268572
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
For large projects, clone bundle is useful because it provided a way to
efficiently transfer a large portion of git objects through CDN, without
needing to interact with git server. However, with partial clones, the
intention is to not download most of the objects, so the use of clone
bundles would defeat the space savings normally seen with partial
clones, as they are downloaded before the first fetch.
A new option, --clone-bundle is added to override this behavior.
Add a new repo.clonebundle variable which remembers the choice if
explicitly given from command line at repo init.
Change-Id: I03638474af303a82af34579e16cd4700690b5f43
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268452
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
exit if no repo_main can be found right before executing the command.
This happens for instance when 'repo init' is run on root path
(for example in a container). Without this counter measure the tool
will crash at exec_command with
TypeError: sequence item 1: expected str instance, NoneType found
Change-Id: Ia8480cfe2151c3b35c9572789ad8cb619288cce1
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263457
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
When developing repo itself, it helps to run repo directly out of it
and to run bisection tools. The current _SetDefaultsTo logic fails
in that situation though as it wants a branch, but the source isn't
checked out to one. Now that we support tracking commits via the
--repo-rev setting, fall back to using the current HEAD commit.
Change-Id: I37d79fd9f7bea87d212421ebed6c8267ec95145f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260192
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
This gets the unittests passing again for now.
Change-Id: Ibed430a305bc26b907ad0ea424c7eec7de37e942
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259994
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We respect this option when running the first `repo init`, but then
silently ignore it once the initial sync is done. Make sure users
are able to change things on the fly.
We refactor the wrapper API to allow reuse between the two init's.
Bug: https://crbug.com/gerrit/11045
Change-Id: Icb89a8cddca32f39a760a6283152457810b2392d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260032
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
While the help/usage suggested that revisions would work, they never
actually did, and just throw confusing errors. Now that we warn if
the checkout isn't tracking a branch, allow people to specify commits
or tags explicitly. Hopefully our nags will be sufficient to keep
most people on the right path.
Bug: https://crbug.com/gerrit/11045
Change-Id: I6ea32c677912185f55ab20faaa23c6c0a4c483b3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259492
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We refer to this as "revision" in help text, and in REPO_REV envvar,
so rename to --repo-rev to be consistent. We keep --repo-branch for
backwards compatibility, but as a hidden option.
Bug: https://crbug.com/gerrit/11045
Change-Id: I1ecc282fba32917ed78a63850360c08469db849a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259352
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
When the launcher handles the init subcommand, it takes care of
setting the repo url & branch itself when cloning. So we don't
need to pass them down to the checked out init subcommand.
Further, the init subcommand has never actually done anything
with those options, so there's no point in passing them.
We'll be changing the latter behavior so that init will reset
the url/branch when specified with an existing repo checkout
which means passing them through adds overhead: the launcher
will checkout to the right value, then chain to the sub-init
which will then reset the checkout to the same value.
Bug: https://crbug.com/gerrit/11045
Change-Id: Ia2a4ab9d86febc470aea4abd73d75bb10e848b56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259312
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
When we download git sources, we get a progress bar (good) and we get
a dump of all the refs we downloaded (bad) as it can easily be 100+ if
not 1000+ depending on the project (for each git repo!). Lets rework
the output behavior so that:
* quiet: Only errors.
* default: Progress bars (if on a tty).
* verbose: Full output (progress bars & downloaded refs).
Bug: https://crbug.com/gerrit/11293
Change-Id: I87a380075e79de6805f91095876dd1b37d32873a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256456
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
We often ask users what OS/version they're running when debugging.
Include that in the version output to simplify the process.
Change-Id: Ie480b6d1c874e6f4c6f4996a03795077b844f858
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256732
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We produce some simple "Get" messages that aren't super clear as to
what they're doing, especially for people not familiar with repo.
Rephrase these to explicitly state the thing we're doing so it's
clear why we're downloading a particular source.
Bug: https://crbug.com/gerrit/11293
Change-Id: I0749504f17c5385c6c65274a274e0ae25b117413
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256455
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We don't use these for much yet, but init passes it down to the
project sync layers which already has support for verbose mode.
Change-Id: I651794f1b300be1688eeccf3941ba92c776812b5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256454
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
This copies the output style we use in git_command for showing output
and input redirections.
Change-Id: I449b27e7b262e1b30b24333109a1d91d9c7b1ce7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256453
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Switch the copyright holder to "The Android Open Source Project" to
match all the other source files in the tree, and move it to the top
of the file to match everything else we do.
Change-Id: Ie15d8e2bc004a626e45f715271deeaf3919dc44a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256235
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This supports [alias] sections with repo subcommands just like git.
Change-Id: Ie9235b5d4449414e6a745814f0110bd6af74ea93
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255833
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Hopefully enough issues should be resolved now that we can start
forcing Windows users into Python 3 too.
Change-Id: Ic4aad6a0b35ffec7d1372e3da6fca11a2b6fde0b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255353
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This provides initial support for using git worktrees internally
instead of our own ad-hoc symlink tree. It's been lightly tested
which is why it's not currently exposed via --help.
When people opt-in to worktrees in an existing repo client checkout,
no projects are migrated. Instead, only new projects will use the
worktree method. This allows for limited testing/opting in without
having to completely blow things away or get a second checkout.
Bug: https://crbug.com/gerrit/11486
Change-Id: Ic3ff891b30940a6ba497b406b2a387e0a8517ed8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254075
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Trying to use booleans with names like "no_xxx" are hard to follow due
to the double negatives. Invert all of them so we only have positive
meanings to follow.
Change-Id: Ifd37d0368f97034d94aa2cf38db52c723ac0c6ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255493
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
If programs emit non-UTF-8 output, we currently throw a fatal error.
We largely only care about the exit status of programs, and even the
output we do parse is a bit minimal. Lets make it into a warning and
mangle the invalid bytes into U+FFFD. This should complain enough to
annoy but not to break when it's not necessary.
Bug: https://crbug.com/gerrit/12337#c2
Change-Id: Idbc94f19ff4d84d2e47e01960dd17d5b492d4a8a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255272
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Some versions of gpg on Windows mishandle native paths with homedir.
It manifests itself like:
gpg: keybox 'C:\Users\.../.repoconfig\gnupg/pubring.kbx' created
gpg: C:\Users\.../.repoconfig\gnupg/trustdb.gpg: trustdb created
gpg: key 16530D5E920F5C65: public key "Repo Maintainer <repo@android.kernel.org>" imported
gpg: can't connect to the agent: Invalid value passed to IPC
gpg: Total number processed: 1
gpg: imported: 1
fatal: registering repo maintainer keys failed
It seems gpg (at least version 2.2.17) needs paths to be specified
in cygwin form (e.g. "/c/Users/.../.repoconfig/gnupg") otherwise
it fails to talk to its own processes. We can work around this
with a minor trick: we cd to the right path and then invoke gpg
with --homedir . and let gpg itself resolve . to whatever form it
really wants.
This is a bit hacky, but we don't control gpg, and this allows us
to avoid having to muck with the environment. Since --homedir has
been around since at least gpg-1.4.x from 2004, backwards compat
shouldn't be an issue.
While we're here, touch up the output a bit: there's no need to
dump all the chatty gpg output if things don't fail, so always
swallow the output. If things do fail, our exception handler
takes care of dumping the full stdout & stderr.
Change-Id: I74ab98e1e61e95318fda6faf57c6a8699f775935
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255120
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We added support for `repo init -c` to main.py, but not to the
launcher, so the -c option only works after the first init has
run which kind of defeats its purpose. Rework the parser setup
so that we can tell it whether it's for "init" or "gitc-init"
and then add the -c option in the same way we do in main.py.
This has the benefit of getting the parser entirely out of the
module scope which makes it a lot easier to reason about, and
it means we can write some unittests.
Change-Id: Icbc2ec3aceb938d5a8f941d5fbce1548553dc5f7
Test: repo help init
Test: repo help gitc-init
Test: repo init -u https://android.googlesource.com/platform/manifest -c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255113
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We've been using a DSA/1024 key to sign our tags. Time to update to
something a bit newer. We'll include RSA & ECC keys, but only use
RSA keys initially for backwards compatibility and see how it goes
with our user base.
Change-Id: I683c97b6fbd860f220ed4ddc7b21f07db279a916
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255112
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
repo:342:5: E306 expected 1 blank line before a nested definition, found 0
repo:617:5: F841 local variable 'ret' is assigned to but never used
Change-Id: I364fdb5dac8ebaff398b848935fe8356cb9ed2d3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255035
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
Now that we have a central run_command point, we can easily add
tracing support to the launcher script.
Change-Id: I9e0335c196cafd6263ff501925abfe835f036c5e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254755
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We can't rely on subprocess.run yet as that requires Python 3.6,
but we can clean up the code we have with some ad-hoc replacement.
This unifies all the inconsistent subprocess.Popen usage we have.
Change-Id: I56af40a3df988ee47b299105d692ff419d07ad6b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254754
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We can get version info when in a checkout, but it'd be helpful
to show that info at all times.
Change-Id: Ieeb44a503c9d7d8c487db4810bdcf3d5f6656c82
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254712
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This helps with people tracing repo/git execution. We use a similar
format to git, but a little simpler since we always initialize the
env var setting, and we want to avoid too much overhead.
Bug: https://crbug.com/gerrit/12314
Change-Id: I75675b6cc4c6f7c4f5e09f54128eba9456364d04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254331
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We import the wrapper on the fly, so minimize how much code we run
in module scope. It's pointless/wasted when importing.
Change-Id: I4a71c2030325d0a639585671cd7ebe8f22687ecd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254072
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
We were perhaps a bit too hasty to jump to git-2.10. Existing LTS
releases of Ubuntu are quite old still: Trusty has 1.9 while Xenial
has 2.5. While we plan on dropping support for those eventually as
we migrate to Python 3.6, we don't need to be so strict just yet on
the git versions.
We also want to disconnect the version the repo launcher requires
from the version the rest of the source tree requires. The repo
launcher doesn't need as many features, and being flexible there
allows us more freedom to upgrade & rollback as needed.
So we'll allow git-1.7 again, but start warning on any users older
than git-1.9. This aligns better with existing LTS releases, and
gives users a chance to start upgrading before we cut them off.
Change-Id: I140305dd8e42c9719c84e2aee0dc6a5c5b18da25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254573
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We want to start warning about Python 2 usage, but we can't do it
simply because the shebang is /usr/bin/python which might be an old
version like python2.7.
We can't change the shebang because program name usage is spotty at
best: on some platforms (like macOS), it's not uncommon to not have
a `python3` wrapper, only a major.minor one like `python3.6`. Using
python3 wouldn't guarantee a new enough version of Python 3 anyways,
and we don't want to require Python 3.6 exactly, just that minimum.
So we check the current Python version. If it's older than the ver
of Python 3 we want, we search for a `python3.X` version to run. If
those don't work, we see if `python3` exists and is a new enough ver.
If it's not, we die if the current Python 3 is too old, and we start
issuing warnings if the current Python version is 2.7. This should
allow the user to take a bit more action by installing Python 3 on
their system without having to worry about changing /usr/bin/python.
Once we require Python 3 completely, we can simplify this logic a bit
by always bootstrapping up to Python 3 and failing with Python 2.
We have a few KI with Windows atm though, so keep it disabled there
until the fixes are merged.
Bug: https://crbug.com/gerrit/10418
Change-Id: I5e157defc788e31efb3e21e93f53fabdc7d75a3c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253136
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
The git-2.10 series was released in 2016. Since we're moving to
require Python 3.6 which was also released in 2016, bumping up the
git version seems reasonable. Also we don't really test any git
versions close to as old as 1.7.2 which was released in 2010.
Change-Id: Ib71b714de6cd0b7dd50d0b300b108a560ee27331
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253134
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This hasn't been used in many years to sign a release, so drop it
from the keyring to avoid confusing people.
Bug: https://crbug.com/gerrit/12229
Change-Id: Ifca7eee713d167c11f32252975724e5858e4c007
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253133
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
