In order to be able to use "--no-repo-verify" to work around an issue with
gpg-agent and long socket paths (see e.g. [1]), this change avoids GPG
being set up at all if that option is passed.
[1] https://github.com/elastic/elasticsearch/issues/17053
Change-Id: I1e5cbd8be2dc0084f12afe0ca33c789fdbc6fef9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/251108
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use open() as a context manager to simplify the close logic and make
the code easier to read & understand. This is also more Pythonic.
Change-Id: I579d03cca86f99b2c6c6a1f557f6e5704e2515a7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244734
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We've rolled quite a number of fixes since the last update, including
a lot of Python 3 improvements. Lets bump the wrapper version for it.
Change-Id: I6c6c04c3c8241bf8e8bcf26603549ae4595fede8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/237812
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We use subprocess a lot in the wrapper, but we don't always read
or write the streams directly. When we do, make sure we convert
to/from bytes before trying to use the content.
Change-Id: I318bcc8e7427998348e359f60c3b49e151ffbdae
Reported-by: Michael Scott <mike@foundries.io>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236612
Reviewed-by: Michael Scott <mike@foundries.io>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Michael Scott <mike@foundries.io>
A new option, --partial-clone is added to 'repo init' which tells repo
to utilize git's partial clone functionality, which reduces disk and
bandwidth usage when downloading by omitting blob downloads initially.
Different from restricting clone-depth, the user will have full access
to change history, etc., as the objects are downloaded on demand.
Change-Id: I60326744875eac16521a007bd7d5481112a98749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
This should help people get some bearings in the codebase.
Change-Id: I951238fe617a3ecb04a47ead3809ec72c8fbf5a1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231232
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
All code that calls ParseGitVersion needs to run `git --version`
itself and parse the output before passing it in. To avoid that
duplication, allow ParseGitVersion to run `git --version` itself
if ver_str=None.
Bug: https://crbug.com/gerrit/11144
Change-Id: Ie07793ca57a40c0231af808df04a576118d5eea3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231054
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We were returning an e.g. tuple(1,2,3), but that strips off the full
version string which we might want in some places e.g. '1.2.3-rc3'.
Change the return value to a namedtuple so we can pass back up the
full version string. For code doing a compare with three elements
(all code today), things still work fine as the namedtuple will DTRT
in this scenario.
Bug: https://crbug.com/gerrit/11144
Change-Id: Ib897b5df308116ad1550b0cf18f49afeb662423e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231053
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Python 3 returns bytes by default with urlopen. Adjust our code to
handle that scenario and decode as necessary.
Bug: https://crbug.com/gerrit/10418
Change-Id: Icf4cd80e7ef92d71a3eefbc6113f1ba11c32eebc
We avoided this future import because Python 2.4 & 2.5 did not
support it. We've dropped support for Python 2.6 at this point,
and those versions are long dead. Since this workaround adds a
bit of complexity to the codebase, drop it. Considering we are
not running any actual tests against older versions, there's no
sense in trying to support them anymore.
Change-Id: Icda874861e8a8eb4fa07c624a9e7c5ee2a0da401
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.
Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
Standard utilities exit normally/zero when users explicitly request
--help, and they write to stdout. Exiting non-zero & using stderr
is meant for incorrect tool usage instead. We're already doing this
for `repo help <init|gitc-init>` calls, so lets fix `repo help` and
`repo --help|-h` to match.
Change-Id: Ia4f352b431c91eefef70dcafc11f00209ee69809
"repo init --reference" has two purposes: to decrease bandwidth used
at clone time, and to decrease disk usage afterward, by using the
reference repositories as an alternate store of objects even after
the clone. The downside is that it makes the borrowing repositories
dependent on the reference repositories, so it is easy to end up
with missing objects by mistake after a cleanup operation like "git
gc".
To prevent that, v2.3.0-rc0~30^2 (clone: --dissociate option to mark
that reference is only temporary, 2014-10-14), "git clone" gained a
--dissociate option that makes --reference reuse objects from the
reference repository at clone time but copy them over instead of
using the reference as an alternate. This is more straightforward to
use than plain --reference, at the cost of higher disk usage.
Introduce a --dissociate to "repo init" that brings the same benefits
to repo. The option is simply passed on to "git clone".
Change-Id: Ib50a549eb71e0a2b3e234aea57537923962a80d4
pylint is not used since bb5b1a0. The pyflakes cleanup mentioned in that
commit has not been done, but given that this project is no longer being
actively developed I don't think it's worth spending time doing it.
Leaving the pylint suppressions causes confusion because it leads people
to think that we are still using pylint.
Change-Id: If7d9f280a0f408c780f15915ffdb80579ae21f69
Commit 27226e742d introduced a warning if
"repo" is not part of the bootstrapped REPO_URL. However, that check was
done too early, directly after the call to _Clone. As the _Clone function
does not actually clone but it only initializes and fetches, the check
needs to be moved to after the call to _Checkout.
To reproduce, call
repo init --no-clone-bundle --repo-branch=master -u https://android.googlesource.com/platform/manifest
which will currently always show the (bogus) warning message. With this
fix, the warning will only be shown if "repo" indeed does not exist.
While at it, also slightly improve the code by using os.path.join().
Change-Id: Ied89e24231addabab6075005065748df1ffa74c4
While on Linux stderr is unbuffered, it is buffered on Windows. Always
flush stderr on Windows to ensure any error messages appear in the right
order to ease diagnosing.
Change-Id: I37300e384ecd3a51a321a48818f0114d6f3357a0
See http://stackoverflow.com/questions/7004687/os-exec-on-windows:
execv on Windows does not behave as on Linux, i.e. a new process is
spawned and the parent process terminates right away, which makes the
shell prompt come back too soon.
Change-Id: I1f8d23208765988629f081e9b949c67cf71c08ae
repo sync can sync submodules via the --fetch-submodules option.
However, if the manifest repo has submodules, those will not be synced.
Having submodules in the manifest repo -- while not commonly done -- can
be useful for inheriting a manifest from another project using <include>
and layering changes on top of it. In this way, you can avoid having to
deal with merge conflicts between your own manifests and the other
project's manifests (for example, if you're managing an Android fork).
Add a --submodule option to init that automatically syncs the submodules
in the manifest repo whenever the manifest repo changes.
Change-Id: I45d34f04517774c1462d7f233f482d1d81a332a8
Signed-off-by: Martin Kelly <mkelly@xevo.com>
If you don't know that the url to git-repo itself can be overridden via
REPO_URL, it's hard to debug cases where REPO_URL is accidentally set to
another repository, e.g. inside a Jenkins CI job. What makes is even
harder is that the ".repo/repo" directory gets silently removed in such
cases as verifications fails, which makes it impossible to look at the
cloned files to understand the problem.
To better protect against such an issue, warn if the cloned git-repo
repository does not contain a top-level "repo" file, and state that the
".repo/repo" directory will be removed in case of a clone failure.
Change-Id: I697b4999205a5967910c0237772ccaada01e74d4
There have been a number of changes in the repo wrapper since the last
increment that was done in fee390ee:
- 9711a98 init: Add --no-clone-bundle option
- 631d0ec Support non-ASCII GNUPGHOME environment variable
- 4088eb4 repo: Cleaned up pylint/pep8 violations
- 5553628 repo: Add check of REPO_URL env variable
- 745b4ad Fix gitc-init behavior
- d3ddcdb Ignore clone.bundle on HTTP 501, i.e. Not Implemented
Change-Id: I3f763ef0ec2df2d726dff429021b48ad474148f1
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a number of violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the repo script.
First I ran this to clean up the formatting:
autopep8 --max-line-length=80 --indent-size 2 repo
Following that the following violations remained:
% pylint --rcfile=.pylintrc repo
************* Module repo
W:220,21: Redefining name 'init_optparse' from outer scope (line 156)
(redefined-outer-name)
W:482, 2: No exception type(s) specified (bare-except)
C:704, 0: Old-style class defined. (old-style-class)
For line 220, the parameter to _GitcInitOptions was renamed so as not to
mask the init_optparse global.
For line 482, a pylint directive was added to disable the bare-execpt
violation for just that line.
For line 704, the _Options class was changed to subclass object.
Additionally, the comments at lines 107-113 were spaced out to line up
with the comment at line 112 that autopep8 moved.
This script now has a pylint score of 10.0
Change-Id: I779b66eb6b061a195d3c4372b99dec1b6d2a214f
We want to be able to run repo on a system that is not connected to
the Internet and cannot access https://gerrit.googlesource.com. We
can put a clone of that repos there, but would prefer to use the
stable version of the repo script instead of a locally modified
version.
This commit adds a check for the REPO_URL environment variable. If
that is set and not empty its value will be set in the REPO_URL
global in repo. Otherwise the standard path will be used.
Change-Id: I0616f5f81ef75f3463b73623b892cb5eed6bb7ba
With gitc-init, a gitc client may be specified using '-c'. If we're
not currently in that client, we need to change directories so that
we don't affect the local checkout, and to ensure that repo is
checked out in the new client.
This also makes '-c' optional if already in a gitc client, to match
the rest of the init options.
Change-Id: Ib514ad9fd101698060ae89bb035499800897e9bd
If the clone.bundle is out of date, repo may be installed with an old
version. It will upgrade with the next sync a day later, or when "repo
selfupdate" is run.
This behavior was added to normal project downloads, but was never added
to the repo launcher.
Change-Id: Ib04bef3a658c98fe1b6c53b3e8d0067165a5e3f7
Updates the repo launcher and gitc_utils to pull the manifest
directory location out of the gitc config file.
Change-Id: Id08381b8a7d61962093d5cddcb3ff6afbb13004b
Adds the new gitc-init command to set up a GITC client. Gitc-init
sets up the client directory and calls repo init within it. Once
the repo is initialized, then generates a GITC manifest file
by using git ls-remote on each project and retrieving the HEAD SHA
to use as the revision attribute.
Gitc-init inherits from and has all the options as repo init.
Change-Id: Icd7e47e90eab752a77de7c80ebc98cfe16bf6de3
If the server returns HTTP 401 (unauthorized) when attempting to
download clone bundle files, ignore it and continue, rather than
exiting with a fatal error.
Change-Id: I2c7ee03e149c354c7e4ad6ea1ebf266534778fe1
Only warn about using Python 3 when running the repo script directly.
This prevents the user being warned twice.
Change-Id: I2ee51ea2fa0127ea310598320e460ec9f38c6488
Part of the cleanup path for _Init is removing the .repo
directory. However, _Init can fail before creating the .repo directory,
so trying to remove it raises another exception:
fatal: invalid branch name 'refs/changes/53/55053/4'
Traceback (most recent call last):
File "/home/mitchelh/bin/repo", line 775, in <module>
main(sys.argv[1:])
File "/home/mitchelh/bin/repo", line 749, in main
os.rmdir(repodir)
OSError: [Errno 2] No such file or directory: '.repo'
Fix this by only removing .repo if it actually exists.
Change-Id: Ia251d29e9c73e013eb296501d11c36263457e235
The repo launcher version needs to be updated so some users can take
advantage of the more robust version number parsing.
Change-Id: Ibcd8036363311528db82db2b252357ffd21eb59b
'repo' and 'git_command.py' had their own git version parsing code.
This change shares that code between the modules. DRY is good.
Change-Id: Ic896d2dc08353644bd4ced57e15a91284d97d54a
This code checks whether a dir exists before creating it. In between the
check and the mkdir call, it is possible that another process will have
created the directory. We have seen this bug occur many times in
practice during our 'repo init' tests.
Change-Id: Ia47d39955739aa38fd303f4e90be7b4c50d9d4ba
This significantly reduces sync time and used brandwidth as only
a tar of each project's revision is checked out, but git is not
accessible from projects anymore.
This is relevant when git is not needed in projects but sync
speed/brandwidth may be important like on CI servers when building
several versions from scratch regularly for example.
Archive is not supported over http/https.
Change-Id: I48c3c7de2cd5a1faec33e295fcdafbc7807d0e4d
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
* Switching from python2 to python3 in the same workspace isn't
currently supported, due to a change in the pickle version (which
isn't supported by python2)
* Basic functionality does work with python3, however not everything
is expected to
Change-Id: I4256b5a9861562d0260b503f972c1569190182aa
Python 2.4 and 2.5 do not have a print_function available, so we need a
compatible print function for displaying an error message when the user
has an older version of Python.
Change-Id: I54d7297be98bb53970e873b36c6605e6dad386c3
cco3@android.com has a new gpg key, so this needs to be updated in the
repo scripts so that he can sign updates.
Change-Id: I9f058263b35bd027502d6e3b814d7aeb801a1e6e
Add a new module with methods for checking the Python version.
Instead of handling Python3 imports with try...except blocks, first
check the python version and then import the relevant modules. This
makes the code a bit cleaner and will result in less diff when/if we
remove support for Python < 3 later.
Use the same mechanism to handle `input` vs. `raw_input` and add
suppression of pylint warnings caused by redefinition of the built-in
method `input`.
Change-Id: Ia403e525b88d77640a741ac50382146e7d635924
Also-by: Chirayu Desai <cdesai@cyanogenmod.org>
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
Change Details:
* Make "default" a special manifest group that matches any project that
does not have the special project group "notdefault"
* Use "default" instead of "all,-notdefault" when user does not specify
manifest group
* Expand -g option help to include example usage of manifest groups
Change Benefits:
* Allow a more intuitive and expressive manifest groups specification:
* "default" instead of "all,-notdefault"
* "default,foo" instead of "all,-notdefault,foo"
* "default,-foo" instead of "all,-notdefault,-foo"
* "foo,-default" which has no equivalent
* Default manifest groups behavior can be restored by the command
'repo init -g default'. This is significantly more intuitive than the
current equivalent command 'repo init -g all,-notdefault'.
Change-Id: I6d0673791d64a650110a917c248bcebb23b279d3
If the server returns HTTP 403 (forbidden) when attempting to
download clone bundle files, ignore it and continue, rather than
exiting with a fatal error.
Change-Id: Icf78cba0332b51b0e7b622f7c7924369b551b6f6
Change Ia6032865f9296b29524c2c25b72bd8e175b30489 improved the
help text for the init command, but the same improvement was not made
in repo.
Change-Id: Idc34e479b5237137b90e8b040824776e4f7883b0
Enable the following Pylint warnings:
C0322: Operator not preceded by a space
C0323: Operator not followed by a space
C0324: Comma not followed by a space
And make the necessary fixes.
Change-Id: I74d74283ad5138cbaf28d492b18614eb355ff9fe
Clean up a few more unnecessary usages of lambda in `repo` that were missed
in the previous sweep that only considered files ending in .py.
Remove a duplicate import.
Change-Id: I03cf467a5630cbe4eee6649520c52e94a7db76be
It should be assumed that on modern development environments, python
is accessible to /usr/bin/env
Change the shebang as necessary and remove the magic hack.
This also means losing the -E option on the call to python, so that
PYTHONPATH and PYTHONHOME will be respected and local configuration
problems in those vars would be noticed
Change-Id: I6f0708ca7693f05a4c3621c338f03619563ba630
In Python3, range() creates a generator rather than a list.
None of the parameters in the ranges changed looked large enough
to create an impact in memory in Python2. Note: the only use of
range() was for iteration and did not need to be changed.
This is part of a series of changes to introduce Python3 support.
Change-Id: I50b665f9296ea160a5076c71f36a65f76e47029f
Previously, if a key was added, a client wouldn't add the key during
the sync step. This would cause issues if a new key were added and a
subsequent release were signed by that key.
Change-Id: I4fac317573cd9d0e8da62aa42e00faf08bfeb26c
"except Exception as e" instead of "except Exception, e"
This is part of a transition to supporting Python 3. Python >= 2.6
support "as" syntax.
Note: this removes Python 2.5 support.
Change-Id: I309599f3981bba2b46111c43102bee38ff132803
manifest_xml: import `HEAD` and `R_HEADS` from correct module
version: import `HEAD` from correct module
`HEAD` and `R_HEADS` should be imported from the git_refs module,
where they are originally defined, rather than from the project
module.
repo: remove unused import of readline
cherry_pick: import standard modules on separate lines
smartsync: import subcmd modules explicitly from subcmd
Use:
`import re
import sys`
and
`from subcmds.sync import Sync`
Instead of:
`import sys, re`
and
`from sync import Sync`
Change-Id: Ie10dd6832710939634c4f5c86b9ba5a9cd6fc92e
This is basically the same repository, but may be slightly more
up-to-date than the one on code.google.com/p/git-repo.
Change-Id: I5c99539f53231958eefb6993f00997c9adf0a3c9
Fix detection for Git not being in $PATH during the initial
run of `repo init` in a new directory.
Change-Id: I2b1fcce1fb8afc47271f5c3bd2a28369009b2fb7
Projects may optionally specify their platform
(eg, groups="platform-linux" in the manifest).
By default, repo will automatically detect the platform. However,
users may specify --platform=[auto|all|linux|darwin].
Change-Id: Ie678851fb2fec5b0938aede01f16c53138a16537
Every project is in group "default". "-default" does not remove
it from this project. All group names specified in the manifest
are positive names as opposed to a mix of negative and positive.
Specified groups are resolved in order. If init is supplied with
--groups="group1,-group2", the following describes the project
selection when syncing:
* all projects in "group1" will be added, and
* all projects in "group2" will be removed.
Change-Id: I1df3dcdb64bbd4cd80d675f9b2d3becbf721f661
Allows specifying a list of groups with a -g argument to repo init.
The groups act on a group= attribute specified on projects in the
manifest.
All projects are implicitly labelled with "default" unless they are
explicitly labelled "-default".
Prefixing a group with "-" removes matching projects from the list
of projects to sync.
If any non-inverted manifest groups are specified, the default label
is ignored.
Change-Id: I3a0dd7a93a8a1756205de1d03eee8c00906af0e5
Reviewed-on: https://gerrit-review.googlesource.com/34570
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
repo tool supports only Basic authentication for now. For those
who want to use this tool to manage their own projects, in case
the administrator has configured the Apache server with Digest
authentication method, users will fail to be authenticated when
they run the command 'repo init'.
Add the digest authentication password manager to the handler
list will fix this issue.
Since Git HTTP protocol will require the user be authenticated
for fetch operation first before pushing commits to the remote,
it is unlikely for the administrator to implement anonymous
read (aka pull) access and write access (aka push) for
authenticated user. Both read and write have to be authenticated.
Be aware that the user may have to add an extra line in his
~/.netrc file:
-------------------
account example.com
-------------------
where 'example.com' is the realm for Apache Digest authentication.
Change-Id: I76eb27b205554426d9ce1965deaaf727b87916cd
Signed-off-by: Xiaodong Xu <stid.smth@gmail.com>
This reverts commit ee1c2f5717.
This breaks a lot of buildbot systems. Rolling it back for now
until we can understand what the breakage was and how to fix it.
If the user has already configured a workspace, use these values
when re-running 'repo init'.
Otherwise, if the user has global name and e-mail set, use these.
It's always possible to override this and be prompted by specifying
--config-name when running 'repo init'.
Change-Id: If45f0e4b14884071439fb02709dc5cb53f070f60
A default manifest URL can be specified using:
git config --global repo-manifest.<id>.url <url>
A default manifest server can be specified using:
git config --global repo-manifest.<id>.server <url>
A default git mirror reference can be specified using:
git config --global repo-manifest.<id>.reference <path>
This will allow the user to use 'repo init -u <id>' as
a shorter alternative to specifying the full URL.
Also, manifest server will not have to be specified in the
manifest XML and the reference will not have to be specified
on the command line. If they are, they will override these
default values however.
Change-Id: Ifdbc160bd5909ec7df9efb0c5d7136f1d9351754
Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
An HTTP (or HTTPS) based remote server may now offer a 'clone.bundle'
file in each repository's Git directory. Over an http:// or https://
remote repo will first ask for '$URL/clone.bundle', and if present
download this to bootstrap the local client, rather than relying
on the native Git transport to initialize the new repository.
Bundles may be hosted elsewhere. The client automatically follows a
HTTP 302 redirect to acquire the bundle file. This allows servers
to direct clients to cached copies residing on content delivery
networks, where the bundle may be closer to the end-user.
Bundle downloads are resumeable from where they last left off,
allowing clients to initialize large repositories even when the
connection gets interrupted.
If a bundle does not exist for a repository (a HTTP 404 response
code is returned for '$URL/clone.bundle'), the native Git transport
is used instead. If the client is performing a shallow sync, the
bundle transport is not used, as there is no way to embed shallow
data into the bundle.
Change-Id: I05dad17792fd6fd20635a0f71589566e557cc743
Signed-off-by: Shawn O. Pearce <sop@google.com>
Windows allows the environment to have unicode values.
This will cause Python to fail to execute the command.
Change-Id: I37d922c3d7ced0d5b4883f0220346ac42defc5e9
Signed-off-by: Shawn O. Pearce <sop@google.com>
The complete help text is printed, so the program executed successfully.
Some tools (like OpenGrok) detects the availibility of a program by
running it with a known set of options and check the return code.
It is an easy and portable way of checking for the existence of a program
instead of searching the path (and handle extensions) ourselves.
Change-Id: Ic13428c77be4a36d599ccb8c86d893308818eae3
Use git clone to initialize a new repository, and when possible
allow callers to use --reference to reuse an existing checkout as
the initial object storage area for the new checkout.
Change-Id: Ie27f760247f311ce484c6d3e85a90d94da2febfc
Signed-off-by: Shawn O. Pearce <sop@google.com>
We'll soon be supporting two different manifest formats, but we
can't immediately remove support for the current XML one that is
in wide spread use within Android.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The mirror option downloads a complete forrest (as described by the
manifest) and creates a replica of the remote repositories rather
than a client working directory. This permits other clients to
sync off the mirror site.
A mirror can be positioned in a "DMZ", where the mirror executes
"repo sync" to obtain changes from the external upstream and
clients inside the protected zone operate off the mirror only,
and therefore do not require direct git:// access to the external
upstream repositories.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Many Linux distributions are including python2.5 by default, as
it is the latest stable release of the language. Using python2.4
(and asking users to specifically install it) is just cruel and
unusual punishment.
Signed-off-by: Shawn O. Pearce <sop@google.com>
