"except Exception as e" instead of "except Exception, e"
This is part of a transition to supporting Python 3. Python >= 2.6
support "as" syntax.
Note: this removes Python 2.5 support.
Change-Id: I309599f3981bba2b46111c43102bee38ff132803
We need a representation of git-submodule in repo; otherwise repo will
not sync submodules, and leave workspace in a broken state. Of course
this will not be a problem if all projects are owned by the owner of the
manifest file, who may simply choose not to use git-submodule in all
projects. However, this is not possible in practice because manifest
file owner is unlikely to own all upstream projects.
As git submodules are simply git repositories, it is natural to treat
them as plain repo projects that live inside a repo project. That is,
we could use recursively declared projects to denote the is-submodule
relation of git repositories.
The behavior of repo remains the same to projects that do not have a
sub-project within. As for parent projects, repo fetches them and their
sub-projects as normal projects, and then checks out subprojects at the
commit specified in parent's commit object. The sub-project is fetched
at a path relative to parent project's working directory; so the path
specified in manifest file should match that of .gitmodules file.
If a submodule is not registered in repo manifest, repo will derive its
properties from itself and its parent project, which might not always be
correct. In such cases, the subproject is called a derived subproject.
To a user, a sub-project is merely a git-submodule; so all tips of
working with a git-submodule apply here, too. For example, you should
not run `repo sync` in a parent repository if its submodule is dirty.
Change-Id: I541e9e2ac1a70304272dbe09724572aa1004eb5c
Fixing more issues found with pylint. Some that were supposed to
have been fixed in the previous sweep (Ie0db839e) but were missed:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
And some more:
W0631: Using possibly undefined loop variable 'name'
W0223: Method 'name' is abstract in class 'name' but is not overridden
W0231: __init__ method from base class 'name' is not called
Change-Id: Ie119183708609d6279e973057a385fde864230c3
Fix the following issues reported by pylint:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
W0612: Unused variable 'name'
W0613: Unused argument 'name'
W0102: Dangerous default value 'value' as argument
W0105: String statement has no effect
Also fixed a few cases of inconsistent indentation.
Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
Currently when doing a sync against a revision locked manifest,
sync has no option but to fall back to sync'ing the entire refs space;
it doesn't know which ref to ask for that contains the sha1 it wants.
This sucks if we're in -c mode; thus when we generate a revision
locked manifest, record the originating branch- and try syncing that
branch first. If the sha1 is found within that branch, this saves
us having to pull down the rest of the repo- a potentially heavy
saving.
If that branch doesn't have the desired sha1, we fallback to sync'ing
everything.
Change-Id: I99a5e44fa1d792dfcada76956a2363187df94cf1
Catch curl failures to download clone.bundle; don't let git try to parse
the 404 page as a bundle file (was causing much user confusion).
This should eliminate false error messages from init and sync such as:
error: '.repo/manifests.git/clone.bundle' does not look like a v2 bundle file
fatal: Could not read bundle '.repo/manifests.git/clone.bundle'.
error: RPC failed; result=22, HTTP code = 400
Signed-off-by: Matt Gumbel <matthew.k.gumbel@intel.com>
Change-Id: I7994f7c0baecfb45bb5a5850c48bd2a0ffabe773
Instead of every group being in the group "default", every project
is now in the group "all". A group that should not be downloaded
by default may be added to the group "notdefault".
This allows all group names to be positive (instead of removing groups
directly in the manifest with -default) and offers a clear way of
selecting every project (--groups all).
Change-Id: I99cd70309adb1f8460db3bbc6eff46bdcd22256f
urllib2 is not thread safe and may be causing sync to lock up or
not work correctly on various platforms. Instead use the command
line curl program.
Change-Id: I36eaf18bb4df089d26ea99d533cb015e7c616eb0
The Content-Length when resuming is the number of bytes that
remain in the file. To compute the total size as expected by
the progress meter, we must add the bytes already stored.
While we are in this method fix uses of % operator to ensure
a tuple is always supplied.
Change-Id: Ic899231b5bc0ab43b3ddb1d29845f6390e820115
The fix for issue #46 in 5d016502eb appears to break syncing in some
situations: the branch is deleted after the point where it's been
configured, which deletes part of its configuration and causes the
config to change each time you call `repo init`, alternating between a
configuration that works and one that doesn't.
Instead of deleting the branch with git branch -D, use git update-ref -d
which just deletes the ref (to avoid the rebase) without touching the
configuration for the branch that was set up during the first repo init.
This appears to ensure the config is left in a valid state all the time
no matter what combination of repo init commands you run, without
reintroducing the rebasing issue.
Change-Id: Iaadaa6e56a46840bbc593fa5b35cb5b34cd3ce69
This patch fixes repo behaviour when running sync -d with unmodified
topic branches.
Prior to this patch sync -d would see the latest revision is already
checked out, thus staying on the branch. Since "-d" means detach we
should follow git's behaviour and actually detach from the branch in
that case.
Basic test case - after a fresh repo init + sync -
* repo start --all testdetach
* repo sync -d
* repo status
-> status shows active topic branch "testdetach",
should show :
nothing to commit (working directory clean)
Change-Id: Ic1351e6b5721b76557a51ab09f9dd42c38a4b415
See repo issue #46 :
https://code.google.com/p/git-repo/issues/detail?id=46
When using repo init -b on an already existing repository,
the next sync will try to rebase changes coming from the old manifest
branch onto the new, leading in the best case scenario to conflicts
and in the worst case scenario to an incorrect "mixed up" manifest.
This patch fixes this by deleting the "default" branch in the local
manifest repository when the -d init switch is used, thus forcing
repo to perform a fresh checkout of the new manifest branch
Change-Id: I379e4875ec5357d8614d1197b6afbe58f9606751
Mirror manifest and repo projects are outside the manifest and
have no groups. Allow project groups to be None for these
projects.
Change-Id: I3e1c4add894fe1c43aa4e77a1fc1558aa10dd191
Allows to ff-only a gerrit patch
This patch is necessary to automatically ensure that the patch will
be correctly submitted on ff-only gerrit projects
You can now use:
repo download (--ff-only|-f) project changeid/patchnumber
This is useful to automate verification of fast forward status of a patch
in the context of build automation, and commit gating (e.g. buildbot)
Change-Id: I403a667557a105411a633e62c8eec23d93724b43
Signed-off-by: Erwan Mahe <erwan.mahe@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
BZ: 4779
Allows to revert a gerrit patch
This patch is necessary for the on-demand creation of
engineering builds using buildbot
You can now use:
repo download [--revert|-r project changeid/patchnumber
This is useful to automate reverting of a patch
in the context of build automation, and regression bisection
Change-Id: I3985e80e4b2a230f83526191ea1379765a54bdcf
Signed-off-by: Erwan Mahe <erwan.mahe@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
default option uses git checkout, and thus overwrite the previous
checkouts. this is a problem for automated builds of several
changesets in the same project for daily builds of pending submission
You can now use:
repo download [--cherry-pick|-c] project changeid/patchnumber
This will parse the manifest, cd to the corresponding project
download the changes to FETCH_HEAD and cherry-pick the result.
This is useful to automate cherry-picking of a patch
in the context of build automation, and commit gating (e.g. buildbot)
Change-Id: Ib638afd87677f1be197afb7b0f73c70fb98909fe
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
There are use-cases when fetching all branch is impractical and
we really need to fetch only one branch/tag.
e.g. there is a large project with binaries and every update of a
binary file is put to a separate branch.
The whole project history might be too large to allow users fetch it.
Add 'sync-c' option to 'project' and 'default' tags to make it possible
to configure 'sync-c' behavior at per-project and per-manifest level.
Note that currently there is no possibility to revert boolean flag from
command line. If 'sync-c' is set in manifest then you cannot make
full fetch by providing a repo tool argument.
Change-Id: Ie36fe5737304930493740370239403986590f593
Every project is in group "default". "-default" does not remove
it from this project. All group names specified in the manifest
are positive names as opposed to a mix of negative and positive.
Specified groups are resolved in order. If init is supplied with
--groups="group1,-group2", the following describes the project
selection when syncing:
* all projects in "group1" will be added, and
* all projects in "group2" will be removed.
Change-Id: I1df3dcdb64bbd4cd80d675f9b2d3becbf721f661
Allow the optional addition of "annotation" nodes nested under
projects. Each annotation node must have "name" and "value"
attributes. These name/value pairs will be exported into the
environment during any forall command, prefixed with "REPO__"
In addition, an optional "keep" attribute with case insensitive "true"
or "false" values can be included to determine whether the annotation
will be exported with 'repo manifest'
Change-Id: Icd7540afaae02c958f769ce3d25661aa721a9de8
Signed-off-by: James W. Mills <jameswmills@gmail.com>
Previously repo had incorrect code that did not really check
if sha1 presents in a project. It worked for tags though.
Check if a revision (either tag or sha1) is present by using
'git rev_parse' functionality.
Change-Id: I1787f3348573948573948753987394839487572b
Allows specifying a list of groups with a -g argument to repo init.
The groups act on a group= attribute specified on projects in the
manifest.
All projects are implicitly labelled with "default" unless they are
explicitly labelled "-default".
Prefixing a group with "-" removes matching projects from the list
of projects to sync.
If any non-inverted manifest groups are specified, the default label
is ignored.
Change-Id: I3a0dd7a93a8a1756205de1d03eee8c00906af0e5
Reviewed-on: https://gerrit-review.googlesource.com/34570
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
The -u option causes 'repo diff' to generate diff output
with file paths relative to the repository root,
so the output can be applied to the Unix 'patch' command.
The name '-u' was selected for convenience, because
both 'diff' and 'git diff' accept the option with the same name
to generate an 'unified diff' output suitable for 'patch' command.
Change-Id: I79c8356db4ed20ecaccc258b3ba139db76666fe0
Reviewed-on: https://gerrit-review.googlesource.com/34380
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
401: Unauthorized, authentication may be required. This is usually
handled internally by the HTTP client in Python. If it reaches
our code in repo, the Python HTTP client didn't find a password
in ~/.netrc that it could use.
403: Authentication was supplied, but is incorrect. It might be
that the CDN doesn't want to offer this clone.bundle file
to the client, but the Git fetch operation would still be
successful. This might arise if branch level read controls
were used in Gerrit Code Review and the /clone.bundle file
contained branches not visible to the client.
404: The server has no /clone.bundle file available.
In all of these cases, sliently ignore the /clone.bundle file HTTP
error and let the Git operation take over.
Change-Id: I1787f3cac86b017351952bbb793fe5874d83c72b
In case of manifest/smart sync repo changes ".merge" config
option from branch to SHA. Doing 'repo upload' fails as
repo tries to upload to a remote branch that looks like SHA
(e.g. refs/for/23423423423423423423423)
Do not update the .merge in case if revision is SHA.
Change-Id: I9139708fa17f21eec5a7e23c3255333626bf529e
Clients might be using their own special git-remote-* helper that
has a hypen in its name. Permit - in the scheme part of the URL
when trying to decide if it is an SSH URL and assume it is *not*
SSH if the URL matches "foo-bar://" style.
Change-Id: I7ba2d810a614f6e605a441d5972902c4a14e73fd
repo status just prints "# on branch oprofile" if you have branched
in clean status. This doesn't really tell which branch is meant.
Instead we can use the same syntax with modified gits which will
give us detailed information.
Change-Id: I55fe5154d278e10a814281dd2ba501ec6e956730
This new attribute can prevent 'repo sync' from automatically rebasing.
I hit a situation in where one of the git repositories I was tracking
was actually an external repository that I wanted to pull commits
into and merge myself. (NOT rebase, since that would lose the merge
history.) In this case, I'm not using 'repo upload', I'm manually
managing the merges to and from this repository.
Everything was going great until I typed 'repo sync' and it rebased
my manually-merged tree. Hence the option to skip it.
Change-Id: I965e0dd1acb87f4a56752ebedc7e2de1c502dbf8
Occassionally, the content-length may be missing when using urlib
in python 2.6 and 2.7. This change assumes the value of the header is
0 if it doesn't exist
Change-Id: Iaf1c8a796bc667823d4d7c30f9b617644b271d00
If SSH is not available, Gerrit returns NOT_AVAILABLE to the /ssh_info
query made by repo upload. In this case fallback to the /p/$PROJECT URL
that Gerrit also exports and use that for uploads.
Change-Id: I1e3e39ab709ecc0a692614a41a42446426f39c08
There is also shortcuts in case if the "current branch" is
a persistent revision such as tag or sha1. We check if the
persistent revision is present locally and if it does - do
no fetch anything from the server.
This greately reduces sync time and size of the on-disk repo
Change-Id: I23c6d95185474ed6e1a03c836a47f489953b99be
If the remote is using authenticated HTTP, but does not have
$GIT_URL/clone.bundle files in each repository, an initial sync
would fail around 8 projects in due to the library not resetting
the number of failures after getting a 404.
Work around this by updating the retry counter ourselves.
The urllib2 library is also not thread-safe. Make it somewhat
safer by wrapping the critical section with a lock.
Change-Id: I886e2750ef4793cbe2150c3b5396eb9f10974f7f
Signed-off-by: Shawn O. Pearce <sop@google.com>
Not every version of urllib2 supplies a reason object on the
HTTPError exception that it throws from urlopen(). Work around
this by using str(e) instead and hope the string formatting includes
sufficient information.
Change-Id: I0f4586dba0aa7152691b2371627c951f91fdfc8d
Signed-off-by: Shawn O. Pearce <sop@google.com>
urllib2 returns a malformed HTTPError object in certain situations.
For example, urllib2 has a couple of places where it creates an
HTTPError object with no fp:
if self.retried > 5:
# retry sending the username:password 5 times before failing.
raise HTTPError(req.get_full_url(), 401, "basic auth failed",
headers, None)
When it does that, HTTPError's ctor doesn't call through to
addinfourl's ctor:
# The addinfourl classes depend on fp being a valid file
# object. In some cases, the HTTPError may not have a valid
# file object. If this happens, the simplest workaround is to
# not initialize the base classes.
if fp is not None:
self.__super_init(fp, hdrs, url, code)
Which means the 'headers' slot in addinfourl is not initialized and
info() fails. It is completely insane that urllib2 decides not to
initialize its own base class sometimes.
Change-Id: I32a0d738f71bdd7d38d86078b71d9001e26f1ec3
Signed-off-by: Shawn O. Pearce <sop@google.com>
After a $GIT_URL/clone.bundle has been applied to the new local
repository, perform an incremental fetch using `git fetch` to ensure
the local repository is up-to-date. This allows the hosting server
to offer stale /clone.bundle files to bootstrap a new client.
If a single git fetch fails, it may succeed again after a short
delay. Transient failures are typical in environments where the
remote Git server happens to have limits on how many requests it
can serve at once (the anonymous git daemon, or an HTTP server).
Wait a randomized delay between 30 and 45 seconds and retry the
failed project once. This delay gives the site time to recover
from a transient traffic spike, and the randomization makes it less
likely that a spike occurs again from all of the same clients.
Change-Id: I97fb0fcb33630fb78ac1a21d1a4a3e2268ab60c0
Signed-off-by: Shawn O. Pearce <sop@google.com>
An HTTP (or HTTPS) based remote server may now offer a 'clone.bundle'
file in each repository's Git directory. Over an http:// or https://
remote repo will first ask for '$URL/clone.bundle', and if present
download this to bootstrap the local client, rather than relying
on the native Git transport to initialize the new repository.
Bundles may be hosted elsewhere. The client automatically follows a
HTTP 302 redirect to acquire the bundle file. This allows servers
to direct clients to cached copies residing on content delivery
networks, where the bundle may be closer to the end-user.
Bundle downloads are resumeable from where they last left off,
allowing clients to initialize large repositories even when the
connection gets interrupted.
If a bundle does not exist for a repository (a HTTP 404 response
code is returned for '$URL/clone.bundle'), the native Git transport
is used instead. If the client is performing a shallow sync, the
bundle transport is not used, as there is no way to embed shallow
data into the bundle.
Change-Id: I05dad17792fd6fd20635a0f71589566e557cc743
Signed-off-by: Shawn O. Pearce <sop@google.com>
The manifest project has - by design - not a review URL associated
with it. It is actually not even a 'project' in repo's sense.
This will prevent the commit-msg hook from being added, which is
not necessarily wanted as the project is managed in gerrit.
This commit will enable the commit-msg hook, which in turn will
add the Change-Id-line to every new commit in it. This simplifies
replacing patch sets (by git push ... refs/for/...).
Change-Id: I42d0f6fd79e6282d9d47074a3819e68d968999a7
Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
