We can't just let this run wild with a high (or even low) -j, since
that would hose a system. Instead, limit the total number of threads
across all git gc subprocesses to the number of CPUs reported by the
multiprocessing module (available in Python 2.6 and above).
Change-Id: Icca0161a1e6116ffa5f7cfc6f5faecda510a7fb9
The repo list -r command will execute a regex search for every
argument provided on both the project name and the project
worktree path.
Useful for finding rarely used gits.
Change-Id: Iaff90dd36c240b3d5d74817d11469be22d77ae03
Try to more accurately estimate which projects take the longest to
sync by keeping an exponentially weighted moving average (a=0.5) of
fetch times, rather than just recording the last observation. This
should discount individual outliers (e.g. an unusually large project
update) and hopefully allow truly slow repos to bubble to the top.
Change-Id: I72b2508cb1266e8a19cf15b616d8a7fc08098cb3
Some projects may consistently take longer to fetch than others, for
example a more active project may have many more Gerrit changes than a
less active project, which take longer to transfer. Use a simple
heuristic based on the last fetch time to fetch slower projects first,
so we do not tend to spend the end of the sync fetching a small number
of outliers.
This algorithm is probably not optimal, and due to inter-run latency
variance and Python thread scheduling, we may not even have good
estimates of a project sync time.
Change-Id: I9a463f214b3ed742e4d807c42925b62cb8b1745b
"except Exception as e" instead of "except Exception, e"
This is part of a transition to supporting Python 3. Python >= 2.6
support "as" syntax.
Note: this removes Python 2.5 support.
Change-Id: I309599f3981bba2b46111c43102bee38ff132803
pylint configuration file (.pylintrc) is added, and submission
instructions are updated to include pylint usage steps.
Deprecated pylint suppression (`disable-msg`) is updated in a few
modules to make it work properly with the latest version (0.26).
Change-Id: I4ec2ef318e23557a374ecdbf40fe12645766830c
We need a representation of git-submodule in repo; otherwise repo will
not sync submodules, and leave workspace in a broken state. Of course
this will not be a problem if all projects are owned by the owner of the
manifest file, who may simply choose not to use git-submodule in all
projects. However, this is not possible in practice because manifest
file owner is unlikely to own all upstream projects.
As git submodules are simply git repositories, it is natural to treat
them as plain repo projects that live inside a repo project. That is,
we could use recursively declared projects to denote the is-submodule
relation of git repositories.
The behavior of repo remains the same to projects that do not have a
sub-project within. As for parent projects, repo fetches them and their
sub-projects as normal projects, and then checks out subprojects at the
commit specified in parent's commit object. The sub-project is fetched
at a path relative to parent project's working directory; so the path
specified in manifest file should match that of .gitmodules file.
If a submodule is not registered in repo manifest, repo will derive its
properties from itself and its parent project, which might not always be
correct. In such cases, the subproject is called a derived subproject.
To a user, a sub-project is merely a git-submodule; so all tips of
working with a git-submodule apply here, too. For example, you should
not run `repo sync` in a parent repository if its submodule is dirty.
Change-Id: I541e9e2ac1a70304272dbe09724572aa1004eb5c
Fixing more issues found with pylint. Some that were supposed to
have been fixed in the previous sweep (Ie0db839e) but were missed:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
And some more:
W0631: Using possibly undefined loop variable 'name'
W0223: Method 'name' is abstract in class 'name' but is not overridden
W0231: __init__ method from base class 'name' is not called
Change-Id: Ie119183708609d6279e973057a385fde864230c3
Fix the following issues reported by pylint:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
W0612: Unused variable 'name'
W0613: Unused argument 'name'
W0102: Dangerous default value 'value' as argument
W0105: String statement has no effect
Also fixed a few cases of inconsistent indentation.
Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
Change 9bb1816b removed part of a block of code, but left the
remaining part unreachable. Remove it.
Change-Id: Icdc6061d00e6027df32dee9a3bad3999fe7cdcbc
Currently when doing a sync against a revision locked manifest,
sync has no option but to fall back to sync'ing the entire refs space;
it doesn't know which ref to ask for that contains the sha1 it wants.
This sucks if we're in -c mode; thus when we generate a revision
locked manifest, record the originating branch- and try syncing that
branch first. If the sha1 is found within that branch, this saves
us having to pull down the rest of the repo- a potentially heavy
saving.
If that branch doesn't have the desired sha1, we fallback to sync'ing
everything.
Change-Id: I99a5e44fa1d792dfcada76956a2363187df94cf1
Add two new command line options, -u/--manifest-server-username and
-p/--manifest-server-password, which can be used to specify a username
and password to authenticate to the manifest server when using the
-s/--smart-sync or -t/--smart-tag option.
If -u and -p are not specified when using the -s or -t option, use
authentication credentials from the .netrc file (if there are any).
Authentication credentials from -u/-p or .netrc are not used if the
manifest server specified in the manifest file already includes
credentials.
Change-Id: I6cf9540d28f6cef64c5694e8928cfe367a71d28d
manifest_xml: import `HEAD` and `R_HEADS` from correct module
version: import `HEAD` from correct module
`HEAD` and `R_HEADS` should be imported from the git_refs module,
where they are originally defined, rather than from the project
module.
repo: remove unused import of readline
cherry_pick: import standard modules on separate lines
smartsync: import subcmd modules explicitly from subcmd
Use:
`import re
import sys`
and
`from subcmds.sync import Sync`
Instead of:
`import sys, re`
and
`from sync import Sync`
Change-Id: Ie10dd6832710939634c4f5c86b9ba5a9cd6fc92e
When using the --smart-sync or --smart-tag option, and the specified
manifest server is hosted on a server that requires authentication,
repo sync fails with the error: HTTP 401 Unauthorized.
Add support for getting the credentials from the .netrc file.
If a .netrc file exists in the user's home directory, and it contains
credentials for the hostname of the manifest server specified in the
manifest, use the credentials to authenticate with the manifest server
using the URL syntax extension for Basic Authentication:
http://user:password@host:port/path
Credentials from the .netrc file are only used if the manifest server
URL specified in the manifest does not already include credentials.
Change-Id: I06e6586e8849d0cd12fa9746789e8d45d5b1f848
`R_HEADS` is imported twice, from both the git_refs and project
modules.
It is actually defined in git_refs, and in project it is imported
from there, so the import of `R_HEADS` from project in the sync
module is redundant. Remove it.
`HEAD` is imported from project, but like `R_HEADS` it is actually
defined in git_refs. Import it from git_refs instead.
Change-Id: I8e2b0217d0d9f9f4ee5ef5b8cd0b026174ac52f4
When connecting to the manifest server, exceptions can occur but
are not caught, resulting in the repo sync exiting with a python
traceback.
Add handling of the following exceptions:
- IOError, which can be raised for example if the manifest server
URL is malformed.
- xmlrpclib.ProtocolError, which can be raised if the connection
to the manifest server fails with HTTP error.
- xmlrpclib.Fault, which can be raised if the RPC call fails for
some other reason.
Change-Id: I3a4830aef0941debadd515aac776a3932e28a943
Instead of every group being in the group "default", every project
is now in the group "all". A group that should not be downloaded
by default may be added to the group "notdefault".
This allows all group names to be positive (instead of removing groups
directly in the manifest with -default) and offers a clear way of
selecting every project (--groups all).
Change-Id: I99cd70309adb1f8460db3bbc6eff46bdcd22256f
The threaded 'repo sync' implementation would very often freeze the
process when interrupted by the user with Ctrl-C. The only solution
being to kill -9 the process explicitly from another terminal.
The reason for this is best explained here:
http://snakesthatbite.blogspot.fr/2010/09/cpython-threading-interrupting.html
This patch makes all helper sync threads 'daemon', which allows the
process to terminate immediately on Ctrl-C.
Note that this will forcefully kill all threads in case of interruption; this
is generally a bad thing, but:
1/ This is equivalent to calling kill -9 in another terminal, which
is the _only_ thing that can currently stop the process.
2/ There doesn't seem to be a way to tell the worker threads to
gently stop when they are in a blocking operation anyway (even
in the non-threaded case).
+ Do the same for "repo status -j<count>".
Change-Id: Ieaf45b0eacee36f35427f8edafd87415c2aa7be4
The overview command shows an overview of each branch in all (or the
specified) projects. The overview lists any local commits that have
not yet been merged into the project.
The report output is inspired by the report displayed following a
"repo prune" event, with the addition of listing the one-line log
messages for each commit that is not yet merged.
The report can also be filtered to show only active branches; by
default all branches that have commits beyond the upstream HEAD will
be listed.
Change-Id: Ibe67793991ad1aa38de3bc9747de4ba64e5591aa
Currently repo-rebase requires that all modifications be committed
locally before it will allow the rebase. In high-velocity environments,
you may want to just pull in newer code without explicitly creating
local commits, which is typically achieved using git-stash.
If called with the --auto-stash command line argument, and it is
determined that the current index is dirty, the local modifications
are stashed, and the rebase continues. If a stash was performed, that
stash is popped once the rebase completes.
Note that there is still a possibility that the git-stash pop will
result in a merge conflict.
Change-Id: Ibe3da96f0b4486cb7ce8d040639187e26501f6af
See repo issue #46 :
https://code.google.com/p/git-repo/issues/detail?id=46
When using repo init -b on an already existing repository,
the next sync will try to rebase changes coming from the old manifest
branch onto the new, leading in the best case scenario to conflicts
and in the worst case scenario to an incorrect "mixed up" manifest.
This patch fixes this by deleting the "default" branch in the local
manifest repository when the -d init switch is used, thus forcing
repo to perform a fresh checkout of the new manifest branch
Change-Id: I379e4875ec5357d8614d1197b6afbe58f9606751
This patch adds the option to include topic branches by adding the
following to a .gitconfig file:
uploadtopic = true
This option is only read in when the -t option is not already
specified at the command line.
Change-Id: I0e0eea49438bb4e4a21c2ac5bd498b68b5a9a845
Allows to ff-only a gerrit patch
This patch is necessary to automatically ensure that the patch will
be correctly submitted on ff-only gerrit projects
You can now use:
repo download (--ff-only|-f) project changeid/patchnumber
This is useful to automate verification of fast forward status of a patch
in the context of build automation, and commit gating (e.g. buildbot)
Change-Id: I403a667557a105411a633e62c8eec23d93724b43
Signed-off-by: Erwan Mahe <erwan.mahe@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
BZ: 4779
Allows to revert a gerrit patch
This patch is necessary for the on-demand creation of
engineering builds using buildbot
You can now use:
repo download [--revert|-r project changeid/patchnumber
This is useful to automate reverting of a patch
in the context of build automation, and regression bisection
Change-Id: I3985e80e4b2a230f83526191ea1379765a54bdcf
Signed-off-by: Erwan Mahe <erwan.mahe@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
default option uses git checkout, and thus overwrite the previous
checkouts. this is a problem for automated builds of several
changesets in the same project for daily builds of pending submission
You can now use:
repo download [--cherry-pick|-c] project changeid/patchnumber
This will parse the manifest, cd to the corresponding project
download the changes to FETCH_HEAD and cherry-pick the result.
This is useful to automate cherry-picking of a patch
in the context of build automation, and commit gating (e.g. buildbot)
Change-Id: Ib638afd87677f1be197afb7b0f73c70fb98909fe
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
repo status should output filenames one by one instead of trying to
build a string from incompatible encodings (like utf-8 and sjis
filenames)
Change-Id: I52282236ececa562f109f9ea4b2e971d2b4bc045
Fixes three errors:
Python doesn't like the line wrap after 'and'.
platform.system is a function, needs to be platform.system().
Typo all_platfroms instead of all_platforms.
Change-Id: Ia875e521bc01ae2eb321ec62d839173c00f86c2d
Projects may optionally specify their platform
(eg, groups="platform-linux" in the manifest).
By default, repo will automatically detect the platform. However,
users may specify --platform=[auto|all|linux|darwin].
Change-Id: Ie678851fb2fec5b0938aede01f16c53138a16537
Every project is in group "default". "-default" does not remove
it from this project. All group names specified in the manifest
are positive names as opposed to a mix of negative and positive.
Specified groups are resolved in order. If init is supplied with
--groups="group1,-group2", the following describes the project
selection when syncing:
* all projects in "group1" will be added, and
* all projects in "group2" will be removed.
Change-Id: I1df3dcdb64bbd4cd80d675f9b2d3becbf721f661
Allow the optional addition of "annotation" nodes nested under
projects. Each annotation node must have "name" and "value"
attributes. These name/value pairs will be exported into the
environment during any forall command, prefixed with "REPO__"
In addition, an optional "keep" attribute with case insensitive "true"
or "false" values can be included to determine whether the annotation
will be exported with 'repo manifest'
Change-Id: Icd7540afaae02c958f769ce3d25661aa721a9de8
Signed-off-by: James W. Mills <jameswmills@gmail.com>
Allows specifying a list of groups with a -g argument to repo init.
The groups act on a group= attribute specified on projects in the
manifest.
All projects are implicitly labelled with "default" unless they are
explicitly labelled "-default".
Prefixing a group with "-" removes matching projects from the list
of projects to sync.
If any non-inverted manifest groups are specified, the default label
is ignored.
Change-Id: I3a0dd7a93a8a1756205de1d03eee8c00906af0e5
Reviewed-on: https://gerrit-review.googlesource.com/34570
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
The -u option causes 'repo diff' to generate diff output
with file paths relative to the repository root,
so the output can be applied to the Unix 'patch' command.
The name '-u' was selected for convenience, because
both 'diff' and 'git diff' accept the option with the same name
to generate an 'unified diff' output suitable for 'patch' command.
Change-Id: I79c8356db4ed20ecaccc258b3ba139db76666fe0
Reviewed-on: https://gerrit-review.googlesource.com/34380
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
A convenient equivalent to `repo upload --br=<current git branch>`.
Note that the head branch will be selected for each project
uploaded by repo, so different branches may be uploaded for
different projects.
Change-Id: I10ad8ceaa63f055105c2d847c6e329fa4226dbaf
repo status just prints "# on branch oprofile" if you have branched
in clean status. This doesn't really tell which branch is meant.
Instead we can use the same syntax with modified gits which will
give us detailed information.
Change-Id: I55fe5154d278e10a814281dd2ba501ec6e956730
This parameter changes the manifest used by 'repo sync' for only
this execution. It should be useful for developers wishing to get
the repo temporarily into a known state, without clobbering their
existing manifest.
Tested by shifting Chrome OS between minilayout and full, and
between several release-builder-generated manifests.
Change-Id: I14194b665195b0e78f368d9ec8b8a83227af2627
This reverts commit ee1c2f5717.
This breaks a lot of buildbot systems. Rolling it back for now
until we can understand what the breakage was and how to fix it.
If the user has already configured a workspace, use these values
when re-running 'repo init'.
Otherwise, if the user has global name and e-mail set, use these.
It's always possible to override this and be prompted by specifying
--config-name when running 'repo init'.
Change-Id: If45f0e4b14884071439fb02709dc5cb53f070f60
A default manifest URL can be specified using:
git config --global repo-manifest.<id>.url <url>
A default manifest server can be specified using:
git config --global repo-manifest.<id>.server <url>
A default git mirror reference can be specified using:
git config --global repo-manifest.<id>.reference <path>
This will allow the user to use 'repo init -u <id>' as
a shorter alternative to specifying the full URL.
Also, manifest server will not have to be specified in the
manifest XML and the reference will not have to be specified
on the command line. If they are, they will override these
default values however.
Change-Id: Ifdbc160bd5909ec7df9efb0c5d7136f1d9351754
Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
Several times one have done an upload only to later notice in gerrit
that the upload was done to the wrong branch as the git has not yet
been branched for the current git. This change will make repo print
what the destination branch is when asking the user if she wants to
go through with the upload.
Change-Id: Ia9c3a92a6a04c022edfebf4f8d651ac062bb1f3b
It is common in command line tools to indicate what the default answer
will be if the user simply hits enter. In repo, the display is just
"y/n" with no indication as to which is the default. So change the n
to N in the messages since that is how repo operates.
Change-Id: I81819ae630355072eb0365e59168b0921289498f
When commit with comment that has non-ASCII characters,
UnicodeDecodeError will be raised
while uploading multiple project/branch changes.
Because some strings in script are not str type, but unicode.
So all the strings are decoded to unicode,
and python use ascii to do this,
it can not decode non-ASCII characters,
so UnicodeDecodeError raised.
Signed-off-by: chenguodong <chenguodong@huawei.com>
Change-Id: I46447f489a4b9760a5899c7ba9d764b688594e46
There is also shortcuts in case if the "current branch" is
a persistent revision such as tag or sha1. We check if the
persistent revision is present locally and if it does - do
no fetch anything from the server.
This greately reduces sync time and size of the on-disk repo
Change-Id: I23c6d95185474ed6e1a03c836a47f489953b99be
help sync crashed as sync required the manifest to be configured to
create the option parser, as the default number of jobs is required.
Change-Id: Ie75e8d75ac0e38313e4aab451cbb24430e84def5
Signed-off-by: Shawn O. Pearce <sop@google.com>
An HTTP (or HTTPS) based remote server may now offer a 'clone.bundle'
file in each repository's Git directory. Over an http:// or https://
remote repo will first ask for '$URL/clone.bundle', and if present
download this to bootstrap the local client, rather than relying
on the native Git transport to initialize the new repository.
Bundles may be hosted elsewhere. The client automatically follows a
HTTP 302 redirect to acquire the bundle file. This allows servers
to direct clients to cached copies residing on content delivery
networks, where the bundle may be closer to the end-user.
Bundle downloads are resumeable from where they last left off,
allowing clients to initialize large repositories even when the
connection gets interrupted.
If a bundle does not exist for a repository (a HTTP 404 response
code is returned for '$URL/clone.bundle'), the native Git transport
is used instead. If the client is performing a shallow sync, the
bundle transport is not used, as there is no way to embed shallow
data into the bundle.
Change-Id: I05dad17792fd6fd20635a0f71589566e557cc743
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the manifest is updated and the default sync-j attribute
was modified, honor it during this sync session if the user
has not supplied a -j flag on the command line.
Change-Id: I127ee5c779e2bbbb40b30bddc10ec1fa704b3bf3
Signed-off-by: Shawn O. Pearce <sop@google.com>
This permits manifest authors to suggest a number of parallel
fetch operations against a remote server. For example, Gerrit
Code Review servers support queuing of requests and processes
them in first-in, first-out order. Running concurrent fetches
can utilize multiple CPUs on the Gerrit server, but will also
decrease overall operation latency by having the request put
into the queue ready to execute as soon as a CPU is free.
Change-Id: I3d3904acb6f63516bae4b071c510ad57a2afab18
Signed-off-by: Shawn O. Pearce <sop@google.com>
Each worker thread requires at least 3 file descriptors to run the
forked 'git fetch' child to operate against the local repository.
Mac OS X has the RLIMIT_NOFILE set to 256 by default, which means
a sync -j128 often fails when the workers run out of pipes within
the Python parent process.
Change-Id: I2cdb14621b899424b079daf7969bc8c16b85b903
Signed-off-by: Shawn O. Pearce <sop@google.com>
This is an evolution of 'smart-sync' that adds a new option, -t,
that allows you to specify a tag/label to use instead of the
"latest good build" on the current manifest branch which -s does.
Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
Change-Id: I8c20fd91104a6aafa0271d4d33f6c4850aade17e
This commit adds a --br=<branch> option to repo upload.
repo currently examines every non-published branch. This is problematic
for my workflow. I have many branches in my kernel tree. Many of these
branches are based off of upstream remotes (I have many remotes) and
will never be uploaded (they'll get sent upstream as a patch).
Having repo scan these branches adds to my upload processing time
and clutters the branch selection buffer. I've also seen repo get
confused when one of my branches is 1000s of commits different from
m/master.
Change-Id: I68fa18951ea59ba373277b57ffcaf8cddd7e7a40
It is undesired to have the same Change-Id:-line for two separate
commits, and when cherry-picking, the user must manually change it.
If this is not done, bad things may happen (such as when the user
is uploading the cherry-picked commit to Gerrit, it will instead
see it as a new patch-set for the original change, or worse).
repo cherry-pick works the same was as git cherry-pick, except that
it replaces the Change-Id with a new one and adds a reference
back to the commit from where it was picked.
On failures (when git can not successfully apply the cherry-picked
commit), instructions will be written to the user.
Change-Id: I5a38b89839f91848fad43386d43cae2f6cdabf83
In the current version of repo checkout, we often get the error:
error: no project has branch xyzzy
...even when the actual error was something else. This fixes it
to only report the 'no project has branch' when that is actually true.
This fix is very similar to one made for 'repo abandon':
https://review.source.android.com/#change,22207
The repo checkout error is filed as: <http://crosbug.com/6514>
TEST=manual
A sample creating a case where 'git checkout' will fail:
$ repo start branch1 .
$ repo start branch2 .
$ touch bogusfile
$ git add bogusfile
$ git commit -m "create bogus file"
[branch2 f8b6b08] create bogus file
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 bogusfile
$ echo "More" >> bogusfile
$ repo checkout branch1 .
error: chromite/: cannot checkout branch1
A sample case showing that we still fail if no project has a branch:
$ repo checkout xyzzy .
error: no project has branch xyzzy
Change-Id: I48a8e258fa7a9c1f2800dafc683787204bbfcc63
This is the simplest fix: if we had problems syncing the
manifest.git directory and we were the ones that created it,
we should delete it. This doesn't try to do anything complex
like try to recover from a .repo directory that got broken in
some other way.
This is filed as: <http://crosbug.com/13403>
TEST=manual
Init once with a bad URL:
$ repo init -u http://foobar.example.com
Getting manifest ...
from http://foobar.example.com
Connection closed by 172.22.121.77
error: Couldn't resolve host 'foobar.example.com' while accessing http://foobar.example.com/info/refs
fatal: HTTP request failed
fatal: cannot obtain manifest http://foobar.example.com
Init again: identical to the first. Good:
$ repo init -u http://foobar.example.com
Getting manifest ...
from http://foobar.example.com
Connection closed by 172.22.121.77
error: Couldn't resolve host 'foobar.example.com' while accessing http://foobar.example.com/info/refs
fatal: HTTP request failed
fatal: cannot obtain manifest http://foobar.example.com
Init with correct URL:
$ repo init -u http://git.chromium.org/git/manifest -m minilayout.xml
Getting manifest ...
from http://git.chromium.org/git/manifest
[ ... cut ... ]
repo initialized in /.../repoiniterr
Try a bad URL after a good one; it doesn't get saved (good):
$ repo init -u http://foobar.example.com
Connection closed by 172.22.121.77
error: Couldn't resolve host 'foobar.example.com' while accessing http://foobar.example.com/info/refs
fatal: HTTP request failed
fatal: cannot obtain manifest http://foobar.example.com
Just to confirm, I can still do a good one after a bad...
$ repo init -u http://git.chromium.org/git/manifest -m minilayout.xml
Your Name [George Washington]:
Your Email [george@washington.example.com]:
Your identity is: George Washington <george@washington.example.com>
is this correct [y/n]? y
repo initialized in /.../repoiniterr
Change-Id: I1692821a330d97b1d218b2e191a93245b33f2362
The main fix is to give an error message if nothing was actually
abandoned. See <http://crosbug.com/6041>.
The secondary fix is to list projects where the abandon happened.
This could be done in a separate CL or dropped altogether if requested.
TEST=manual
$ repo abandon dougabc; echo $?
Abandon dougabc: 100% (127/127), done.
Abandoned in 2 project(s):
chromite
src/platform/init
0
$ repo abandon dougabc; echo $?
Abandon dougabc: 100% (127/127), done.
error: no project has branch dougabc
1
$ repo abandon dougabc; echo $?
Abandon dougabc: 100% (127/127), done.
error: chromite/: cannot abandon dougabc
1
Change-Id: I79520cc3279291acadc1a24ca34a761e9de04ed4
Event.isSet was renamed to is_set in 2.6, but we should
use the earlier syntax to avoid breaking compatibility
with older Python installations.
Change-Id: I41888ed38df278191d7496c1a6eed15e881733f4
The bug that this is fixing is described here:
http://code.google.com/p/chromium-os/issues/detail?id=6813
This fix allows the helper threads to signal the main thread that they
saw an error. When the main thread sees the error, it will let all
existing threads finish, then exit with an error.
Change-Id: If3019bc6b0b3ab9304d49ed2eea53e9d57f3095a
All repo-level hooks are expected to live in a single project at the
top level of that project. The name of the hooks project is provided
in the manifest.xml. The manifest also lists which hooks are enabled
to make it obvious if a file somehow failed to sync down (or got
deleted).
Before running any hook, we will prompt the user to make sure that it
is OK. A user can deny running the hook, allow once, or allow
"forever" (until hooks change). This tries to keep with the git
spirit of not automatically running anything on the user's computer
that got synced down. Note that individual repo commands can add
always options to avoid these prompts as they see fit (see below for
the 'upload' options).
When hooks are run, they are loaded into the current interpreter (the
one running repo) and their main() function is run. This mechanism is
used (instead of using subprocess) to make it easier to expand to a
richer hook interface in the future. During loading, the
interpreter's sys.path is updated to contain the directory containing
the hooks so that hooks can be split into multiple files.
The upload command has two options that control hook behavior:
- no-verify=False, verify=False (DEFAULT):
If stdout is a tty, can prompt about running upload hooks if needed.
If user denies running hooks, the upload is cancelled. If stdout is
not a tty and we would need to prompt about upload hooks, upload is
cancelled.
- no-verify=False, verify=True:
Always run upload hooks with no prompt.
- no-verify=True, verify=False:
Never run upload hooks, but upload anyway (AKA bypass hooks).
- no-verify=True, verify=True:
Invalid
Sample bit of manifest.xml code for enabling hooks (assumes you have a
project named 'hooks' where hooks are stored):
<repo-hooks in-project="hooks" enabled-list="pre-upload" />
Sample main() function in pre-upload.py in hooks directory:
def main(project_list, **kwargs):
print ('These projects will be uploaded: %s' %
', '.join(project_list))
print ('I am being a good boy and ignoring anything in kwargs\n'
'that I don\'t understand.')
print 'I fail 50% of the time. How flaky.'
if random.random() <= .5:
raise Exception('Pre-upload hook failed. Have a nice day.')
Change-Id: I5cefa2cd5865c72589263cf8e2f152a43c122f70
Users may wind up with a lot of loose object content in projects they
don't frequently make changes in, but that are modified by others.
Since we bypass many git code paths that would have otherwise called
out to `git gc --auto`, its possible for these projects to have
their loose object database grow out of control. To help prevent
that, we now invoke it ourselves during the network half of sync.
Signed-off-by: Shawn O. Pearce <sop@google.com>
(cherry picked from commit 1875ddd47c)
Instead of giving a Python backtrace when there is a connectivity
problem during repo upload, report that we cannot access the host,
and why, with a halfway decent error message.
Bug: REPO-45
Change-Id: I9a45b387e86e48073a2d99bd6d594c1a7d6d99d4
Signed-off-by: Shawn O. Pearce <sop@google.com>
(cherry picked from commit d2dfac81ad)
If a project is missing locally, it might be OK to skip over it
and continue running the same command in other projects.
Bug: REPO-43
Change-Id: I64f97eb315f379ab2c51fc53d24ed340b3d09250
Signed-off-by: Shawn O. Pearce <sop@google.com>
(cherry picked from commit d4cd69bdef)
Windows allows the environment to have unicode values.
This will cause Python to fail to execute the command.
Change-Id: I37d922c3d7ced0d5b4883f0220346ac42defc5e9
Signed-off-by: Shawn O. Pearce <sop@google.com>
This feature is used to convey information on a when a branch has
ceased development or if it is an experimental branch with a few
gotchas, etc.
You add it to your manifest XML by doing something like this:
<manifest>
<notice>
NOTE TO DEVELOPERS:
If you checkin code, you have to pinky-swear that it contains no bugs.
Anyone who breaks their promise will have tomatoes thrown at them in the
team meeting. Be sure to bring an extra set of clothes.
</notice>
<remote ... />
...
</manifest>
Carriage returns and indentation are relevant for the text in this tag.
This feature was requested by Anush Elangovan on the ChromiumOS team.
This adds a new flag -f/--force-broken that will allow the rest of
the sync process to continue instead of bailing when a particular
project fails to sync.
Change-Id: I23680f2ee7927410f7ed930b1d469424c9aa246e
Signed-off-by: Andrei Warkentin <andreiw@motorola.com>
Signed-off-by: Shawn O. Pearce <sop@google.com>
It hasn't been necessary for a long time, and its
functionality can be accomplished with 'git push'.
Change-Id: Ic00d3adbe4cee7be3955117489c69d6e90106559
Use git clone to initialize a new repository, and when possible
allow callers to use --reference to reuse an existing checkout as
the initial object storage area for the new checkout.
Change-Id: Ie27f760247f311ce484c6d3e85a90d94da2febfc
Signed-off-by: Shawn O. Pearce <sop@google.com>
--replace started to fail due to a Python error, I forgot to pass
through the opt structure to the replace function.
Change-Id: Ifcd7a0c715c3fd9070a4c58208612a626382de35
Signed-off-by: Shawn O. Pearce <sop@google.com>
Passing through --whitespace=fix to rebase can be useful
to clean up a branch prior to uploading it for review.
Change-Id: Id85f1912e5e11ff9602e3b342c2fd7441abe67d7
Signed-off-by: Shawn O. Pearce <sop@google.com>
Some users might need to use a different login name than the local
part of their email address for their Gerrit Code Review user
account. Allow it to be overridden with the review.HOST.username
configuration variable.
Change-Id: I714469142ac7feadf09fee9c26680c0e09076b75
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the -t flag is given to upload, the local branch name is
automatically sent to Gerrit Code Review as the topic branch name
for the change(s). This requires the server to be Gerrit Code
Review v2.1.3-53-gd50c94e or later, which isn't widely deployed
right now, so the default is opt-out.
Change-Id: I034fcacb405b7cb909147152db427fe69dd7bcbf
Signed-off-by: Shawn O. Pearce <sop@google.com>
Usage: repo rebase [[-i] <project>...]
Rebases the current topic branch of the specified (or all)
projects against the appropriate upstream.
Note: Interactive rebase is currently only supported when
exactly one project is specified on the command line.
Change-Id: I7376e35f27a6585149def82938c1ca99f36db2c4
Signed-off-by: Shawn O. Pearce <sop@google.com>
The upload command will read review.URL.autocopy from the project's
configuration and append the list of e-mails specified to the
--cc argument of the upload command if a non-empty --re argument
was provided.
Change-Id: I2424517d17dd3444b20f0e6a003be6e70b8904f6
Signed-off-by: Shawn O. Pearce <sop@google.com>
This fixes some format string bugs in grep which cause repo to with
"TypeError: not enough arguments for format string" when grepping and
the output contains a valid Python format string.
Change-Id: Ice8968ea106148d409490e4f71a2833b0cc80816
This patch does two things for being compatibile with
those Python which are built without threading support:
1. As the Python document and Shawn suggested, import dummy_threading
when the threading is not available.
2. Reserve the single threaded code and make it default.
In cases the --jobs does not work properly with dummy_threading,
we still have a safe fallback.
Change-Id: I40909ef8e9b5c22f315c0a1da9be38eed8b0a2dc
Add a sentinel check to require a second explicit confirmation if the
user is attempting to upload (or upload --replace) an unusually large
number of commits. This may help the user to catch an accidentally
incorrect rebase they had done previously.
Change-Id: I12c4d102f90a631d6ad193486a70ffd520ef6ae0
The manifest server doesn't want to have refs/heads passed to it, so
we need to strip that when the branch contains it.
Change-Id: I044f8a9629220e886fd5e02e3c1ac4b4bb6020ba
Do not error if a project is missing on the filesystem, is deleted
from manifest.xml, but still exists in project.list.
Change-Id: I1d13e435473c83091e27e4df571504ef493282dd
This option allows the user to specify a manifest server to use when
syncing. This manifest server will provide a manifest pegging each
project to a known green build. This allows developers to work on a
known good tree that is known to build and pass tests, preventing
failed builds to hamper productivity.
The manifest used is not "sticky" so as to allow subsequent
'repo sync' calls to sync to the tip of the tree.
Change-Id: Id0a24ece20f5a88034ad364b416a1dd2e394226d
This bug happens when a project gets added to the manifest, and
then is renamed. Users who happened to have run "repo sync" after
the project was added but before the rename happened will try to
read the data from the old project, as the manifest was only updated
after all projects were updated successfully.
When someone copies and pastes a setup line from a web page,
they might actually copy 'repo sync' onto the clipboard and wind
up pasting it into the "Your Name" prompt. This means they will
initialize their client with the user name of "repo sync", creating
some rather funny looking commits later on. For example:
To setup your source tree:
mkdir ~/code
cd ~/code
repo init -u git://....
repo sync
If this entire block was just blindly copy and pasted into the
terminal, the shell won't read "repo sync" but "repo init" will.
By showing the user their full identity string, and asking them
to confirm it before we continue, we can give the hapless user a
chance to recover from this mistake, without unfairly harming those
who were actually named 'repo' by their parents.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The --color flag wasn't introduced until git 1.6.3. Prior to that
version, `git grep --color` just produces a fatal error, as it is
an unsupported option. Since this is just pretty output and is not
critical to execution, we can simply omit the option if the version
of git we are running on doesn't support it.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way we can use it to detect feature support in the underlying
git, such as new options or commands that have been added in more
recent versions.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If a line is blank in project.list, its not a relevant project path,
so skip over it. Existing project.list files may have blank lines if
sync was run with no projects at all, and the file was created empty.
Signed-off-by: Shawn O. Pearce <sop@google.com>
We have no working tree, so we cannot update the project.list
state file, nor should we try to delete a directory if a project is
removed from the manifest. Clients would still need the repository
for historical records.
Signed-off-by: Shawn O. Pearce <sop@google.com>
After a repo sync, some of the project paths might need
to be removed. This changes maintains a list of project
paths from the previous sync operation and compares.
The revisionExpr field now holds an expression from the manifest,
such as "refs/heads/master", while revisionId holds the current
commit-ish SHA-1 of the revisionExpr. Currently that is only
filled in if the manifest points directly to a SHA-1.
Signed-off-by: Shawn O. Pearce <sop@google.com>
We now feed Project a RemoteSpec, instead of the Remote directly
from the XmlManifest. This way the RemoteSpec already has the
full project URL, rather than just the base, permitting other
types of manifests to produce the URL in their own style.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Extensive discussion with users lead to the fact that needing to
supply -a to view what they really wanted to see was just wrong.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The level 2 headings (denoted by ~) indent the heading two spaces,
but continue to use the bold formatter to offset them from the
other surrounding text.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way its clear the command did something, and reported
that it had nothing to show you, because you have no active
branches in this client.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Most projects will have their branch heads matching in all branches,
so switching between them should be just a matter of updating the
work tree's HEAD symref. This can be done in pure Python, saving
quite a bit of time over forking 'git checkout'.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This is mostly useful if the number of projects to switch is many
(e.g. all of Android) and a large number of them are behind the
current manifest revision. We wind up needing to run git just to
make the working tree match, and that often makes the command take
a couple of seconds longer than we'd like.
Signed-off-by: Shawn O. Pearce <sop@google.com>
When trying to read log output from many projects at once it can
be difficult to make sense of which messages came from where.
For many professional developers it is common to want to view the
last week's worth of your work, so you can write a weekly summary
of your activity for your status report.
This is easier with the new -p option:
repo forall -pc git log --reverse --since=1.week.ago --author=sop
produces a report of all commits written by me in the last week,
formatted in a paged output display, with headers inserted in
front of each project's output.
Where this can be even more useful is with git log's pickaxe,
e.g. now we can use:
repo forall -pc git log -Sbar v1.0..v1.1
to locate all additions or removals of the symbol 'bar' since v1.0,
up to and including v1.1. Before displaying the matching commits in
a project, a project header is shown, giving the user some context
information for the matching results.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Generally we only show the project path, relative from the top of the
client. Showing the project name may be confusing for the end-user.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This gives the user the last chance to confirm where the change is
going to be sent to. Knowing the review server URL will help the
user decide if continuing with the upload makes sense.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Its unlikely that a new version of repo will be delivered in any
given day, so we now check only once every 24 hours to see if repo
has been updated. This reduces the sync cost, as we no longer need
to contact the repo distribution servers every time we do a sync.
repo selfupdate can still be used to force a check.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the user has disabled a prompt, skip the two commands we use to
obtain the list of commits and the date of the branch. These will
never be displayed and just waste the end-user's time.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If review.URL.autoupload is set to true in a project's .git/config
or in ~/.gitconfig then `repo upload` will automatically upload,
and skip prompting the end-user.
Conversely, if review.URL.autoupload is set to false, then repo
will refuse to upload to that project.
Bug: REPO-25
Signed-off-by: Shawn O. Pearce <sop@google.com>
We now try to sync all projects that can be done safely first, before
we start rebasing user commits over the upstream. This has the nice
effect of making the local tree as close to the upstream as possible
before the user has to start resolving merge conflicts, as that extra
information in other projects may aid in the conflict resolution.
Informational output is buffered and delayed until calculation for
all projects has been done, so that the user gets one concise list
of notice messages, rather than it interrupting the progress meter.
Fast-forward output is now prefixed with the project header, so the
user can see which project that update is taking place in, and make
some relation of the diffstat back to the project name.
Rebase output is now prefixed with the project header, so that if
the rebase fails, the user can see which project we were operating
on and can try to address the failure themselves.
Since rebase sits on a detached HEAD, we now look for an in-progress
rebase during sync, so we can alert the user that the given project
is in a state we cannot handle.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Users may want to upgrade only repo to the latest release, but
leave their working tree state alone and avoid 'repo sync'.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Users can now use 'repo grep' to search all projects, rather than
'repo forall -c git grep'. Its not only shorter to type, but it
also filters results better by highlighting which projects matched
in the client workspace.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If there is nothing output at all, tell the user the working tree is
completely clean. It just gives them a bit more of a warm-fuzzy
feeling knowing repo and until the end. It also more closely
matches with the output of git status.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way users can see how much is left during fetch. Its
especially useful when most syncs are no-ops but there are
hundreds of repositories to poll.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This permits usage of 'repo sync' while offline, as we bypass the
network based portions of the code and do only the local sync.
An example use case might be:
repo sync -n ; # while we have network
... some time later ...
repo sync -l ; # while without network, come up to date
Signed-off-by: Shawn O. Pearce <sop@google.com>
The -d flag moves the project back to a detached HEAD state,
matching what is listed in the manifest. This can be useful to
set a client to something stable (or at least well-known), such as
before a sequence of 'repo download' commands are used to get some
changes for testing.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This makes it easier to update all repositories, without actually
impacting the working directory, or learning about how to use
`repo forall -c 'git fetch $REPO_REMOTE' `.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This is only meant to be passed through while repo upgrades itself
during a sync. It should never be something a user invokes on
their own.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If `repo start foo` fails due to uncommitted and unmergeable changes
in a single project, we have switched half of the projects over to
the new target branches, but didn't on the one that failed to move.
This change improves the situation by doing three things differently:
- We keep going when we encounter an error, so other projects
that can successfully switch still switch.
- We ignore projects whose current branch is already on the
requested name; they are logically already setup.
- We checkout the branch if it already exists, rather than
trying to recreate the branch.
Bug: REPO-22
Signed-off-by: Shawn O. Pearce <sop@google.com>
We now display a summary of the available topic branches in this
client, based upon a sorted union of all existing projects.
Bug: REPO-21
Signed-off-by: Shawn O. Pearce <sop@google.com>
Modern Gerrit2 automatically outputs the URL for each commit to
stderr as it creates the records. Dumping the URL ourselves is
unnecessary additional output, and worse is just an approximate
guess for the correct web URL. Gerrit might not live at the top
level directory for the server, or might even prefer a different
hostname for web connections than what is listed in the manifest.
Signed-off-by: Shawn O. Pearce <sop@google.com>
On a mirror client we don't prompt for user.name,user.email as the
data is only necessary if you will make new commits. On a re-init
we were testing the command line option, not the existing IsMirror
property from the manifest configuration file.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This allows the user to run "repo init -u" again after an
initial attempt failed due to an invalid URL.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Instead of a stack trace ending in origin/master not existing we
now tell the user the manifest url is invalid if 'git fetch' has
failed out early.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the value obtained is None we now set the variable to
'' instead, in an attempt to make execve() happier about
our 3rd argument, the env dictionary.
Signed-off-by: Shawn O. Pearce <sop@google.com>
We now correctly support re-initializing an existing client to point
to a different branch of the same manifest repository, effectively
allowing the client to switch the baseline it is operating on.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Simply setting repo.mirror true doesn't make a client into a mirror.
The on-disk layout is completely wrong for a mirror repository,
and until we fix our layout for a non-mirror client to more closely
resemble the upstream we can't do anything to easily turn on or
turn off the mirror status flag.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This can be useful to create a new manifest from an existing client,
especially if the client wants to use the "-r" option to set each
project's revision to the current commit SHA-1, making a sort of a
tag file that can be used to recreate this exact state elsewhere.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If a client was created with "repo init --mirror" then there are
no working directories present, and no files checked out. Using
a command like "repo status" in this context makes no sense, and
actually throws back a Pytyon traceback at the console when the
underlying commands fail out.
We now tag commands with the MirrorSafeCommand type if they are
able to be executed within a mirror directory safely. Using a
command in a mirror which lacks this base class results in a
useful error letting you know the command isn't supported.
Bug: REPO-14
Signed-off-by: Shawn O. Pearce <sop@google.com>
REPO_PATH is the path relative the the root of the client.
REPO_REMOTE is the name of the remote system from the manifest.
REPO_LREV is the name of the revision from the manifest, but
translated to something the local repository knows.
REPO_RREV is the name of the revision from the manifest.
This allows us to do commands like:
repo forall -c 'echo "(cd $REPO_PATH && git checkout `git rev-parse HEAD`)"'
Gerrit won't permit more than one commit using the same change
number during a replacement request, so we should error out if
the user has asked for this in their upload edit script.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Users are prompted with the list of known changes we are about
to upload, and they can fill out the current change numbers for
any changes which already exist in the data store. For each of
those changes the change number and commit id is sent as part of
the upload request, so Gerrit can insert the new commit as a new
patch set of the existing change, rather than make a new change.
This facility permits developers to replace a patch so they can
address comments made on a prior version of the same change.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The mirror option downloads a complete forrest (as described by the
manifest) and creates a replica of the remote repositories rather
than a client working directory. This permits other clients to
sync off the mirror site.
A mirror can be positioned in a "DMZ", where the mirror executes
"repo sync" to obtain changes from the external upstream and
clients inside the protected zone operate off the mirror only,
and therefore do not require direct git:// access to the external
upstream repositories.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This destroys a local development branch, removing all history
of that branch from ever existing. If the branch is currently
checked out we move back to the upstream revision.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This hook is evaluated by `git gc --auto` to determine if it is a
good idea to execute a GC at this time, or defer it to some later
date. When working on a laptop its a good idea to avoid GC if you
are on battery power as the extra CPU and disk IO would consume a
decent amount of the charge.
The hook is the standard sample hook from git.git contrib/hooks,
last modified in git.git by 84ed4c5d117d72f02cc918e413b9861a9d2846d7.
I added the GPLv2 header to the script to ensure the license notice
is clear, as it does not match repo's own APLv2 license.
We only update hooks during initial repository creation or on
a repo sync. This way we don't incur huge overheads from the
hook stat operations during "repo status" or even the normal
"repo sync" cases.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Now that repo relies only on the git data stream (as it is much
faster to download through) we don't really need to be parsing the
<snapshot> elements within manifest. Its a lot of complex code to
convert the tar (or zip) through to a fast import stream, and we
just aren't calling it anymore.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Now `repo download . 1402` would download the change numbered 1402
into the current project and check it out for the user, using a
detached HEAD. `repo sync .` would back out of the change and
return to the upstream version.
Multiple projects can be fetched at once by listing them out on
the command line as different arguments.
Individual patch sets can be selected by adding a '/n' to indicate
the n-th patch set should be downloaded instead of the default of
patch set 1.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way users are well aware of which account we used when the
uploads are complete, so they can be certain to sign into the web
application with that user identity.
Signed-off-by: Shawn O. Pearce <sop@google.com>
