urllib2 is not thread safe and may be causing sync to lock up or
not work correctly on various platforms. Instead use the command
line curl program.
Change-Id: I36eaf18bb4df089d26ea99d533cb015e7c616eb0
The Content-Length when resuming is the number of bytes that
remain in the file. To compute the total size as expected by
the progress meter, we must add the bytes already stored.
While we are in this method fix uses of % operator to ensure
a tuple is always supplied.
Change-Id: Ic899231b5bc0ab43b3ddb1d29845f6390e820115
The fix for issue #46 in 5d016502eb appears to break syncing in some
situations: the branch is deleted after the point where it's been
configured, which deletes part of its configuration and causes the
config to change each time you call `repo init`, alternating between a
configuration that works and one that doesn't.
Instead of deleting the branch with git branch -D, use git update-ref -d
which just deletes the ref (to avoid the rebase) without touching the
configuration for the branch that was set up during the first repo init.
This appears to ensure the config is left in a valid state all the time
no matter what combination of repo init commands you run, without
reintroducing the rebasing issue.
Change-Id: Iaadaa6e56a46840bbc593fa5b35cb5b34cd3ce69
This patch fixes repo behaviour when running sync -d with unmodified
topic branches.
Prior to this patch sync -d would see the latest revision is already
checked out, thus staying on the branch. Since "-d" means detach we
should follow git's behaviour and actually detach from the branch in
that case.
Basic test case - after a fresh repo init + sync -
* repo start --all testdetach
* repo sync -d
* repo status
-> status shows active topic branch "testdetach",
should show :
nothing to commit (working directory clean)
Change-Id: Ic1351e6b5721b76557a51ab09f9dd42c38a4b415
See repo issue #46 :
https://code.google.com/p/git-repo/issues/detail?id=46
When using repo init -b on an already existing repository,
the next sync will try to rebase changes coming from the old manifest
branch onto the new, leading in the best case scenario to conflicts
and in the worst case scenario to an incorrect "mixed up" manifest.
This patch fixes this by deleting the "default" branch in the local
manifest repository when the -d init switch is used, thus forcing
repo to perform a fresh checkout of the new manifest branch
Change-Id: I379e4875ec5357d8614d1197b6afbe58f9606751
Mirror manifest and repo projects are outside the manifest and
have no groups. Allow project groups to be None for these
projects.
Change-Id: I3e1c4add894fe1c43aa4e77a1fc1558aa10dd191
Allows to ff-only a gerrit patch
This patch is necessary to automatically ensure that the patch will
be correctly submitted on ff-only gerrit projects
You can now use:
repo download (--ff-only|-f) project changeid/patchnumber
This is useful to automate verification of fast forward status of a patch
in the context of build automation, and commit gating (e.g. buildbot)
Change-Id: I403a667557a105411a633e62c8eec23d93724b43
Signed-off-by: Erwan Mahe <erwan.mahe@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
BZ: 4779
Allows to revert a gerrit patch
This patch is necessary for the on-demand creation of
engineering builds using buildbot
You can now use:
repo download [--revert|-r project changeid/patchnumber
This is useful to automate reverting of a patch
in the context of build automation, and regression bisection
Change-Id: I3985e80e4b2a230f83526191ea1379765a54bdcf
Signed-off-by: Erwan Mahe <erwan.mahe@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
default option uses git checkout, and thus overwrite the previous
checkouts. this is a problem for automated builds of several
changesets in the same project for daily builds of pending submission
You can now use:
repo download [--cherry-pick|-c] project changeid/patchnumber
This will parse the manifest, cd to the corresponding project
download the changes to FETCH_HEAD and cherry-pick the result.
This is useful to automate cherry-picking of a patch
in the context of build automation, and commit gating (e.g. buildbot)
Change-Id: Ib638afd87677f1be197afb7b0f73c70fb98909fe
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
There are use-cases when fetching all branch is impractical and
we really need to fetch only one branch/tag.
e.g. there is a large project with binaries and every update of a
binary file is put to a separate branch.
The whole project history might be too large to allow users fetch it.
Add 'sync-c' option to 'project' and 'default' tags to make it possible
to configure 'sync-c' behavior at per-project and per-manifest level.
Note that currently there is no possibility to revert boolean flag from
command line. If 'sync-c' is set in manifest then you cannot make
full fetch by providing a repo tool argument.
Change-Id: Ie36fe5737304930493740370239403986590f593
Every project is in group "default". "-default" does not remove
it from this project. All group names specified in the manifest
are positive names as opposed to a mix of negative and positive.
Specified groups are resolved in order. If init is supplied with
--groups="group1,-group2", the following describes the project
selection when syncing:
* all projects in "group1" will be added, and
* all projects in "group2" will be removed.
Change-Id: I1df3dcdb64bbd4cd80d675f9b2d3becbf721f661
Allow the optional addition of "annotation" nodes nested under
projects. Each annotation node must have "name" and "value"
attributes. These name/value pairs will be exported into the
environment during any forall command, prefixed with "REPO__"
In addition, an optional "keep" attribute with case insensitive "true"
or "false" values can be included to determine whether the annotation
will be exported with 'repo manifest'
Change-Id: Icd7540afaae02c958f769ce3d25661aa721a9de8
Signed-off-by: James W. Mills <jameswmills@gmail.com>
Previously repo had incorrect code that did not really check
if sha1 presents in a project. It worked for tags though.
Check if a revision (either tag or sha1) is present by using
'git rev_parse' functionality.
Change-Id: I1787f3348573948573948753987394839487572b
Allows specifying a list of groups with a -g argument to repo init.
The groups act on a group= attribute specified on projects in the
manifest.
All projects are implicitly labelled with "default" unless they are
explicitly labelled "-default".
Prefixing a group with "-" removes matching projects from the list
of projects to sync.
If any non-inverted manifest groups are specified, the default label
is ignored.
Change-Id: I3a0dd7a93a8a1756205de1d03eee8c00906af0e5
Reviewed-on: https://gerrit-review.googlesource.com/34570
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
The -u option causes 'repo diff' to generate diff output
with file paths relative to the repository root,
so the output can be applied to the Unix 'patch' command.
The name '-u' was selected for convenience, because
both 'diff' and 'git diff' accept the option with the same name
to generate an 'unified diff' output suitable for 'patch' command.
Change-Id: I79c8356db4ed20ecaccc258b3ba139db76666fe0
Reviewed-on: https://gerrit-review.googlesource.com/34380
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
401: Unauthorized, authentication may be required. This is usually
handled internally by the HTTP client in Python. If it reaches
our code in repo, the Python HTTP client didn't find a password
in ~/.netrc that it could use.
403: Authentication was supplied, but is incorrect. It might be
that the CDN doesn't want to offer this clone.bundle file
to the client, but the Git fetch operation would still be
successful. This might arise if branch level read controls
were used in Gerrit Code Review and the /clone.bundle file
contained branches not visible to the client.
404: The server has no /clone.bundle file available.
In all of these cases, sliently ignore the /clone.bundle file HTTP
error and let the Git operation take over.
Change-Id: I1787f3cac86b017351952bbb793fe5874d83c72b
In case of manifest/smart sync repo changes ".merge" config
option from branch to SHA. Doing 'repo upload' fails as
repo tries to upload to a remote branch that looks like SHA
(e.g. refs/for/23423423423423423423423)
Do not update the .merge in case if revision is SHA.
Change-Id: I9139708fa17f21eec5a7e23c3255333626bf529e
Clients might be using their own special git-remote-* helper that
has a hypen in its name. Permit - in the scheme part of the URL
when trying to decide if it is an SSH URL and assume it is *not*
SSH if the URL matches "foo-bar://" style.
Change-Id: I7ba2d810a614f6e605a441d5972902c4a14e73fd
repo status just prints "# on branch oprofile" if you have branched
in clean status. This doesn't really tell which branch is meant.
Instead we can use the same syntax with modified gits which will
give us detailed information.
Change-Id: I55fe5154d278e10a814281dd2ba501ec6e956730
This new attribute can prevent 'repo sync' from automatically rebasing.
I hit a situation in where one of the git repositories I was tracking
was actually an external repository that I wanted to pull commits
into and merge myself. (NOT rebase, since that would lose the merge
history.) In this case, I'm not using 'repo upload', I'm manually
managing the merges to and from this repository.
Everything was going great until I typed 'repo sync' and it rebased
my manually-merged tree. Hence the option to skip it.
Change-Id: I965e0dd1acb87f4a56752ebedc7e2de1c502dbf8
Occassionally, the content-length may be missing when using urlib
in python 2.6 and 2.7. This change assumes the value of the header is
0 if it doesn't exist
Change-Id: Iaf1c8a796bc667823d4d7c30f9b617644b271d00
If SSH is not available, Gerrit returns NOT_AVAILABLE to the /ssh_info
query made by repo upload. In this case fallback to the /p/$PROJECT URL
that Gerrit also exports and use that for uploads.
Change-Id: I1e3e39ab709ecc0a692614a41a42446426f39c08
There is also shortcuts in case if the "current branch" is
a persistent revision such as tag or sha1. We check if the
persistent revision is present locally and if it does - do
no fetch anything from the server.
This greately reduces sync time and size of the on-disk repo
Change-Id: I23c6d95185474ed6e1a03c836a47f489953b99be
If the remote is using authenticated HTTP, but does not have
$GIT_URL/clone.bundle files in each repository, an initial sync
would fail around 8 projects in due to the library not resetting
the number of failures after getting a 404.
Work around this by updating the retry counter ourselves.
The urllib2 library is also not thread-safe. Make it somewhat
safer by wrapping the critical section with a lock.
Change-Id: I886e2750ef4793cbe2150c3b5396eb9f10974f7f
Signed-off-by: Shawn O. Pearce <sop@google.com>
Not every version of urllib2 supplies a reason object on the
HTTPError exception that it throws from urlopen(). Work around
this by using str(e) instead and hope the string formatting includes
sufficient information.
Change-Id: I0f4586dba0aa7152691b2371627c951f91fdfc8d
Signed-off-by: Shawn O. Pearce <sop@google.com>
urllib2 returns a malformed HTTPError object in certain situations.
For example, urllib2 has a couple of places where it creates an
HTTPError object with no fp:
if self.retried > 5:
# retry sending the username:password 5 times before failing.
raise HTTPError(req.get_full_url(), 401, "basic auth failed",
headers, None)
When it does that, HTTPError's ctor doesn't call through to
addinfourl's ctor:
# The addinfourl classes depend on fp being a valid file
# object. In some cases, the HTTPError may not have a valid
# file object. If this happens, the simplest workaround is to
# not initialize the base classes.
if fp is not None:
self.__super_init(fp, hdrs, url, code)
Which means the 'headers' slot in addinfourl is not initialized and
info() fails. It is completely insane that urllib2 decides not to
initialize its own base class sometimes.
Change-Id: I32a0d738f71bdd7d38d86078b71d9001e26f1ec3
Signed-off-by: Shawn O. Pearce <sop@google.com>
After a $GIT_URL/clone.bundle has been applied to the new local
repository, perform an incremental fetch using `git fetch` to ensure
the local repository is up-to-date. This allows the hosting server
to offer stale /clone.bundle files to bootstrap a new client.
If a single git fetch fails, it may succeed again after a short
delay. Transient failures are typical in environments where the
remote Git server happens to have limits on how many requests it
can serve at once (the anonymous git daemon, or an HTTP server).
Wait a randomized delay between 30 and 45 seconds and retry the
failed project once. This delay gives the site time to recover
from a transient traffic spike, and the randomization makes it less
likely that a spike occurs again from all of the same clients.
Change-Id: I97fb0fcb33630fb78ac1a21d1a4a3e2268ab60c0
Signed-off-by: Shawn O. Pearce <sop@google.com>
An HTTP (or HTTPS) based remote server may now offer a 'clone.bundle'
file in each repository's Git directory. Over an http:// or https://
remote repo will first ask for '$URL/clone.bundle', and if present
download this to bootstrap the local client, rather than relying
on the native Git transport to initialize the new repository.
Bundles may be hosted elsewhere. The client automatically follows a
HTTP 302 redirect to acquire the bundle file. This allows servers
to direct clients to cached copies residing on content delivery
networks, where the bundle may be closer to the end-user.
Bundle downloads are resumeable from where they last left off,
allowing clients to initialize large repositories even when the
connection gets interrupted.
If a bundle does not exist for a repository (a HTTP 404 response
code is returned for '$URL/clone.bundle'), the native Git transport
is used instead. If the client is performing a shallow sync, the
bundle transport is not used, as there is no way to embed shallow
data into the bundle.
Change-Id: I05dad17792fd6fd20635a0f71589566e557cc743
Signed-off-by: Shawn O. Pearce <sop@google.com>
The manifest project has - by design - not a review URL associated
with it. It is actually not even a 'project' in repo's sense.
This will prevent the commit-msg hook from being added, which is
not necessarily wanted as the project is managed in gerrit.
This commit will enable the commit-msg hook, which in turn will
add the Change-Id-line to every new commit in it. This simplifies
replacing patch sets (by git push ... refs/for/...).
Change-Id: I42d0f6fd79e6282d9d47074a3819e68d968999a7
Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
This commit adds a --br=<branch> option to repo upload.
repo currently examines every non-published branch. This is problematic
for my workflow. I have many branches in my kernel tree. Many of these
branches are based off of upstream remotes (I have many remotes) and
will never be uploaded (they'll get sent upstream as a patch).
Having repo scan these branches adds to my upload processing time
and clutters the branch selection buffer. I've also seen repo get
confused when one of my branches is 1000s of commits different from
m/master.
Change-Id: I68fa18951ea59ba373277b57ffcaf8cddd7e7a40
In the current version of repo checkout, we often get the error:
error: no project has branch xyzzy
...even when the actual error was something else. This fixes it
to only report the 'no project has branch' when that is actually true.
This fix is very similar to one made for 'repo abandon':
https://review.source.android.com/#change,22207
The repo checkout error is filed as: <http://crosbug.com/6514>
TEST=manual
A sample creating a case where 'git checkout' will fail:
$ repo start branch1 .
$ repo start branch2 .
$ touch bogusfile
$ git add bogusfile
$ git commit -m "create bogus file"
[branch2 f8b6b08] create bogus file
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 bogusfile
$ echo "More" >> bogusfile
$ repo checkout branch1 .
error: chromite/: cannot checkout branch1
A sample case showing that we still fail if no project has a branch:
$ repo checkout xyzzy .
error: no project has branch xyzzy
Change-Id: I48a8e258fa7a9c1f2800dafc683787204bbfcc63
The main fix is to give an error message if nothing was actually
abandoned. See <http://crosbug.com/6041>.
The secondary fix is to list projects where the abandon happened.
This could be done in a separate CL or dropped altogether if requested.
TEST=manual
$ repo abandon dougabc; echo $?
Abandon dougabc: 100% (127/127), done.
Abandoned in 2 project(s):
chromite
src/platform/init
0
$ repo abandon dougabc; echo $?
Abandon dougabc: 100% (127/127), done.
error: no project has branch dougabc
1
$ repo abandon dougabc; echo $?
Abandon dougabc: 100% (127/127), done.
error: chromite/: cannot abandon dougabc
1
Change-Id: I79520cc3279291acadc1a24ca34a761e9de04ed4
If git-rerere is enabled, it uses the rr-cache directory that
repo currently creates a symlink from, but doesn't create the
destination directory (inside the project's directory). Git
will then complain during merges and rebases.
This commit creates the rr-cache directory inside the project.
Change-Id: If8b57a04f022fc6ed6a7007d05aa2e876e6611ee
All repo-level hooks are expected to live in a single project at the
top level of that project. The name of the hooks project is provided
in the manifest.xml. The manifest also lists which hooks are enabled
to make it obvious if a file somehow failed to sync down (or got
deleted).
Before running any hook, we will prompt the user to make sure that it
is OK. A user can deny running the hook, allow once, or allow
"forever" (until hooks change). This tries to keep with the git
spirit of not automatically running anything on the user's computer
that got synced down. Note that individual repo commands can add
always options to avoid these prompts as they see fit (see below for
the 'upload' options).
When hooks are run, they are loaded into the current interpreter (the
one running repo) and their main() function is run. This mechanism is
used (instead of using subprocess) to make it easier to expand to a
richer hook interface in the future. During loading, the
interpreter's sys.path is updated to contain the directory containing
the hooks so that hooks can be split into multiple files.
The upload command has two options that control hook behavior:
- no-verify=False, verify=False (DEFAULT):
If stdout is a tty, can prompt about running upload hooks if needed.
If user denies running hooks, the upload is cancelled. If stdout is
not a tty and we would need to prompt about upload hooks, upload is
cancelled.
- no-verify=False, verify=True:
Always run upload hooks with no prompt.
- no-verify=True, verify=False:
Never run upload hooks, but upload anyway (AKA bypass hooks).
- no-verify=True, verify=True:
Invalid
Sample bit of manifest.xml code for enabling hooks (assumes you have a
project named 'hooks' where hooks are stored):
<repo-hooks in-project="hooks" enabled-list="pre-upload" />
Sample main() function in pre-upload.py in hooks directory:
def main(project_list, **kwargs):
print ('These projects will be uploaded: %s' %
', '.join(project_list))
print ('I am being a good boy and ignoring anything in kwargs\n'
'that I don\'t understand.')
print 'I fail 50% of the time. How flaky.'
if random.random() <= .5:
raise Exception('Pre-upload hook failed. Have a nice day.')
Change-Id: I5cefa2cd5865c72589263cf8e2f152a43c122f70
Fix for the bug that leaves a fractional .git directory after attempting to
perform an initial sync to a nonexistent revision. Moved the initialization of
the working directory to after the revision ID has already been checked. Now,
no project/.git directory gets created at all if the revision ID is bad.
Change-Id: I0c9b2a59573410f1d11de7661591bf02e4ce326b
This renaming was done for two reasons:
1. The hooks are actually project-level hooks, not repo-level
hooks. Since we are talking about adding repo-level hooks,
It keeps things less confusing if we name the existing hooks
to be "ProjectHooks"
2. The function is a private function in project.py and so
should have capitalization to match.
I also added a docstring describing this function.
Change-Id: I1d30f5de08e8f9f99f78146e68c76f906782d97e
There was a minor typo that would cause repo to (I believe)
mistakenly identify any file that contained a substring of the
word 'commit-msg' as a commit message hook. For example, the file
'mit' or the file 'msg' would be treated as a commit message hook.
I believe that it was intended that repo only recognize files
named exactly 'commit-msg'.
Change-Id: I93edbddf3da3cf0935641e6efb19b0a8ee6e2308
Commit "Make path references OS independent" (df14a70c45)
broke mirror clients by trying to invoke replace() on None
when there is no worktree.
Change-Id: Ie0a187058358f7dcdf83119e45cc65409c980f11
It hasn't been necessary for a long time, and its
functionality can be accomplished with 'git push'.
Change-Id: Ic00d3adbe4cee7be3955117489c69d6e90106559
Use git clone to initialize a new repository, and when possible
allow callers to use --reference to reuse an existing checkout as
the initial object storage area for the new checkout.
Change-Id: Ie27f760247f311ce484c6d3e85a90d94da2febfc
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the -t flag is given to upload, the local branch name is
automatically sent to Gerrit Code Review as the topic branch name
for the change(s). This requires the server to be Gerrit Code
Review v2.1.3-53-gd50c94e or later, which isn't widely deployed
right now, so the default is opt-out.
Change-Id: I034fcacb405b7cb909147152db427fe69dd7bcbf
Signed-off-by: Shawn O. Pearce <sop@google.com>
If a tagged commit is not reachable by the fetch refspec configured
for the git (usually refs/heads/*) it will not be downloaded by
'git fetch'. The tag can however be downloaded with 'git fetch
--tags' or 'git fetch tag <tag>'.
This patch fixes the situation when a tag is not found after a
'git fetch'. Repo will issue 'git fetch tag <tag>' before giving
up completely.
Change-Id: I87796a5e1d51fcf398f346a274b7a069df37599a
Signed-off-by: Shawn O. Pearce <sop@google.com>
Most users of repo are also using Gerrit Code Review, and will want
the commit-msg hook to be automatically installed into their local
projects so that Change-Ids are assigned when commits are created,
not when they are first uploaded.
(cherry picked from commit a949fa5d20
but squashed with latest hook script from version 2.1.2)
Change-Id: Ie68b2d60ac85d8c2285d2e1e6a4536eb76695547
Signed-off-by: Shawn O. Pearce <sop@google.com>
This is almost always something the user needs to address
before continuing work, so promoting it to a failure (rather
than simply an informational message) seems the right way to
go. As a side-effect, repo will now exit with a non-zero
status code in this situation, so pipelines of the form
`repo sync && make` will fail if there are branches that
are stalled due to uploaded but unmerged patches.
If a local git repository exists within the same folder as a new project that
is added, when the user syncs the repo, the sync will overwrite the local
files under the project's .git repository with its own symlinks. Make sure
that we do not overwrite 'normal' files in repo and throw an error when
that happens.
If an email address in a commit object contains a space, like a few
malformed ones on the Linux kernel, we still want to split only on
the first space.
Unfortunately my brain was too damaged by Perl and originally wrote
the split asking for 2 results; in Python split's argument is how
many splits to perform. Here we want only 1 split, to break apart
the commit identity from the email address on the same line.
Signed-off-by: Shawn O. Pearce <sop@google.com>
We accidentally introduced this message during 1.6.8 by always
invoking `git rebase` when there were no new commits from the
upstream, but the user had local commits.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The revisionExpr field now holds an expression from the manifest,
such as "refs/heads/master", while revisionId holds the current
commit-ish SHA-1 of the revisionExpr. Currently that is only
filled in if the manifest points directly to a SHA-1.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The trick of looking at the reflog for the remote tracking branch
and only going back one commit works some of the time, but not all of
the time. Its sort of relying on the fact that the user didn't use
`repo sync -n` or `git fetch` to only update the tracking branches
and skip the working directory update.
Doing this right requires looking through the history of the SHA-1
source (what the upstream used to be) and finding a spot where the
DAG diveraged away suddenly, and consider that to be the rewind
point. That's really difficult to do, as we don't have a clear
picture of what that old point was.
A close approximation is to list all of the commits that are in
HEAD, but not the new upstream, and rebase all of those where the
committer email address is this user's email address. In most cases,
this will effectively rebase only the user's new original work.
If the user is the project maintainer and rewound the branch
themselves, and they don't want all of the commits they have created
to be rebased onto the new upstream, they should handle the rebase
on their own, after the sync is complete.
Signed-off-by: Shawn O. Pearce <sop@google.com>
We now feed Project a RemoteSpec, instead of the Remote directly
from the XmlManifest. This way the RemoteSpec already has the
full project URL, rather than just the base, permitting other
types of manifests to produce the URL in their own style.
Signed-off-by: Shawn O. Pearce <sop@google.com>
These aren't that widely used, and actually make it difficult for
users to fully mirror a forest of repositories, and then permit
someone else to clone off that forest, rather then the original
upstream servers.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Performance improvements in repo sync caused us to skip out of the
initial Sync_LocalHalf without ever running CopyFiles, so we didn't
create the top level Makefile in new clients whose manifest request
one with a <copyfile> element.
Now we run CopyFiles after the initial read-tree that populates
the project working directory.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the current branch is published, but all published commits are
merged into the manifest revision, but there is also at least one
unpublished commit on the current branch, we should rebase the
unpublished commit, rather than creating a merge commit.
Signed-off-by: Shawn O. Pearce <sop@google.com>
By creating a background ssh "control master" process which lives
for the duration of our sync cycle we can easily cut the time for
a no-op sync of 132 projects from 60s to 18s.
Bug: REPO-11
Signed-off-by: Shawn O. Pearce <sop@google.com>
Most projects will have their branch heads matching in all branches,
so switching between them should be just a matter of updating the
work tree's HEAD symref. This can be done in pure Python, saving
quite a bit of time over forking 'git checkout'.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This is mostly useful if the number of projects to switch is many
(e.g. all of Android) and a large number of them are behind the
current manifest revision. We wind up needing to run git just to
make the working tree match, and that often makes the command take
a couple of seconds longer than we'd like.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Its quite common for most projects to be matching the current
manifest revision, as most developers only modify one or two projects
at any one time. We can speed up `repo start foo` (that impacts
the entire client) by performing most of the branch creation and
switch operations in pure Python, and thus avoid 4 forks per project.
Signed-off-by: Shawn O. Pearce <sop@google.com>
These used to be used back when we had Gerrit 1.x support and used
HTTP based uploads to transmit changes for review. Since we moved
entirely to Gerrit 2.x, these are no longer called.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Its unlikely that a new version of repo will be delivered in any
given day, so we now check only once every 24 hours to see if repo
has been updated. This reduces the sync cost, as we no longer need
to contact the repo distribution servers every time we do a sync.
repo selfupdate can still be used to force a check.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Usually repo is upgraded only once a week, if that often. Most of
the time we invoke HasChanges on the repo project (or even on the
manifest project) the current HEAD will resolve to the same SHA-1
as the remote tracking ref, and there are therefore no changes.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the m/BRANCH ref is already pointing at the value set in the
manifest there is no reason to set it again. Leave it alone,
thus saving a full fork+exec call.
Signed-off-by: Shawn O. Pearce <sop@google.com>
By resolving the current HEAD and the manifest revision using pure
Python, we can in the common case of "no changes" avoid a lot of
git operations and directly jump out of the local sync method.
This reduces the no-op `repo sync -l` time for Android's 114 projects
from more than 6s to under 0.8s.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Its much faster to read the refs from 114 projects when the reader
is pure Python and just doing file IO than forking 114 git commands
and parsing their output.
The reader caches refs based upon file mtimes. If any single ref
file has been modified since the last read, we re-read the entire
repository's ref namespace. This simplifies the code as we don't
need to worry about shooting down symbolic-refs, but it may cause
more IO than is necessary if only one ref gets updated.
This change drops `repo branches` in Android from 1.658s to 0.206s.
Likewise, `repo sync` improves dramatically as well.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This is invoked once per project in `repo sync`. Taking it out
saves about 1/114 of a second, so on a large set of projects like
Android it can save up to a full second of sync time.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Someone pointed out this message isn't always the truth; so we
shouldn't print it. The code path is executed when there are
published commits, yet our output talks about unpublished ones.
Signed-off-by: Shawn O. Pearce <sop@google.com>
We now try to sync all projects that can be done safely first, before
we start rebasing user commits over the upstream. This has the nice
effect of making the local tree as close to the upstream as possible
before the user has to start resolving merge conflicts, as that extra
information in other projects may aid in the conflict resolution.
Informational output is buffered and delayed until calculation for
all projects has been done, so that the user gets one concise list
of notice messages, rather than it interrupting the progress meter.
Fast-forward output is now prefixed with the project header, so the
user can see which project that update is taking place in, and make
some relation of the diffstat back to the project name.
Rebase output is now prefixed with the project header, so that if
the rebase fails, the user can see which project we were operating
on and can try to address the failure themselves.
Since rebase sits on a detached HEAD, we now look for an in-progress
rebase during sync, so we can alert the user that the given project
is in a state we cannot handle.
Signed-off-by: Shawn O. Pearce <sop@google.com>
rebase interactive (aka rebase -i) has changed in newer versions
of git, and doesn't always generate the sequence of commits the
same way it used to. It also doesn't handle having a previously
applied commit try to be applied again.
The default rebase algorithm is better suited to our needs.
It uses --ignore-if-in-upstream when generating the patch series
for git-am, and git-am with its 3-way fallback is able to handle
a rename case just as well as the cherry-pick variant used by -m.
Its also a generally faster implementation.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If there is nothing output at all, tell the user the working tree is
completely clean. It just gives them a bit more of a warm-fuzzy
feeling knowing repo and until the end. It also more closely
matches with the output of git status.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The -d flag moves the project back to a detached HEAD state,
matching what is listed in the manifest. This can be useful to
set a client to something stable (or at least well-known), such as
before a sequence of 'repo download' commands are used to get some
changes for testing.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If `repo start foo` fails due to uncommitted and unmergeable changes
in a single project, we have switched half of the projects over to
the new target branches, but didn't on the one that failed to move.
This change improves the situation by doing three things differently:
- We keep going when we encounter an error, so other projects
that can successfully switch still switch.
- We ignore projects whose current branch is already on the
requested name; they are logically already setup.
- We checkout the branch if it already exists, rather than
trying to recreate the branch.
Bug: REPO-22
Signed-off-by: Shawn O. Pearce <sop@google.com>
There isn't any great value in buffering stdout into memory
coming from git checkout. So don't bother doing it.
Signed-off-by: Shawn O. Pearce <sop@google.com>
We now display a summary of the available topic branches in this
client, based upon a sorted union of all existing projects.
Bug: REPO-21
Signed-off-by: Shawn O. Pearce <sop@google.com>
This can be useful to create a new manifest from an existing client,
especially if the client wants to use the "-r" option to set each
project's revision to the current commit SHA-1, making a sort of a
tag file that can be used to recreate this exact state elsewhere.
Signed-off-by: Shawn O. Pearce <sop@google.com>
When creating a mirror repository we will always be using a bare
repository. Setting $GIT_DIR/config to have core.bare = true is
reasonable and helps Git to recognize the environment it is in.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Months ago when the Android Open Source Project launched we had some
import errors that had to be fixed and worked over. These hacks
were here to help users update their clients to newer versions of
the imported code.
Its very likely all clients have either been deleted, or have been
updated and have the fixed imports. So we don't need this hack in
repo anymore.
If a very ancient client still existed, it would need to be created
from scratch anyway, due to the Android cupcake branch merging
into master and the manifest changes not being able to be handled
correctly by repo. A new client wouldn't have the incorrectly
imported code in it, and thus wouldn't need this hack.
Signed-off-by: Shawn O. Pearce <sop@google.com>
I missed a parameter in the format string, but still provided the
value in the parameter list, so the format failed to produce an
output message.
Bug: REPO-15
Signed-off-by: Shawn O. Pearce <sop@google.com>
If a manifest specifies an invalid revision property, give the
user a better error message detaling the problem, instead of an
ugly Python traceback with a strange Git error message.
Bug: REPO-2
Signed-off-by: Shawn O. Pearce <sop@google.com>
Prior to git 1.6.1-rc3~5 the output of 'git branch -d' matched:
Deleted branch (.*)\.
where the subgroup grabbed the branch name. In v1.6.1-rc3~5 (aka
a126ed0a01e265d7f3b2972a34e85636e12e6d34) Brandon Casey changed
the output to include the SHA-1 of the branch name, now matching
the pattern:
Deleted branch (.*) \([0-9a-f]*\)\.
Instead of parsing the output of git branch we now re-obtain the
list of branches after the deletion attempt and perform a set
difference in memory to determine which branches we were able to
successfully delete.
Bug: REPO-9
Signed-off-by: Shawn O. Pearce <sop@google.com>
Now that Gerrit2 has been released and the only supported upload
protocol is direct git push over SSH we no longer need the large
and complex protobuf client library, or the upload chunking logic
in gerrit_upload.py.
Signed-off-by: Shawn O. Pearce <sop@google.com>
In Gerrit2 uploads are sent over "git push ssh://...", as this
is a more efficient transport and is easier to code from external
scripts and/or direct command line usage by an end-user.
Gerrit1's HTTP POST based format is assumed if the review server
does not have the /ssh_info URL available on it.
Signed-off-by: Shawn O. Pearce <sop@google.com>
There's an extra "," at the end of the line, which is causing
trouble when the manifest file specifies a revision for a
project. Since the default manifest file doesn't specify
revisions for the projects, the problem has gone unnoticed.
Thanks to Barry Silverman <barry@disus.com> for spotting the
issue and providing a patch.
Signed-off-by: Marcelo E. Magallon <marcelo.magallon@gmail.com>
Users are prompted with the list of known changes we are about
to upload, and they can fill out the current change numbers for
any changes which already exist in the data store. For each of
those changes the change number and commit id is sent as part of
the upload request, so Gerrit can insert the new commit as a new
patch set of the existing change, rather than make a new change.
This facility permits developers to replace a patch so they can
address comments made on a prior version of the same change.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way project.GetUploadableBranch(project.CurrentBranch) can tell
us how (if at all) to upload the currently checked out branch.
Signed-off-by: Shawn O. Pearce <sop@google.com>
By setting a project-name on a remote nested within a project forks
of a project like the Linux kernel can be easily handled by fetching
all relevant forks into the same client side project under different
remote names. Developers can create branches off different remotes
using `git checkout --track -b $myname $remote/$branch` and later
`repo upload` automatically redirects to the proper fork project
in the code review server.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way "forks" of a project, e.g. the linux kernel, can be setup to
use different destination projects in the review server by creating
different remotes in the client side Git repository.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The mirror option downloads a complete forrest (as described by the
manifest) and creates a replica of the remote repositories rather
than a client working directory. This permits other clients to
sync off the mirror site.
A mirror can be positioned in a "DMZ", where the mirror executes
"repo sync" to obtain changes from the external upstream and
clients inside the protected zone operate off the mirror only,
and therefore do not require direct git:// access to the external
upstream repositories.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This destroys a local development branch, removing all history
of that branch from ever existing. If the branch is currently
checked out we move back to the upstream revision.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This hook is evaluated by `git gc --auto` to determine if it is a
good idea to execute a GC at this time, or defer it to some later
date. When working on a laptop its a good idea to avoid GC if you
are on battery power as the extra CPU and disk IO would consume a
decent amount of the charge.
The hook is the standard sample hook from git.git contrib/hooks,
last modified in git.git by 84ed4c5d117d72f02cc918e413b9861a9d2846d7.
I added the GPLv2 header to the script to ensure the license notice
is clear, as it does not match repo's own APLv2 license.
We only update hooks during initial repository creation or on
a repo sync. This way we don't incur huge overheads from the
hook stat operations during "repo status" or even the normal
"repo sync" cases.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Some users have noticed that repo doesn't work on VFAT, as we
require a POSIX filesystem with POSIX symlink support. Catching the
OSError during our symlink creation and raising a GitError with a
more descriptive message will help users to troubleshoot and fix
their own installation problems.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If we are already up-to-date we just want to display no output.
This means we have to avoid calling "git merge" if there aren't
commits to be merged into the working directory.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Instead of trying to rebase the changes on a topic branch that
has been fully merged into the upstream branch we track, we should
just fast-forward the topic branch to the new upstream revision.
This way the branch doesn't try to rewrite commits that are already
merged in the upstream.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Git may have been installed without its hooks directory, which
means we won't have any hooks in a repo created git repository.
Since we are just deleting the hooks it doesn't matter.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Downloading and streaming a tar into Git is slower than just
letting the native git:// protocol handle the data transfer,
especially when there are multiple revisions available and
Git can perform delta compression across revisions.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Hiding error messages from the remote peer is not a good idea,
as users should be made aware when the remote peer is not a
complete Git repository so they can alert the administrators
and have the repository corrected.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The initial open source release of the Android 1.0 platform had
some problems with its Perforce->Git imports. Google was forced
to rewrite some history to redirect users onto more stable upstream
sources and correct errors in the imports.
Not everyone has the correct android-1.0 tags, as some users did
manage to fetch the platform early, before the mirror sites crashed
and the history was rewritten.
This change is a band-aid to ensure any stale android-1.0 tags are
get updated to the corrected version. It should be backed out at
some point in the near future, when we can be fairly certain that
everyone has the correct android-1.0 tags.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Now `repo download . 1402` would download the change numbered 1402
into the current project and check it out for the user, using a
detached HEAD. `repo sync .` would back out of the change and
return to the upstream version.
Multiple projects can be fetched at once by listing them out on
the command line as different arguments.
Individual patch sets can be selected by adding a '/n' to indicate
the n-th patch set should be downloaded instead of the default of
patch set 1.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way users are well aware of which account we used when the
uploads are complete, so they can be certain to sign into the web
application with that user identity.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the reflog for the upstream branch has only 1 entry in it, as
the branch has been updated only once, we can get back the 0{40}
object id from `git rev-parse upstream@{1}`, in which case we should
consider it to be the same as if upstream@{1} is not defined.
Signed-off-by: Shawn O. Pearce <sop@google.com>
