By creating a background ssh "control master" process which lives
for the duration of our sync cycle we can easily cut the time for
a no-op sync of 132 projects from 60s to 18s.
Bug: REPO-11
Signed-off-by: Shawn O. Pearce <sop@google.com>
Most projects will have their branch heads matching in all branches,
so switching between them should be just a matter of updating the
work tree's HEAD symref. This can be done in pure Python, saving
quite a bit of time over forking 'git checkout'.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This is mostly useful if the number of projects to switch is many
(e.g. all of Android) and a large number of them are behind the
current manifest revision. We wind up needing to run git just to
make the working tree match, and that often makes the command take
a couple of seconds longer than we'd like.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Its quite common for most projects to be matching the current
manifest revision, as most developers only modify one or two projects
at any one time. We can speed up `repo start foo` (that impacts
the entire client) by performing most of the branch creation and
switch operations in pure Python, and thus avoid 4 forks per project.
Signed-off-by: Shawn O. Pearce <sop@google.com>
These used to be used back when we had Gerrit 1.x support and used
HTTP based uploads to transmit changes for review. Since we moved
entirely to Gerrit 2.x, these are no longer called.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Its unlikely that a new version of repo will be delivered in any
given day, so we now check only once every 24 hours to see if repo
has been updated. This reduces the sync cost, as we no longer need
to contact the repo distribution servers every time we do a sync.
repo selfupdate can still be used to force a check.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Usually repo is upgraded only once a week, if that often. Most of
the time we invoke HasChanges on the repo project (or even on the
manifest project) the current HEAD will resolve to the same SHA-1
as the remote tracking ref, and there are therefore no changes.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the m/BRANCH ref is already pointing at the value set in the
manifest there is no reason to set it again. Leave it alone,
thus saving a full fork+exec call.
Signed-off-by: Shawn O. Pearce <sop@google.com>
By resolving the current HEAD and the manifest revision using pure
Python, we can in the common case of "no changes" avoid a lot of
git operations and directly jump out of the local sync method.
This reduces the no-op `repo sync -l` time for Android's 114 projects
from more than 6s to under 0.8s.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Its much faster to read the refs from 114 projects when the reader
is pure Python and just doing file IO than forking 114 git commands
and parsing their output.
The reader caches refs based upon file mtimes. If any single ref
file has been modified since the last read, we re-read the entire
repository's ref namespace. This simplifies the code as we don't
need to worry about shooting down symbolic-refs, but it may cause
more IO than is necessary if only one ref gets updated.
This change drops `repo branches` in Android from 1.658s to 0.206s.
Likewise, `repo sync` improves dramatically as well.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This is invoked once per project in `repo sync`. Taking it out
saves about 1/114 of a second, so on a large set of projects like
Android it can save up to a full second of sync time.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Someone pointed out this message isn't always the truth; so we
shouldn't print it. The code path is executed when there are
published commits, yet our output talks about unpublished ones.
Signed-off-by: Shawn O. Pearce <sop@google.com>
We now try to sync all projects that can be done safely first, before
we start rebasing user commits over the upstream. This has the nice
effect of making the local tree as close to the upstream as possible
before the user has to start resolving merge conflicts, as that extra
information in other projects may aid in the conflict resolution.
Informational output is buffered and delayed until calculation for
all projects has been done, so that the user gets one concise list
of notice messages, rather than it interrupting the progress meter.
Fast-forward output is now prefixed with the project header, so the
user can see which project that update is taking place in, and make
some relation of the diffstat back to the project name.
Rebase output is now prefixed with the project header, so that if
the rebase fails, the user can see which project we were operating
on and can try to address the failure themselves.
Since rebase sits on a detached HEAD, we now look for an in-progress
rebase during sync, so we can alert the user that the given project
is in a state we cannot handle.
Signed-off-by: Shawn O. Pearce <sop@google.com>
rebase interactive (aka rebase -i) has changed in newer versions
of git, and doesn't always generate the sequence of commits the
same way it used to. It also doesn't handle having a previously
applied commit try to be applied again.
The default rebase algorithm is better suited to our needs.
It uses --ignore-if-in-upstream when generating the patch series
for git-am, and git-am with its 3-way fallback is able to handle
a rename case just as well as the cherry-pick variant used by -m.
Its also a generally faster implementation.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If there is nothing output at all, tell the user the working tree is
completely clean. It just gives them a bit more of a warm-fuzzy
feeling knowing repo and until the end. It also more closely
matches with the output of git status.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The -d flag moves the project back to a detached HEAD state,
matching what is listed in the manifest. This can be useful to
set a client to something stable (or at least well-known), such as
before a sequence of 'repo download' commands are used to get some
changes for testing.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If `repo start foo` fails due to uncommitted and unmergeable changes
in a single project, we have switched half of the projects over to
the new target branches, but didn't on the one that failed to move.
This change improves the situation by doing three things differently:
- We keep going when we encounter an error, so other projects
that can successfully switch still switch.
- We ignore projects whose current branch is already on the
requested name; they are logically already setup.
- We checkout the branch if it already exists, rather than
trying to recreate the branch.
Bug: REPO-22
Signed-off-by: Shawn O. Pearce <sop@google.com>
There isn't any great value in buffering stdout into memory
coming from git checkout. So don't bother doing it.
Signed-off-by: Shawn O. Pearce <sop@google.com>
We now display a summary of the available topic branches in this
client, based upon a sorted union of all existing projects.
Bug: REPO-21
Signed-off-by: Shawn O. Pearce <sop@google.com>
This can be useful to create a new manifest from an existing client,
especially if the client wants to use the "-r" option to set each
project's revision to the current commit SHA-1, making a sort of a
tag file that can be used to recreate this exact state elsewhere.
Signed-off-by: Shawn O. Pearce <sop@google.com>
When creating a mirror repository we will always be using a bare
repository. Setting $GIT_DIR/config to have core.bare = true is
reasonable and helps Git to recognize the environment it is in.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Months ago when the Android Open Source Project launched we had some
import errors that had to be fixed and worked over. These hacks
were here to help users update their clients to newer versions of
the imported code.
Its very likely all clients have either been deleted, or have been
updated and have the fixed imports. So we don't need this hack in
repo anymore.
If a very ancient client still existed, it would need to be created
from scratch anyway, due to the Android cupcake branch merging
into master and the manifest changes not being able to be handled
correctly by repo. A new client wouldn't have the incorrectly
imported code in it, and thus wouldn't need this hack.
Signed-off-by: Shawn O. Pearce <sop@google.com>
I missed a parameter in the format string, but still provided the
value in the parameter list, so the format failed to produce an
output message.
Bug: REPO-15
Signed-off-by: Shawn O. Pearce <sop@google.com>
If a manifest specifies an invalid revision property, give the
user a better error message detaling the problem, instead of an
ugly Python traceback with a strange Git error message.
Bug: REPO-2
Signed-off-by: Shawn O. Pearce <sop@google.com>
Prior to git 1.6.1-rc3~5 the output of 'git branch -d' matched:
Deleted branch (.*)\.
where the subgroup grabbed the branch name. In v1.6.1-rc3~5 (aka
a126ed0a01e265d7f3b2972a34e85636e12e6d34) Brandon Casey changed
the output to include the SHA-1 of the branch name, now matching
the pattern:
Deleted branch (.*) \([0-9a-f]*\)\.
Instead of parsing the output of git branch we now re-obtain the
list of branches after the deletion attempt and perform a set
difference in memory to determine which branches we were able to
successfully delete.
Bug: REPO-9
Signed-off-by: Shawn O. Pearce <sop@google.com>
Now that Gerrit2 has been released and the only supported upload
protocol is direct git push over SSH we no longer need the large
and complex protobuf client library, or the upload chunking logic
in gerrit_upload.py.
Signed-off-by: Shawn O. Pearce <sop@google.com>
In Gerrit2 uploads are sent over "git push ssh://...", as this
is a more efficient transport and is easier to code from external
scripts and/or direct command line usage by an end-user.
Gerrit1's HTTP POST based format is assumed if the review server
does not have the /ssh_info URL available on it.
Signed-off-by: Shawn O. Pearce <sop@google.com>
There's an extra "," at the end of the line, which is causing
trouble when the manifest file specifies a revision for a
project. Since the default manifest file doesn't specify
revisions for the projects, the problem has gone unnoticed.
Thanks to Barry Silverman <barry@disus.com> for spotting the
issue and providing a patch.
Signed-off-by: Marcelo E. Magallon <marcelo.magallon@gmail.com>
Users are prompted with the list of known changes we are about
to upload, and they can fill out the current change numbers for
any changes which already exist in the data store. For each of
those changes the change number and commit id is sent as part of
the upload request, so Gerrit can insert the new commit as a new
patch set of the existing change, rather than make a new change.
This facility permits developers to replace a patch so they can
address comments made on a prior version of the same change.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way project.GetUploadableBranch(project.CurrentBranch) can tell
us how (if at all) to upload the currently checked out branch.
Signed-off-by: Shawn O. Pearce <sop@google.com>
By setting a project-name on a remote nested within a project forks
of a project like the Linux kernel can be easily handled by fetching
all relevant forks into the same client side project under different
remote names. Developers can create branches off different remotes
using `git checkout --track -b $myname $remote/$branch` and later
`repo upload` automatically redirects to the proper fork project
in the code review server.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way "forks" of a project, e.g. the linux kernel, can be setup to
use different destination projects in the review server by creating
different remotes in the client side Git repository.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The mirror option downloads a complete forrest (as described by the
manifest) and creates a replica of the remote repositories rather
than a client working directory. This permits other clients to
sync off the mirror site.
A mirror can be positioned in a "DMZ", where the mirror executes
"repo sync" to obtain changes from the external upstream and
clients inside the protected zone operate off the mirror only,
and therefore do not require direct git:// access to the external
upstream repositories.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This destroys a local development branch, removing all history
of that branch from ever existing. If the branch is currently
checked out we move back to the upstream revision.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This hook is evaluated by `git gc --auto` to determine if it is a
good idea to execute a GC at this time, or defer it to some later
date. When working on a laptop its a good idea to avoid GC if you
are on battery power as the extra CPU and disk IO would consume a
decent amount of the charge.
The hook is the standard sample hook from git.git contrib/hooks,
last modified in git.git by 84ed4c5d117d72f02cc918e413b9861a9d2846d7.
I added the GPLv2 header to the script to ensure the license notice
is clear, as it does not match repo's own APLv2 license.
We only update hooks during initial repository creation or on
a repo sync. This way we don't incur huge overheads from the
hook stat operations during "repo status" or even the normal
"repo sync" cases.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Some users have noticed that repo doesn't work on VFAT, as we
require a POSIX filesystem with POSIX symlink support. Catching the
OSError during our symlink creation and raising a GitError with a
more descriptive message will help users to troubleshoot and fix
their own installation problems.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If we are already up-to-date we just want to display no output.
This means we have to avoid calling "git merge" if there aren't
commits to be merged into the working directory.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Instead of trying to rebase the changes on a topic branch that
has been fully merged into the upstream branch we track, we should
just fast-forward the topic branch to the new upstream revision.
This way the branch doesn't try to rewrite commits that are already
merged in the upstream.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Git may have been installed without its hooks directory, which
means we won't have any hooks in a repo created git repository.
Since we are just deleting the hooks it doesn't matter.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Downloading and streaming a tar into Git is slower than just
letting the native git:// protocol handle the data transfer,
especially when there are multiple revisions available and
Git can perform delta compression across revisions.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Hiding error messages from the remote peer is not a good idea,
as users should be made aware when the remote peer is not a
complete Git repository so they can alert the administrators
and have the repository corrected.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The initial open source release of the Android 1.0 platform had
some problems with its Perforce->Git imports. Google was forced
to rewrite some history to redirect users onto more stable upstream
sources and correct errors in the imports.
Not everyone has the correct android-1.0 tags, as some users did
manage to fetch the platform early, before the mirror sites crashed
and the history was rewritten.
This change is a band-aid to ensure any stale android-1.0 tags are
get updated to the corrected version. It should be backed out at
some point in the near future, when we can be fairly certain that
everyone has the correct android-1.0 tags.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Now `repo download . 1402` would download the change numbered 1402
into the current project and check it out for the user, using a
detached HEAD. `repo sync .` would back out of the change and
return to the upstream version.
Multiple projects can be fetched at once by listing them out on
the command line as different arguments.
Individual patch sets can be selected by adding a '/n' to indicate
the n-th patch set should be downloaded instead of the default of
patch set 1.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way users are well aware of which account we used when the
uploads are complete, so they can be certain to sign into the web
application with that user identity.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the reflog for the upstream branch has only 1 entry in it, as
the branch has been updated only once, we can get back the 0{40}
object id from `git rev-parse upstream@{1}`, in which case we should
consider it to be the same as if upstream@{1} is not defined.
Signed-off-by: Shawn O. Pearce <sop@google.com>
