Commit Graph

538 Commits

Author SHA1 Message Date
Conley Owens
971de8ea7b Refine groups functionality
Every project is in group "default".  "-default" does not remove
it from this project.  All group names specified in the manifest
are positive names as opposed to a mix of negative and positive.

Specified groups are resolved in order.  If init is supplied with
--groups="group1,-group2", the following describes the project
selection when syncing:

  * all projects in "group1" will be added, and
  * all projects in "group2" will be removed.

Change-Id: I1df3dcdb64bbd4cd80d675f9b2d3becbf721f661
2012-04-23 12:39:05 -07:00
James W. Mills
24c1308840 Add project annotation handling to repo
Allow the optional addition of "annotation" nodes nested under
projects.  Each annotation node must have "name" and "value"
attributes.  These name/value pairs will be exported into the
environment during any forall command, prefixed with "REPO__"

In addition, an optional "keep" attribute with case insensitive "true"
or "false" values can be included to determine whether the annotation
will be exported with 'repo manifest'

Change-Id: Icd7540afaae02c958f769ce3d25661aa721a9de8
Signed-off-by: James W. Mills <jameswmills@gmail.com>
2012-04-23 12:35:08 -07:00
Anatol Pomazau
b962a1f5e0 Check if SHA1 presents in repository
Previously repo had incorrect code that did not really check
if sha1 presents in a project. It worked for tags though.

Check if a revision (either tag or sha1) is present by using
'git rev_parse' functionality.

Change-Id: I1787f3348573948573948753987394839487572b
2012-04-23 11:09:17 -07:00
Colin Cross
5acde75e5d Add manifest groups
Allows specifying a list of groups with a -g argument to repo init.
The groups act on a group= attribute specified on projects in the
manifest.
All projects are implicitly labelled with "default" unless they are
explicitly labelled "-default".
Prefixing a group with "-" removes matching projects from the list
of projects to sync.
If any non-inverted manifest groups are specified, the default label
is ignored.

Change-Id: I3a0dd7a93a8a1756205de1d03eee8c00906af0e5
Reviewed-on: https://gerrit-review.googlesource.com/34570
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
2012-04-13 09:46:00 -07:00
pelya
d67872d2f4 Option for 'repo diff' to generate output suitable for 'patch' cmd
The -u option causes 'repo diff' to generate diff output
with file paths relative to the repository root,
so the output can be applied to the Unix 'patch' command.
The name '-u' was selected for convenience, because
both 'diff' and 'git diff' accept the option with the same name
to generate an 'unified diff' output suitable for 'patch' command.

Change-Id: I79c8356db4ed20ecaccc258b3ba139db76666fe0
Reviewed-on: https://gerrit-review.googlesource.com/34380
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
2012-04-13 09:20:10 -07:00
Daniel Sandler
e9d6b611c5 New flag for repo upload: --current_branch (--cbr)
A convenient equivalent to `repo upload --br=<current git branch>`.

Note that the head branch will be selected for each project
uploaded by repo, so different branches may be uploaded for
different projects.

Change-Id: I10ad8ceaa63f055105c2d847c6e329fa4226dbaf
2012-04-06 10:43:36 -04:00
Shawn O. Pearce
c3d2f2b76f Ignore /clone.bundle on HTTP 401, 403 and 404
401: Unauthorized, authentication may be required. This is usually
     handled internally by the HTTP client in Python. If it reaches
     our code in repo, the Python HTTP client didn't find a password
     in ~/.netrc that it could use.

403: Authentication was supplied, but is incorrect. It might be
     that the CDN doesn't want to offer this clone.bundle file
     to the client, but the Git fetch operation would still be
     successful. This might arise if branch level read controls
     were used in Gerrit Code Review and the /clone.bundle file
     contained branches not visible to the client.

404: The server has no /clone.bundle file available.

In all of these cases, sliently ignore the /clone.bundle file HTTP
error and let the Git operation take over.

Change-Id: I1787f3cac86b017351952bbb793fe5874d83c72b
2012-03-22 14:18:40 -07:00
Anatol Pomazau
cd7c5deca0 Do not change branch.foo.merge in case of manifest sync
In case of manifest/smart sync repo changes ".merge" config
option from branch to SHA. Doing 'repo upload' fails as
repo tries to upload to a remote branch that looks like SHA
(e.g. refs/for/23423423423423423423423)

Do not update the .merge in case if revision is SHA.

Change-Id: I9139708fa17f21eec5a7e23c3255333626bf529e
2012-03-20 14:11:56 -07:00
Shawn O. Pearce
e02ac0af2e sync: --no-clone-bundle disables the clone bundle support
Change-Id: Ia9ed7da8451b273c1be620c3dd0dcad777b29096
2012-03-14 15:38:28 -07:00
Shawn O. Pearce
898e12a2d9 Permit - in URL schemes for special URLs
Clients might be using their own special git-remote-* helper that
has a hypen in its name. Permit - in the scheme part of the URL
when trying to decide if it is an SSH URL and assume it is *not*
SSH if the URL matches "foo-bar://" style.

Change-Id: I7ba2d810a614f6e605a441d5972902c4a14e73fd
2012-03-14 15:28:22 -07:00
Xiaodong Xu
ae0a36c9a5 Add support for Apache Digest authentication for repo init.
repo tool supports only Basic authentication for now. For those
who want to use this tool to manage their own projects, in case
the administrator has configured the Apache server with Digest
authentication method, users will fail to be authenticated when
they run the command 'repo init'.
Add the digest authentication password manager to the handler
list will fix this issue.

Since Git HTTP protocol will require the user be authenticated
for fetch operation first before pushing commits to the remote,
it is unlikely for the administrator to implement anonymous
read (aka pull) access and write access (aka push) for
authenticated user. Both read and write have to be authenticated.
Be aware that the user may have to add an extra line in his
~/.netrc file:
-------------------
account example.com
-------------------
where 'example.com' is the realm for Apache Digest authentication.

Change-Id: I76eb27b205554426d9ce1965deaaf727b87916cd
Signed-off-by: Xiaodong Xu <stid.smth@gmail.com>
2012-03-14 15:01:34 -07:00
Ali Utku Selen
76abcc1d1e repo status to print project name on clean gits
repo status just prints "# on branch oprofile" if you have branched
in clean status. This doesn't really tell which branch is meant.

Instead we can use the same syntax with modified gits which will
give us detailed information.

Change-Id: I55fe5154d278e10a814281dd2ba501ec6e956730
2012-03-12 12:25:40 -07:00
Mike Pontillo
d315382572 Add 'rebase="false"' attribute to the <project/> XML.
This new attribute can prevent 'repo sync' from automatically rebasing.

I hit a situation in where one of the git repositories I was tracking
was actually an external repository that I wanted to pull commits
into and merge myself. (NOT rebase, since that would lose the merge
history.) In this case, I'm not using 'repo upload', I'm manually
managing the merges to and from this repository.

Everything was going great until I typed 'repo sync' and it rebased
my manually-merged tree. Hence the option to skip it.

Change-Id: I965e0dd1acb87f4a56752ebedc7e2de1c502dbf8
2012-03-12 12:24:22 -07:00
Conley Owens
43bda84362 Avoid missing content-length header in project.py
Occassionally, the content-length may be missing when using urlib
in python 2.6 and 2.7.  This change assumes the value of the header is
0 if it doesn't exist

Change-Id: Iaf1c8a796bc667823d4d7c30f9b617644b271d00
2012-03-12 12:13:15 -07:00
Shawn O. Pearce
9b017dab46 Update SUBMITTING_PATCHES
The review server is now at gerrit-review.googlesource.com.

Change-Id: I4be67fdb1876eb2e2af4420ac63557596b9e233b
2012-02-28 18:54:33 -08:00
Chris Wolfe
e9dc3b3368 sync: Add manifest_name parameter
This parameter changes the manifest used by 'repo sync' for only
this execution. It should be useful for developers wishing to get
the repo temporarily into a known state, without clobbering their
existing manifest.

Tested by shifting Chrome OS between minilayout and full, and
between several release-builder-generated manifests.

Change-Id: I14194b665195b0e78f368d9ec8b8a83227af2627
2012-01-26 12:32:36 -05:00
Shawn O. Pearce
c9571423f8 upload: Support uploading to Gerrit over https://
If SSH is not available, Gerrit returns NOT_AVAILABLE to the /ssh_info
query made by repo upload. In this case fallback to the /p/$PROJECT URL
that Gerrit also exports and use that for uploads.

Change-Id: I1e3e39ab709ecc0a692614a41a42446426f39c08
2012-01-11 16:18:40 -08:00
Shawn O. Pearce
34fb20f67c Revert "Default repo manifest settings in git config"
This reverts commit ee1c2f5717.

This breaks a lot of buildbot systems. Rolling it back for now
until we can understand what the breakage was and how to fix it.
2011-11-30 13:41:02 -08:00
Shawn O. Pearce
ecff4f17b0 Describe the repo launch version in repo version
repo version v1.7.8
         (from https://android.googlesource.com/tools/repo.git)
  repo launcher version 1.14
         (from /home/sop/bin/repo)
  git version 1.7.8.rc2.256.gcc761
  Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
  [GCC 4.4.3]

Change-Id: Ifcbe5b0e226a1a6ca85455eb62e4da5e9a0f0ca0
2011-11-29 15:02:15 -08:00
Shawn O. Pearce
cc14fa9820 Improve error handling when reading loose refs
When repo is trying to figure out branches the repository has by
traversing refs/heads, add exception handling for readline.

Change-Id: If3b2a3720c6496f52f629aa9a2539f186d6ec882
2011-11-29 14:43:04 -08:00
Daniel Sandler
3ce2a6b46b Propagate result codes from subcmds to sys.exit().
Allows scripts driving repo to know when git failures have
occurred, not just repo internal errors.

Change-Id: Id20fbbb405c35a148e72c87b822da3f3bf93839c
2011-11-29 14:38:19 -08:00
Victor Boivie
841be34968 Don't prompt the user for name/email unless necessary
If the user has already configured a workspace, use these values
when re-running 'repo init'.

Otherwise, if the user has global name and e-mail set, use these.

It's always possible to override this and be prompted by specifying
--config-name when running 'repo init'.

Change-Id: If45f0e4b14884071439fb02709dc5cb53f070f60
2011-11-29 14:31:56 -08:00
Victor Boivie
ee1c2f5717 Default repo manifest settings in git config
A default manifest URL can be specified using:
  git config --global repo-manifest.<id>.url <url>

A default manifest server can be specified using:
  git config --global repo-manifest.<id>.server <url>

A default git mirror reference can be specified using:
  git config --global repo-manifest.<id>.reference <path>

This will allow the user to use 'repo init -u <id>' as
a shorter alternative to specifying the full URL.

Also, manifest server will not have to be specified in the
manifest XML and the reference will not have to be specified
on the command line. If they are, they will override these
default values however.

Change-Id: Ifdbc160bd5909ec7df9efb0c5d7136f1d9351754
Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
2011-11-29 14:24:58 -08:00
Christer Fletcher
6a1f737380 Added remote destination branch information when uploading.
Several times one have done an upload only to later notice in gerrit
that the upload was done to the wrong branch as the git has not yet
been branched for the current git. This change will make repo print
what the destination branch is when asking the user if she wants to
go through with the upload.

Change-Id: Ia9c3a92a6a04c022edfebf4f8d651ac062bb1f3b
2011-11-29 14:01:57 -08:00
Mike Frysinger
e9311273dd repo: capitalize default prompt char
It is common in command line tools to indicate what the default answer
will be if the user simply hits enter.  In repo, the display is just
"y/n" with no indication as to which is the default.  So change the n
to N in the messages since that is how repo operates.

Change-Id: I81819ae630355072eb0365e59168b0921289498f
2011-11-29 12:38:52 -08:00
chenguodong
605a9a487b Fixed UnicodeDecodeError while uploading changes.
When commit with comment that has non-ASCII characters,
UnicodeDecodeError will be raised
while uploading multiple project/branch changes.
Because some strings in script are not str type, but unicode.
So all the strings are decoded to unicode,
and python use ascii to do this,
it can not decode non-ASCII characters,
so UnicodeDecodeError raised.

Signed-off-by: chenguodong <chenguodong@huawei.com>

Change-Id: I46447f489a4b9760a5899c7ba9d764b688594e46
2011-11-29 12:11:41 -08:00
Anatol Pomazau
2a32f6afa6 Fix typo
Change-Id: Idd68ad0a34fcf4bd4e18b0248f50187a539d610a
2011-11-29 12:09:35 -08:00
Ulrik Sjölin
498fe90b45 Stabilize repo communication with subprocesses.
Make repo use the standard way in python to work with pipes.
Communication via pipes to sub processes is done by calling
communicate(). This will make repo not hang every now and
then.

Change-Id: Ibe2c4ecbdbcbe72f0b725ca50d54088e5646fc5d
2011-11-29 11:54:58 -08:00
Anatol Pomazau
53d6f4d17e Add a sync flag that fetches only current branch
There is also shortcuts in case if the "current branch" is
a persistent revision such as tag or sha1. We check if the
persistent revision is present locally and if it does - do
no fetch anything from the server.

This greately reduces sync time and size of the on-disk repo

Change-Id: I23c6d95185474ed6e1a03c836a47f489953b99be
2011-11-03 13:08:27 -07:00
Conley Owens
9d8f914fe8 Remove extra '/' in RemoteSpec
urljoin appends a '/' if only the domain is in the url path.  This
change strips that off before creating a RemoteSpec
2011-11-03 13:05:14 -07:00
Conley Owens
ceea368e88 Correctly name projects when mirroring
A bug introduced by relative urls caused projects such as manifest.git
to be placed in the root directory instead of the directory they should
by in.

This fix creates and refers to a resolvedFetchUrl in the _XmlRemote
class in order to get a fetchUrl that is never relative.
2011-10-20 11:01:38 -07:00
Shawn O. Pearce
b660539c4a Fix sync on Python 2.6.6
Python 2.6.6 has the same bug as Python 2.7, where HTTP
authentication just stops working, but does not have the
setter method to clear the retry counter. Work around by
setting the field directly if it exists.

Change-Id: I6a742e606bb7750dc66c33fc7c5d1310541db2c8
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 15:58:07 -07:00
Shawn O. Pearce
752371d91b help: Fix help sync
help sync crashed as sync required the manifest to be configured to
create the option parser, as the default number of jobs is required.

Change-Id: Ie75e8d75ac0e38313e4aab451cbb24430e84def5
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 15:23:41 -07:00
Shawn O. Pearce
1a68dc58eb upload: Honor REPO_HOST_PORT_INFO environment variable
REPO_HOST_PORT_INFO can be set to 'host:port' and be used
instead of the review URL given in the manifest.

Change-Id: I440bdecb2c2249fe5285ec5d0c28a937b4053450
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 14:12:46 -07:00
Shawn O. Pearce
df5ee52050 Fix Python 2.4 support
Change-Id: I89521ae52fa564f0d849cc51e71fee65b3c47bab
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 14:06:11 -07:00
Shawn O. Pearce
fab96c68e3 Work around Python 2.7 urllib2 bug
If the remote is using authenticated HTTP, but does not have
$GIT_URL/clone.bundle files in each repository, an initial sync
would fail around 8 projects in due to the library not resetting
the number of failures after getting a 404.

Work around this by updating the retry counter ourselves.

The urllib2 library is also not thread-safe. Make it somewhat
safer by wrapping the critical section with a lock.

Change-Id: I886e2750ef4793cbe2150c3b5396eb9f10974f7f
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 12:18:07 -07:00
Shawn O. Pearce
bf1fbb20ab Fix AttributeError: 'HTTPError' object has no attribute 'reason'
Not every version of urllib2 supplies a reason object on the
HTTPError exception that it throws from urlopen().  Work around
this by using str(e) instead and hope the string formatting includes
sufficient information.

Change-Id: I0f4586dba0aa7152691b2371627c951f91fdfc8d
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 09:31:58 -07:00
Shawn O. Pearce
29472463ba Work around Python 2.7 failure to initialize base class
urllib2 returns a malformed HTTPError object in certain situations.
For example, urllib2 has a couple of places where it creates an
HTTPError object with no fp:

  if self.retried > 5:
    # retry sending the username:password 5 times before failing.
    raise HTTPError(req.get_full_url(), 401, "basic auth failed",
                    headers, None)

When it does that, HTTPError's ctor doesn't call through to
addinfourl's ctor:

  # The addinfourl classes depend on fp being a valid file
  # object.  In some cases, the HTTPError may not have a valid
  # file object.  If this happens, the simplest workaround is to
  # not initialize the base classes.
  if fp is not None:
    self.__super_init(fp, hdrs, url, code)

Which means the 'headers' slot in addinfourl is not initialized and
info() fails.  It is completely insane that urllib2 decides not to
initialize its own base class sometimes.

Change-Id: I32a0d738f71bdd7d38d86078b71d9001e26f1ec3
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 09:24:07 -07:00
Shawn O. Pearce
c325dc35f6 sync: Fetch after applying bundle and retry after errors
After a $GIT_URL/clone.bundle has been applied to the new local
repository, perform an incremental fetch using `git fetch` to ensure
the local repository is up-to-date. This allows the hosting server
to offer stale /clone.bundle files to bootstrap a new client.

If a single git fetch fails, it may succeed again after a short
delay.  Transient failures are typical in environments where the
remote Git server happens to have limits on how many requests it
can serve at once (the anonymous git daemon, or an HTTP server).
Wait a randomized delay between 30 and 45 seconds and retry the
failed project once.  This delay gives the site time to recover
from a transient traffic spike, and the randomization makes it less
likely that a spike occurs again from all of the same clients.

Change-Id: I97fb0fcb33630fb78ac1a21d1a4a3e2268ab60c0
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-03 08:30:24 -07:00
Shawn O. Pearce
f322b9abb4 sync: Support downloading bundle to initialize repository
An HTTP (or HTTPS) based remote server may now offer a 'clone.bundle'
file in each repository's Git directory. Over an http:// or https://
remote repo will first ask for '$URL/clone.bundle', and if present
download this to bootstrap the local client, rather than relying
on the native Git transport to initialize the new repository.

Bundles may be hosted elsewhere. The client automatically follows a
HTTP 302 redirect to acquire the bundle file. This allows servers
to direct clients to cached copies residing on content delivery
networks, where the bundle may be closer to the end-user.

Bundle downloads are resumeable from where they last left off,
allowing clients to initialize large repositories even when the
connection gets interrupted.

If a bundle does not exist for a repository (a HTTP 404 response
code is returned for '$URL/clone.bundle'), the native Git transport
is used instead. If the client is performing a shallow sync, the
bundle transport is not used, as there is no way to embed shallow
data into the bundle.

Change-Id: I05dad17792fd6fd20635a0f71589566e557cc743
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-28 10:07:36 -07:00
Conley Owens
db728cd866 Allow remote url to be relative to manifst url 2011-09-28 10:07:01 -07:00
Shawn O. Pearce
c4657969eb sync: Update default -j flag from manifest
If the manifest is updated and the default sync-j attribute
was modified, honor it during this sync session if the user
has not supplied a -j flag on the command line.

Change-Id: I127ee5c779e2bbbb40b30bddc10ec1fa704b3bf3
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-26 09:08:44 -07:00
Shawn O. Pearce
7b947de1ee Ignore missing ~/.netrc
Change-Id: Ifa6065d57a6cb11ad57ddd44bc88d9690fe234ab
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-23 11:50:31 -07:00
Shawn O. Pearce
6392c87945 sync: Allow -j to have a default in manifest
This permits manifest authors to suggest a number of parallel
fetch operations against a remote server. For example, Gerrit
Code Review servers support queuing of requests and processes
them in first-in, first-out order. Running concurrent fetches
can utilize multiple CPUs on the Gerrit server, but will also
decrease overall operation latency by having the request put
into the queue ready to execute as soon as a CPU is free.

Change-Id: I3d3904acb6f63516bae4b071c510ad57a2afab18
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-22 18:08:27 -07:00
Shawn O. Pearce
97d2b2f7a0 sync: Limit -j to file descriptors
Each worker thread requires at least 3 file descriptors to run the
forked 'git fetch' child to operate against the local repository.
Mac OS X has the RLIMIT_NOFILE set to 256 by default, which means
a sync -j128 often fails when the workers run out of pipes within
the Python parent process.

Change-Id: I2cdb14621b899424b079daf7969bc8c16b85b903
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-22 18:08:26 -07:00
Shawn O. Pearce
3a0e782790 Add global option --time to track execution
This prints a simple line after a command ends, providing
information about how long it executed for using real wall
clock time. Its mostly useful for looking at sync times.

Change-Id: Ie0997df0a0f90150270835d94b58a01a10bc3956
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-22 18:08:18 -07:00
Shawn O. Pearce
490d09a314 Support units in progress messages
This allows our progress meter to be used for bytes transferred, by
setting the units to KB or MB to let the user know the size.

Change-Id: Ie8653d4a40d79439026c18bd51915845b2c5bba9
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-19 14:52:57 -07:00
Shawn O. Pearce
13111b4e97 Add support for url.*.insteadof
Teach repo how to resolve URLs using the url.insteadof feature
that C Git natively uses during clone, fetch or push. This will
later allow repo to resolve a URL before accessing it directly.
We do not want to pre-resolve things and store the resolved URL
into individual projects, as this makes it impossible for the
user to undo the insteadof mapping at a later date.

Change-Id: I0f62e811197c53fbc8a8be424e3cabf4ed07b4cb
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-19 14:52:57 -07:00
Shawn O. Pearce
bd0312a484 Support ~/.netrc for HTTP Basic authentication
If repo tries to access a URL over HTTP and the user needs to
authenticate, offer a match from ~/.netrc. This matches behavior
with the Git command line client.

Change-Id: I803f3c5d562177ea0330941350cff3cc1e1bef08
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-19 14:52:32 -07:00
Shawn O. Pearce
334851e4b6 Enhance HTTP support
Setting REPO_CURL_VERBOSE=1 in the environment will register a debug
level HTTPHandler on the urllib2 library, showing HTTP requests and
responses on the stderr channel of repo.

During any HTTP or HTTPS request created inside of the repo process,
a custom User-Agent header is now defined:

  User-Agent: git-repo/1.7.5 (Linux) git/1.7.7 Python/2.6.5

Change-Id: Ia5026fb1e1500659bd2af27416d85e205048bf26
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-19 14:51:47 -07:00