Compare commits

...

67 Commits

Author SHA1 Message Date
34acdd2534 Fix ManifestParseError when first child node is comment
If the first line of manifest.xml is a XML comment, root.childNodes[0]
is not a 'manifest' element node. The python minidom module will makes
a 'Comment' node as root.childNodes[0]. Since the original code only
checks whether the first child node is 'manifest', it couldn't do any
command including 'sync' due to the 'ManifestParseError' exception. This
patch could allow the comments between '<?xml ...?>' and '<manifest>' in
the manifest.xml file.

Change-Id: I0b81dea4f806965eca90f704c8aa7df49c579402
2012-09-07 08:38:08 -07:00
d94aaef39e sync: Correct imports of R_HEADS and HEAD
`R_HEADS` is imported twice, from both the git_refs and project
modules.

It is actually defined in git_refs, and in project it is imported
from there, so the import of `R_HEADS` from project in the sync
module is redundant.  Remove it.

`HEAD` is imported from project, but like `R_HEADS` it is actually
defined in git_refs.  Import it from git_refs instead.

Change-Id: I8e2b0217d0d9f9f4ee5ef5b8cd0b026174ac52f4
2012-09-07 10:17:00 +02:00
bd489c4eaa sync: catch exceptions when connecting to the manifest server
When connecting to the manifest server, exceptions can occur but
are not caught, resulting in the repo sync exiting with a python
traceback.

Add handling of the following exceptions:

- IOError, which can be raised for example if the manifest server
URL is malformed.
- xmlrpclib.ProtocolError, which can be raised if the connection
to the manifest server fails with HTTP error.
- xmlrpclib.Fault, which can be raised if the RPC call fails for
some other reason.

Change-Id: I3a4830aef0941debadd515aac776a3932e28a943
2012-09-06 11:18:25 -07:00
2dc810c2e4 Fix errors when clone.bundle missing on server
Catch curl failures to download clone.bundle; don't let git try to parse
the 404 page as a bundle file (was causing much user confusion).

This should eliminate false error messages from init and sync such as:
  error: '.repo/manifests.git/clone.bundle' does not look like a v2 bundle file
  fatal: Could not read bundle '.repo/manifests.git/clone.bundle'.
  error: RPC failed; result=22, HTTP code = 400

Signed-off-by: Matt Gumbel <matthew.k.gumbel@intel.com>
Change-Id: I7994f7c0baecfb45bb5a5850c48bd2a0ffabe773
2012-09-06 10:54:46 -07:00
bb1b5f5f86 Allow projects to be specified as notdefault
Instead of every group being in the group "default", every project
is now in the group "all".   A group that should not be downloaded
by default may be added to the group "notdefault".

This allows all group names to be positive (instead of removing groups
directly in the manifest with -default) and offers a clear way of
selecting every project (--groups all).

Change-Id: I99cd70309adb1f8460db3bbc6eff46bdcd22256f
2012-09-05 11:46:48 -07:00
e2126652a3 Make "repo sync -j<count>" stop properly on Ctrl-C.
The threaded 'repo sync' implementation would very often freeze the
process when interrupted by the user with Ctrl-C. The only solution
being to kill -9 the process explicitly from another terminal.

The reason for this is best explained here:

http://snakesthatbite.blogspot.fr/2010/09/cpython-threading-interrupting.html

This patch makes all helper sync threads 'daemon', which allows the
process to terminate immediately on Ctrl-C.

Note that this will forcefully kill all threads in case of interruption; this
is generally a bad thing, but:

  1/ This is equivalent to calling kill -9 in another terminal, which
     is the _only_ thing that can currently stop the process.

  2/ There doesn't seem to be a way to tell the worker threads to
     gently stop when they are in a blocking operation anyway (even
     in the non-threaded case).

+ Do the same for "repo status -j<count>".

Change-Id: Ieaf45b0eacee36f35427f8edafd87415c2aa7be4
2012-09-05 11:38:41 -07:00
9a27d0111d manifest-format.txt: Add documentation for GetManifest RPC method
Add documentation of the GetManifest RPC method in the
manifest-server section.

Change-Id: I5cda5929bc8a0ca9d3f2b9da63216427041d2823
2012-09-05 06:00:47 -07:00
918ff85c1e repo manifest: default to stdout if no "-o"
Change-Id: I1b0ff9ed5df6386f0c2a851c6c48d063199fe663
2012-09-04 09:30:18 -07:00
3d07da82ab init: Improved help text for the --mirror option
Change-Id: Ia6032865f9296b29524c2c25b72bd8e175b30489
2012-08-23 12:15:49 +02:00
e15c65abc2 Remove unused imports
There are several imports that are not used.  Remove them.

Change-Id: I2ac3be66827bd68d3faedcef7d6bbf30ea01d3f2
2012-08-23 12:15:26 +02:00
daa851f6cd manifest-format.txt: Fix a couple of minor spelling mistakes
Change-Id: Ic2d266c8cf08827a71846db9d3711feb02885f01
2012-08-22 09:39:41 -07:00
a43f42f9ff Patches should be submitted to master, not maint
Update SUBMITTING_PATCHES accordingly.

Change-Id: I6fd57a84c67d3762f1f23276d95cac2aeecd5e8f
2012-08-21 14:06:10 +02:00
bb8337fe0f Merge branch 'master' into maint
master's original purpose was to forge ahead on using git submodules,
but this route has been abandoned.

Change-Id: I164a9efc7821bcd1b941ad76649764722046081b
2012-08-14 11:34:34 -07:00
17f85eab24 Omit all default groups when generating a manifest
One of the recent changes introduced implicit path:xxx and name:xxx groups
to every project, however they are not being stripped when generating
a manifest using "repo manifest" command resulting in clutter

Change-Id: Iec8610ba794b2fe4a6cdf0f59ca561595b66f9b5
2012-08-07 11:42:54 -07:00
b9477bc2dd project.py: Replace the relpath function with os.path.relpath
Change-Id: Ib313340344968211cecfc0a718f6072e41da1a91
2012-08-06 23:51:43 +02:00
5e7127d00b Use curl command line tool for clone.bundle
urllib2 is not thread safe and may be causing sync to lock up or
not work correctly on various platforms. Instead use the command
line curl program.

Change-Id: I36eaf18bb4df089d26ea99d533cb015e7c616eb0
2012-08-02 15:18:10 -07:00
5d0efdb14a sync: Honor --no-clone-bundle with -j1
Change-Id: I7c12902e386121a374d525be673092360c67c53d
2012-08-02 12:13:01 -07:00
f35b2d9c31 Fix mirror mode
Change-Id: Ica0e8392562a7ae5aad7e45441c1540e5e2b0238
2012-08-02 11:46:22 -07:00
e0904f721b Fix unsupported operand type(s) for +: 'int' and 'str'
Change-Id: I88455107d63daaa60c3b33c010aa8c730a590c70
2012-08-01 20:44:23 -07:00
9830553748 Fix percent done on resumed /clone.bundle
The Content-Length when resuming is the number of bytes that
remain in the file. To compute the total size as expected by
the progress meter, we must add the bytes already stored.

While we are in this method fix uses of % operator to ensure
a tuple is always supplied.

Change-Id: Ic899231b5bc0ab43b3ddb1d29845f6390e820115
2012-08-01 17:41:26 -07:00
2bc7f5cb3a Fix bug in version_tuple to handle strings with -rc#
Example of version string that caused a problem: git version 1.7.11-rc3

Change-Id: I8a68b6b37f7b2ded23a1f8ae0d12131050a8807b
CC: sop@google.com
2012-07-31 22:18:47 -07:00
b292b98c3e Add remote alias support in manifest
The `alias` is an optional attribute in element `remote`. It can be
used to override attibute `name` to be set as the remote name in each
project's .git/config. Its value can be duplicated while attribute
`name` has to be unique across the manifest file. This helps each
project to be able to have same remote name which actually points
to different remote url.

It eases some automation scripts to be able to checkout/push to same
remote name but actually different remote url, like:

repo forall -c "git checkout -b work same_remote/work"
repo forall -c "git push same_remote work:work"

for example:
The manifest with 'alias' will look like:

<?xml version='1.0' encoding='UTF-8'?>
<manifest>
  <remote alias="same_alias" fetch="git://git.external1.org/" name="ext1"
      review="http://review.external1.org"/>
  <remote alias="same_alias" fetch="git://git.external2.org/" name="ext2"
      review="http://review.external2.org"/>
  <remote alias="same_alias" fetch="ssh://git.internal.com:29418" name="int"
      review="http://review.internal.com"/>
  <default remote="int" revision="int-branch" sync-j="2"/>
  <project name="path/to/project1" path="project1" remote="ext1"/>
  <project name="path/to/project2" path="project2" remote="ext2"/>
  <project name="path/to/project3" path="project3"/>
  ...
</manifest>

In each project, use command "git remote -v"

project1:
same_alias  git://git.external1.org/project1 (fetch)
same_alias  git://git.external1.org/project1 (push)

project2:
same_alias  git://git.external2.org/project2 (fetch)
same_alias  git://git.external2.org/project2 (push)

project3:
same_alias  ssh://git.internal.com:29418/project3 (fetch)
same_alias  ssh://git.internal.com:29418/project3 (push)

Change-Id: I2c48263097ff107f0c978f3e83966ae71d06cb90
2012-07-31 22:13:13 -07:00
2f127de752 Add "repo overview" command.
The overview command shows an overview of each branch in all (or the
specified) projects.  The overview lists any local commits that have
not yet been merged into the project.

The report output is inspired by the report displayed following a
"repo prune" event, with the addition of listing the one-line log
messages for each commit that is not yet merged.

The report can also be filtered to show only active branches; by
default all branches that have commits beyond the upstream HEAD will
be listed.

Change-Id: Ibe67793991ad1aa38de3bc9747de4ba64e5591aa
2012-07-31 22:08:32 -07:00
7da1314e38 Inject the project name into each projects groups.
For CrOS, we have scenarios were people checkout a smaller version
of our manifest via groups, and enable individual repositories as
needed for their work.  Previously this was via local_manifest
manipulation, which breaks via manifest-groups would require a
remove-project tag.

Via injecting the projects name into the projects groups, this
allows us to instead manipulate the configured groups allowing
the user to turn on/off projects as necessary.

Change-Id: I07b7918e16cc9dc28eb47e19a46a04dc4fd0be74
2012-07-31 22:05:44 -07:00
435370c6f0 upload: add --draft option.
Change-Id: I6967ff2f8163cd4116027b3f15ddb36875942af4
2012-07-28 15:44:05 -07:00
e8f75fa368 Don't delete the branch config when switching branches.
The fix for issue #46 in 5d016502eb appears to break syncing in some
situations: the branch is deleted after the point where it's been
configured, which deletes part of its configuration and causes the
config to change each time you call `repo init`, alternating between a
configuration that works and one that doesn't.

Instead of deleting the branch with git branch -D, use git update-ref -d
which just deletes the ref (to avoid the rebase) without touching the
configuration for the branch that was set up during the first repo init.

This appears to ensure the config is left in a valid state all the time
no matter what combination of repo init commands you run, without
reintroducing the rebasing issue.

Change-Id: Iaadaa6e56a46840bbc593fa5b35cb5b34cd3ce69
2012-07-20 15:33:17 +01:00
87636f2ac2 Fix for failures with repo upload for projects that have a SHA1 for a revision; instead use the default manifest revision
Change-Id: Ie5ef5a45ed6b0ca1a52a550df3cd7bd72e745f5f
2012-06-14 16:54:32 -07:00
e7a3bcbbb8 Merge branch 'stable'
* stable:
  Fix mirror clients with no worktree
2011-01-10 13:28:22 -08:00
25b51d8cb7 Merge branch 'stable'
* stable:
  Bump repo version to 1,10
2011-01-10 09:02:23 -08:00
cef005c3e8 Merge branch 'maint'
* maint:
  help: Don't show empty Summary or Description sections
  sync: Run `git gc --auto` after fetch
  Add "repo branch" as an alias for "repo branches"
  upload: Catch and cleanly report connectivity errors
  forall: Silently skip missing projects
  Fix to display the usage message of the command download when the user don't provide any arguments to 'repo download'.
  Use os.environ.copy() instead of dict()
  Make path references OS independent
2011-01-09 17:39:30 -08:00
71cab95b4c Merge branch 'stable'
* stable:
  Encode the environment variables passed to git
  Exit with statuscode 0 for repo help init
2011-01-09 17:29:50 -08:00
9275fd4329 Merge branch 'stable'
* stable:
  Fixed race condition in 'repo sync -jN' that would open multiple masters.
2010-12-22 14:46:15 -08:00
13f3da50d4 Merge branch 'stable'
* stable: (33 commits)
  Added feature to print a <notice> from manifest at the end of a sync.
  sync: Use --force-broken to continue other projects
  upload: Remove --replace option
  sync --quiet: be more quiet
  sync: Enable use of git clone --reference
  Only delete corrupt pickle config files if they exist
  Don't allow git fetch to start ControlMaster
  Check for existing SSH ControlMaster
  Fix for handling values of EDITOR which contain a space.
  upload: Fix --replace flag
  rebase: Pass through more options
  upload: Allow review.HOST.username to override email
  upload -t: Automatically include local branch name
  Warn users before uploading if there are local changes
  sync: Try fetching a tag as a last resort before giving up
  rebase: Automatically rebase branch on upstrea
  upload: Automatically --cc folks in review.URL.autocopy
  Fix format string bugs in grep
  Do not invoke ssh with -p argument when no port has been specified.
  Allow files to be copied into new folders
  ...

Conflicts:
	git_config.py
	manifest_xml.py
	subcmds/init.py
	subcmds/sync.py
	subcmds/upload.py

Change-Id: I4756a6908277e91505c35287a122a775b68f4df5
2010-12-07 11:13:29 -08:00
3218c13205 Use os.environ.copy() instead of dict()
Signed-off-by: Shawn O. Pearce <sop@google.com>
2010-12-07 08:46:14 -08:00
b0f9a02394 Make path references OS independent
Change-Id: I5573995adfd52fd54bddc62d1d1ea78fb1328130
2010-11-29 13:17:53 -06:00
69b1e8aa65 Merge branch 'stable'
* stable:
  Automatically install Gerrit Code Review's commit-msg hook
  Fail sync when encountering "N commits behind."
  Check that we are not overwriting a local repository when syncing.
  Honor url.insteadOf when setting up SSH control master connection
  sync: Fix split call on malformed email addresses
  Fixing project renaming bug.

Conflicts:
	hooks/commit-msg
	project.py
	subcmds/sync.py

Change-Id: I5eaf8fef8cbe4a95d124368112293a9ca64325bf
2010-03-06 19:29:56 -08:00
840ed0fab7 Fix to display the usage message of the command download when the user
don't provide any arguments to 'repo download'.

Signed-off-by: Thiago Farina <thiago.farina@gmail.com>
2009-09-09 00:41:34 -04:00
c024912fb8 commit-msg: Don't create message with only Change-Id
If a user aborts a commit, the commit-msg hook is still called,
but with an empty file.  We need to leave the empty file alone.

Change-Id: I13766135dac267823cb08ab76f67d2000ba2d1ce
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-08-25 12:03:55 -07:00
15f6579eb3 commit-msg: Update the commit message hook
This version fixes a bug where Change-Id lines become the subject
line, if the subject used a pattern like the subject of this
message does.

Change-Id: I7f7e0363091d03eb05dead2992fc19763214de65
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-08-22 19:23:55 -07:00
d4cd69bdef forall: Silently skip missing projects
If a project is missing locally, it might be OK to skip over it
and continue running the same command in other projects.

Bug: REPO-43
Change-Id: I64f97eb315f379ab2c51fc53d24ed340b3d09250
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-08-22 18:51:02 -07:00
d2dfac81ad upload: Catch and cleanly report connectivity errors
Instead of giving a Python backtrace when there is a connectivity
problem during repo upload, report that we cannot access the host,
and why, with a halfway decent error message.

Bug: REPO-45
Change-Id: I9a45b387e86e48073a2d99bd6d594c1a7d6d99d4
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-08-22 18:41:16 -07:00
4719901067 upload: Document --replace is deprecated
Change-Id: I52715bcfec9c038d0e02505aa7e4054ebc0434aa
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-08-22 18:41:09 -07:00
a949fa5d20 Automatically install Gerrit Code Review's commit-msg hook
Most users of repo are also using Gerrit Code Review, and will want
the commit-msg hook to be automatically installed into their local
projects so that Change-Ids are assigned when commits are created,
not when they are first uploaded.

Change-Id: Ide42e93b068832f099d68a79c2863d22145d05ad
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-08-22 18:22:04 -07:00
0afac0856c Merge change 11206
* change 11206:
  Do not invoke ssh with -p argument when no port has been specified.
2009-08-17 08:09:05 -07:00
4c0f670465 Do not invoke ssh with -p argument when no port has been specified.
This change allows local SSH configuration to choose the port number to
use when not explicitly set in the manifest.
2009-08-16 11:26:57 -07:00
33f0e786bb Add "repo branch" as an alias for "repo branches"
For those of us that are used to typing "git branch".

Signed-off-by: Mike Lockwood <lockwood@android.com>
2009-07-14 15:23:39 -04:00
57272ba82e manifest: Support --upgrade to submodule format, from XML
By running `repo manifest --uprade` an administrator can update the
current manifest format from the XML format to submodule format, but
we need all projects to be checked out in a work tree for this to
function correctly.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:51:13 -07:00
0125ae2fda Introduce manifest format using git submodules
If a manifest top level directory contains '.gitmodules' we now
assume this is a git module format manifest and switch to using
that code, rather than the legacy XML based manifest.

At the same time, we move the bare repository for a project from
$TOP/.repo/projects/$REPO_PATH.git to be $REPO_NAME.git instead.
This makes it easier for us to later support a repo init from an
existing work tree, as we can more accurately predict the path of
the project's repository in the workspace.  It also means that the
$TOP/.repo/projects/ directory is layed out like a mirror would be.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:50:52 -07:00
a7ce096047 Allow meta projects to be created not under .repo/
Some types of manifests might prefer to put their meta project work
tree under topdir, rather than inside of the .repo/ directory.  We
can support that by allowing relpath to be optionally passed in.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:50:52 -07:00
87bda12e85 sync: Support upgrading manifest formats
If the manifest format changes during init or sync we need to do
a full reparse of the manifest, and possibly allow the new object
to reconfigure the local workspace to match its expectations.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:50:52 -07:00
5f947bba69 init: add -o, --origin to name manifest remote
The -o option permits the user to control the name of the manifest's
remote, which normally is hardcoded to be 'origin', but can differ
because we derive it at runtime from the configuration file.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:50:52 -07:00
b3d2c9214b init (wrapper): Note that -m is now deprecated
If the manifest format isn't XML, this option isn't available.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:50:52 -07:00
7354d88914 init: Ensure repo.mirror is noticed once set
If we don't clear the cache, there can be a timestamp race between
the pickle file and the raw text file, and we may not pick up the
edit when we create a new config object around the same path name.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:50:52 -07:00
ce86abbe8a Allow the manifest to be accessed it if is in work tree
If the manifest's work tree is actually inside of the rest of
the client work tree then its only fair that we include it as
a project that the user can access.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:03:38 -07:00
75b87c8a51 Abstract manifest branch creation from init to the manifest object
This permits the XML style manifest to use 'default', while other
types can use their own creation strategy for the current branch.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:03:38 -07:00
abb7a3dfec Allow callers to request a specific type of manifest
If the caller knows exactly what the manifest type must be we
can now ask the loader to directly construct that type, rather
than guessing it from the working directory.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:03:38 -07:00
cc6c79643e Make refs/remotes/m management the manifest object's responsibility
I plan to have the new submodule manifest format use a different
layout for the m refs than the XML manifest format has used in
the past.  Thus we need to move the behavior management into the
manifest object, and out of the project, so we can change it.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:03:38 -07:00
2095179bee Cleanup import formatting in manifest_xml.py
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:03:38 -07:00
b0ca41e19a Only remove a stale pickle file if it exists
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 20:03:38 -07:00
1875ddd47c sync: Run git gc --auto after fetch
Users may wind up with a lot of loose object content in projects they
don't frequently make changes in, but that are modified by others.

Since we bypass many git code paths that would have otherwise called
out to `git gc --auto`, its possible for these projects to have
their loose object database grow out of control.  To help prevent
that, we now invoke it ourselves during the network half of sync.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 16:39:19 -07:00
446c4e5556 init: Allow -m only on XML formatted manifest
If the manifest is the newer SubmoduleManifest style, then the -m
option makes no sense, as you cannot select a specific file within
the current branch.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 11:00:16 -07:00
67f4563acb manifest: Only support -o option on XML formatted manifest
If the manifest isn't a single file format manifest, the -o option
makes no sense, as you cannot export multiple files to a single
stream for display or redirection.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 11:00:16 -07:00
050e4fd591 manifest: Only display XML help on XML manifest
Some of the help text is only related to the XML formatted manifest,
so only display that text if that is the current format.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 11:00:16 -07:00
60e679209a help: Don't show empty Summary or Description sections
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 11:00:16 -07:00
f1a6b14fdc Create an abstract Manifest base class
This will help as we add support for another manifest type.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 11:00:16 -07:00
ca3d8ff4fc Teach Project how to relink a .git/ in the work tree
The _LinkWorkTree method can now be used to relink the work tree,
such as if the real repository was moved to a different location
on disk.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 11:00:16 -07:00
98ea26b8d8 Allow callers to reset the git config cache
If commands modify the git config too rapidly we might not notice
the .git/config file has been modified, as they could run in the
same filesystem timestamp window and thus not cause a change on
the config's mtime.  This can cause repo to miss re-reading the
config file after running a command.

Allowing the cache to be flushed forces us to re-read the config.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 11:00:16 -07:00
20 changed files with 230 additions and 180 deletions

View File

@ -5,7 +5,7 @@ Short Version:
- Make sure all code is under the Apache License, 2.0.
- Publish your changes for review:
git push https://gerrit-review.googlesource.com/git-repo HEAD:refs/for/maint
git push https://gerrit-review.googlesource.com/git-repo HEAD:refs/for/master
Long Version:
@ -71,7 +71,7 @@ Push your patches over HTTPS to the review server, possibly through
a remembered remote to make this easier in the future:
git config remote.review.url https://gerrit-review.googlesource.com/git-repo
git config remote.review.push HEAD:refs/for/maint
git config remote.review.push HEAD:refs/for/master
git push review

View File

@ -17,7 +17,6 @@ import os
import sys
import pager
from git_config import GitConfig
COLORS = {None :-1,
'normal' :-1,

View File

@ -70,7 +70,7 @@ class Command(object):
groups = mp.config.GetString('manifest.groups')
if not groups:
groups = 'default,platform-' + platform.system().lower()
groups = 'all,-notdefault,platform-' + platform.system().lower()
groups = [x for x in re.split('[,\s]+', groups) if x]
if not args:

View File

@ -32,6 +32,7 @@ following DTD:
<!ELEMENT remote (EMPTY)>
<!ATTLIST remote name ID #REQUIRED>
<!ATTLIST remote alias CDATA #IMPLIED>
<!ATTLIST remote fetch CDATA #REQUIRED>
<!ATTLIST remote review CDATA #IMPLIED>
@ -89,6 +90,12 @@ name specified here is used as the remote name in each project's
.git/config, and is therefore automatically available to commands
like `git fetch`, `git remote`, `git pull` and `git push`.
Attribute `alias`: The alias, if specified, is used to override
`name` to be set as the remote name in each project's .git/config.
Its value can be duplicated while attribute `name` has to be unique
in the manifest file. This helps each project to be able to have
same remote name which actually points to different remote url.
Attribute `fetch`: The Git URL prefix for all projects which use
this remote. Each project's name is appended to this prefix to
form the actual URL used to clone the project.
@ -118,21 +125,27 @@ Element manifest-server
At most one manifest-server may be specified. The url attribute
is used to specify the URL of a manifest server, which is an
XML RPC service that will return a manifest in which each project
is pegged to a known good revision for the current branch and
target.
XML RPC service.
The manifest server should implement:
The manifest server should implement the following RPC methods:
GetApprovedManifest(branch, target)
Return a manifest in which each project is pegged to a known good revision
for the current branch and target.
The target to use is defined by environment variables TARGET_PRODUCT
and TARGET_BUILD_VARIANT. These variables are used to create a string
of the form $TARGET_PRODUCT-$TARGET_BUILD_VARIANT, e.g. passion-userdebug.
If one of those variables or both are not present, the program will call
GetApprovedManifest without the target paramater and the manifest server
GetApprovedManifest without the target parameter and the manifest server
should choose a reasonable default target.
GetManifest(tag)
Return a manifest in which each project is pegged to the revision at
the specified tag.
Element project
---------------
@ -149,7 +162,7 @@ URL to configure the Git remote with. The URL gets formed as:
where ${remote_fetch} is the remote's fetch attribute and
${project_name} is the project's name attribute. The suffix ".git"
is always appended as repo assumes the upstream is a forrest of
is always appended as repo assumes the upstream is a forest of
bare Git repositories.
The project name must match the name Gerrit knows, if Gerrit is
@ -171,7 +184,12 @@ the default element is used.
Attribute `groups`: List of groups to which this project belongs,
whitespace or comma separated. All projects belong to the group
"default".
"all", and each project automatically belongs to a group of
its name:`name` and path:`path`. E.g. for
<project name="monkeys" path="barrel-of"/>, that project
definition is implicitly in the following manifest groups:
default, name:monkeys, and path:barrel-of. If you place a project in the
group "notdefault", it will not be automatically downloaded by repo.
Element annotation
------------------

View File

@ -89,7 +89,7 @@ class _GitCall(object):
if ver_str.startswith('git version '):
_git_version = tuple(
map(lambda x: int(x),
ver_str[len('git version '):].strip().split('.')[0:3]
ver_str[len('git version '):].strip().split('-')[0].split('.')[0:3]
))
else:
print >>sys.stderr, 'fatal: "%s" unsupported' % ver_str

View File

@ -14,7 +14,6 @@
# limitations under the License.
import os
import sys
from trace import Trace
HEAD = 'HEAD'

View File

@ -35,7 +35,6 @@ from git_command import git, GitCommand
from git_config import init_ssh, close_ssh
from command import InteractiveCommand
from command import MirrorSafeCommand
from command import PagedCommand
from subcmds.version import Version
from editor import Editor
from error import DownloadError

View File

@ -20,7 +20,7 @@ import sys
import urlparse
import xml.dom.minidom
from git_config import GitConfig, IsId
from git_config import GitConfig
from project import RemoteSpec, Project, MetaProject, R_HEADS, HEAD
from error import ManifestParseError
@ -41,12 +41,14 @@ class _Default(object):
class _XmlRemote(object):
def __init__(self,
name,
alias=None,
fetch=None,
manifestUrl=None,
review=None):
self.name = name
self.fetchUrl = fetch
self.manifestUrl = manifestUrl
self.remoteAlias = alias
self.reviewUrl = review
self.resolvedFetchUrl = self._resolveFetchUrl()
@ -62,7 +64,10 @@ class _XmlRemote(object):
def ToRemoteSpec(self, projectName):
url = self.resolvedFetchUrl.rstrip('/') + '/' + projectName
return RemoteSpec(self.name, url, self.reviewUrl)
remoteName = self.name
if self.remoteAlias:
remoteName = self.remoteAlias
return RemoteSpec(remoteName, url, self.reviewUrl)
class XmlManifest(object):
"""manages the repo configuration file"""
@ -125,7 +130,7 @@ class XmlManifest(object):
groups = mp.config.GetString('manifest.groups')
if not groups:
groups = 'default'
groups = 'all'
groups = [x for x in re.split(r'[,\s]+', groups) if x]
doc = xml.dom.minidom.Document()
@ -206,7 +211,8 @@ class XmlManifest(object):
ce.setAttribute('dest', c.dest)
e.appendChild(ce)
egroups = [g for g in p.groups if g != 'default']
default_groups = ['all', 'name:%s' % p.name, 'path:%s' % p.relpath]
egroups = [g for g in p.groups if g not in default_groups]
if egroups:
e.setAttribute('groups', ','.join(egroups))
@ -303,12 +309,14 @@ class XmlManifest(object):
if not root or not root.childNodes:
raise ManifestParseError("no root node in %s" % (path,))
config = root.childNodes[0]
if config.nodeName != 'manifest':
for manifest in root.childNodes:
if manifest.nodeName == 'manifest':
break
else:
raise ManifestParseError("no <manifest> in %s" % (path,))
nodes = []
for node in config.childNodes:
for node in manifest.childNodes:
if node.nodeName == 'include':
name = self._reqatt(node, 'name')
fp = os.path.join(include_root, name)
@ -427,7 +435,7 @@ class XmlManifest(object):
if name is None:
s = m_url.rindex('/') + 1
manifestUrl = self.manifestProject.config.GetString('remote.origin.url')
remote = _XmlRemote('origin', m_url[:s], manifestUrl)
remote = _XmlRemote('origin', fetch=m_url[:s], manifestUrl=manifestUrl)
name = m_url[s:]
if name.endswith('.git'):
@ -451,12 +459,15 @@ class XmlManifest(object):
reads a <remote> element from the manifest file
"""
name = self._reqatt(node, 'name')
alias = node.getAttribute('alias')
if alias == '':
alias = None
fetch = self._reqatt(node, 'fetch')
review = node.getAttribute('review')
if review == '':
review = None
manifestUrl = self.manifestProject.config.GetString('remote.origin.url')
return _XmlRemote(name, fetch, manifestUrl, review)
return _XmlRemote(name, alias, fetch, manifestUrl, review)
def _ParseDefault(self, node):
"""
@ -566,8 +577,9 @@ class XmlManifest(object):
if node.hasAttribute('groups'):
groups = node.getAttribute('groups')
groups = [x for x in re.split('[,\s]+', groups) if x]
if 'default' not in groups:
groups.append('default')
default_groups = ['default', 'name:%s' % name, 'path:%s' % path]
groups.extend(set(default_groups).difference(groups))
if self.IsMirror:
relpath = None

View File

@ -20,32 +20,19 @@ import random
import re
import shutil
import stat
import subprocess
import sys
import time
import urllib2
try:
import threading as _threading
except ImportError:
import dummy_threading as _threading
try:
from os import SEEK_END
except ImportError:
SEEK_END = 2
from color import Coloring
from git_command import GitCommand
from git_config import GitConfig, IsId, GetSchemeFromUrl, ID_RE
from error import DownloadError
from error import GitError, HookError, ImportError, UploadError
from error import GitError, HookError, UploadError
from error import ManifestInvalidRevisionError
from progress import Progress
from trace import IsTrace, Trace
from git_refs import GitRefs, HEAD, R_HEADS, R_TAGS, R_PUB, R_M
_urllib_lock = _threading.Lock()
def _lwrite(path, content):
lock = '%s.lock' % path
@ -91,21 +78,6 @@ def _ProjectHooks():
_project_hook_list = map(lambda x: os.path.join(d, x), os.listdir(d))
return _project_hook_list
def relpath(dst, src):
src = os.path.dirname(src)
top = os.path.commonprefix([dst, src])
if top.endswith('/'):
top = top[:-1]
else:
top = os.path.dirname(top)
tmp = src
rel = ''
while top != tmp:
rel += '../'
tmp = os.path.dirname(tmp)
return rel + dst[len(top) + 1:]
class DownloadedChange(object):
_commit_cache = None
@ -176,10 +148,11 @@ class ReviewableBranch(object):
R_HEADS + self.name,
'--')
def UploadForReview(self, people, auto_topic=False):
def UploadForReview(self, people, auto_topic=False, draft=False):
self.project.UploadForReview(self.name,
people,
auto_topic=auto_topic)
auto_topic=auto_topic,
draft=draft)
def GetPublishedRefs(self):
refs = {}
@ -659,20 +632,21 @@ class Project(object):
"""Returns true if the manifest groups specified at init should cause
this project to be synced.
Prefixing a manifest group with "-" inverts the meaning of a group.
All projects are implicitly labelled with "default".
All projects are implicitly labelled with "all".
labels are resolved in order. In the example case of
project_groups: "default,group1,group2"
project_groups: "all,group1,group2"
manifest_groups: "-group1,group2"
the project will be matched.
"""
if self.groups is None:
return True
expanded_manifest_groups = manifest_groups or ['all', '-notdefault']
expanded_project_groups = ['all'] + (self.groups or [])
matched = False
for group in manifest_groups:
if group.startswith('-') and group[1:] in self.groups:
for group in expanded_manifest_groups:
if group.startswith('-') and group[1:] in expanded_project_groups:
matched = False
elif group in self.groups:
elif group in expanded_project_groups:
matched = True
return matched
@ -881,7 +855,8 @@ class Project(object):
def UploadForReview(self, branch=None,
people=([],[]),
auto_topic=False):
auto_topic=False,
draft=False):
"""Uploads the named branch for code review.
"""
if branch is None:
@ -920,7 +895,13 @@ class Project(object):
if dest_branch.startswith(R_HEADS):
dest_branch = dest_branch[len(R_HEADS):]
ref_spec = '%s:refs/for/%s' % (R_HEADS + branch.name, dest_branch)
upload_type = 'for'
if draft:
upload_type = 'drafts'
ref_spec = '%s:refs/%s/%s' % (R_HEADS + branch.name, upload_type,
dest_branch)
if auto_topic:
ref_spec = ref_spec + '/' + branch.name
cmd.append(ref_spec)
@ -1545,100 +1526,49 @@ class Project(object):
return ok
def _FetchBundle(self, srcUrl, tmpPath, dstPath, quiet):
keep = True
done = False
dest = open(tmpPath, 'a+b')
try:
dest.seek(0, SEEK_END)
pos = dest.tell()
_urllib_lock.acquire()
try:
req = urllib2.Request(srcUrl)
if pos > 0:
req.add_header('Range', 'bytes=%d-' % pos)
try:
r = urllib2.urlopen(req)
except urllib2.HTTPError, e:
def _content_type():
try:
return e.info()['content-type']
except:
return None
if e.code in (401, 403, 404):
keep = False
return False
elif _content_type() == 'text/plain':
try:
msg = e.read()
if len(msg) > 0 and msg[-1] == '\n':
msg = msg[0:-1]
msg = ' (%s)' % msg
except:
msg = ''
else:
try:
from BaseHTTPServer import BaseHTTPRequestHandler
res = BaseHTTPRequestHandler.responses[e.code]
msg = ' (%s: %s)' % (res[0], res[1])
except:
msg = ''
raise DownloadError('HTTP %s%s' % (e.code, msg))
except urllib2.URLError, e:
raise DownloadError('%s: %s ' % (req.get_host(), str(e)))
finally:
_urllib_lock.release()
p = None
try:
size = r.headers.get('content-length', 0)
unit = 1 << 10
if size and not quiet:
if size > 1024 * 1.3:
unit = 1 << 20
desc = 'MB'
else:
desc = 'KB'
p = Progress(
'Downloading %s' % self.relpath,
int(size) / unit,
units=desc)
if pos > 0:
p.update(pos / unit)
s = 0
while True:
d = r.read(8192)
if d == '':
done = True
return True
dest.write(d)
if p:
s += len(d)
if s >= unit:
p.update(s / unit)
s = s % unit
if p:
if s >= unit:
p.update(s / unit)
else:
p.update(1)
finally:
r.close()
if p:
p.end()
finally:
dest.close()
if os.path.exists(dstPath):
os.remove(dstPath)
if done:
os.rename(tmpPath, dstPath)
elif not keep:
cmd = ['curl', '--fail', '--output', tmpPath, '--netrc', '--location']
if quiet:
cmd += ['--silent']
if os.path.exists(tmpPath):
size = os.stat(tmpPath).st_size
if size >= 1024:
cmd += ['--continue-at', '%d' % (size,)]
else:
os.remove(tmpPath)
if 'http_proxy' in os.environ and 'darwin' == sys.platform:
cmd += ['--proxy', os.environ['http_proxy']]
cmd += [srcUrl]
if IsTrace():
Trace('%s', ' '.join(cmd))
try:
proc = subprocess.Popen(cmd)
except OSError:
return False
curlret = proc.wait()
if curlret == 22:
# From curl man page:
# 22: HTTP page not retrieved. The requested url was not found or
# returned another error with the HTTP error code being 400 or above.
# This return code only appears if -f, --fail is used.
if not quiet:
print >> sys.stderr, "Server does not provide clone.bundle; ignoring."
return False
if os.path.exists(tmpPath):
if curlret == 0 and os.stat(tmpPath).st_size > 16:
os.rename(tmpPath, dstPath)
return True
else:
os.remove(tmpPath)
return False
else:
return False
def _Checkout(self, rev, quiet=False):
cmd = ['checkout']
@ -1761,7 +1691,7 @@ class Project(object):
_error("%s: Not replacing %s hook", self.relpath, name)
continue
try:
os.symlink(relpath(stock_hook, dst), dst)
os.symlink(os.path.relpath(stock_hook, os.path.dirname(dst)), dst)
except OSError, e:
if e.errno == errno.EPERM:
raise GitError('filesystem must support symlinks')
@ -1822,7 +1752,7 @@ class Project(object):
src = os.path.join(self.gitdir, name)
dst = os.path.join(dotgit, name)
if os.path.islink(dst) or not os.path.exists(dst):
os.symlink(relpath(src, dst), dst)
os.symlink(os.path.relpath(src, os.path.dirname(dst)), dst)
else:
raise GitError('cannot overwrite a local work tree')
except OSError, e:
@ -2181,7 +2111,7 @@ class MetaProject(Project):
syncbuf.Finish()
return GitCommand(self,
['branch', '-D', 'default'],
['update-ref', '-d', 'refs/heads/default'],
capture_stdout = True,
capture_stderr = True).Wait() == 0

View File

@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import sys, re, string, random, os
import sys, re
from command import Command
from git_command import GitCommand

View File

@ -13,7 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import re
import sys

View File

@ -14,7 +14,6 @@
# limitations under the License.
import sys
from optparse import SUPPRESS_HELP
from color import Coloring
from command import PagedCommand
from git_command import git_require, GitCommand

View File

@ -81,7 +81,8 @@ to update the working directory files.
help='initial manifest file', metavar='NAME.xml')
g.add_option('--mirror',
dest='mirror', action='store_true',
help='mirror the forrest')
help='create a replica of the remote repositories '
'rather than a client working directory')
g.add_option('--reference',
dest='reference',
help='location of mirror directory', metavar='DIR')
@ -89,12 +90,12 @@ to update the working directory files.
dest='depth',
help='create a shallow clone with given depth; see git clone')
g.add_option('-g', '--groups',
dest='groups', default='default',
dest='groups', default='all,-notdefault',
help='restrict manifest projects to ones with a specified group',
metavar='GROUP')
g.add_option('-p', '--platform',
dest='platform', default='auto',
help='restrict manifest projects to ones with a specified'
help='restrict manifest projects to ones with a specified '
'platform group [auto|all|none|linux|darwin|...]',
metavar='PLATFORM')
@ -163,7 +164,7 @@ to update the working directory files.
groups = [x for x in groups if x]
groupstr = ','.join(groups)
if opt.platform == 'auto' and groupstr == 'default,platform-' + platform.system().lower():
if opt.platform == 'auto' and groupstr == 'all,-notdefault,platform-' + platform.system().lower():
groupstr = None
m.config.SetString('manifest.groups', groupstr)

View File

@ -50,6 +50,7 @@ in a Git repository for use during future 'repo init' invocations.
help='Save revisions as current HEAD')
p.add_option('-o', '--output-file',
dest='output_file',
default='-',
help='File to save the manifest to',
metavar='-|NAME.xml')

80
subcmds/overview.py Normal file
View File

@ -0,0 +1,80 @@
#
# Copyright (C) 2012 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from color import Coloring
from command import PagedCommand
class Overview(PagedCommand):
common = True
helpSummary = "Display overview of unmerged project branches"
helpUsage = """
%prog [--current-branch] [<project>...]
"""
helpDescription = """
The '%prog' command is used to display an overview of the projects branches,
and list any local commits that have not yet been merged into the project.
The -b/--current-branch option can be used to restrict the output to only
branches currently checked out in each project. By default, all branches
are displayed.
"""
def _Options(self, p):
p.add_option('-b', '--current-branch',
dest="current_branch", action="store_true",
help="Consider only checked out branches")
def Execute(self, opt, args):
all = []
for project in self.GetProjects(args):
br = [project.GetUploadableBranch(x)
for x in project.GetBranches().keys()]
br = [x for x in br if x]
if opt.current_branch:
br = [x for x in br if x.name == project.CurrentBranch]
all.extend(br)
if not all:
return
class Report(Coloring):
def __init__(self, config):
Coloring.__init__(self, config, 'status')
self.project = self.printer('header', attr='bold')
out = Report(all[0].project.config)
out.project('Projects Overview')
out.nl()
project = None
for branch in all:
if project != branch.project:
project = branch.project
out.nl()
out.project('project %s/' % project.relpath)
out.nl()
commits = branch.commits
date = branch.date
print '%s %-33s (%2d commit%s, %s)' % (
branch.name == project.CurrentBranch and '*' or ' ',
branch.name,
len(commits),
len(commits) != 1 and 's' or ' ',
date)
for commit in commits:
print '%-35s - %s' % ('', commit)

View File

@ -17,8 +17,6 @@ import sys
from command import Command
from git_command import GitCommand
from git_refs import GitRefs, HEAD, R_HEADS, R_TAGS, R_PUB, R_M
from error import GitError
class Rebase(Command):
common = True

View File

@ -15,6 +15,7 @@
import sys
from command import Command
from git_config import IsId
from git_command import git
from progress import Progress
@ -56,6 +57,10 @@ revision specified in the manifest.
pm = Progress('Starting %s' % nb, len(all))
for project in all:
pm.update()
# If the current revision is a specific SHA1 then we can't push back
# to it so substitute the manifest default revision instead.
if IsId(project.revisionExpr):
project.revisionExpr = self.manifest.default.revisionExpr
if not project.StartBranch(nb):
err.append(project)
pm.end()

View File

@ -122,6 +122,7 @@ the following meanings:
t = _threading.Thread(target=self._StatusHelper,
args=(project, counter, sem, output))
threads_and_output.append((t, output))
t.daemon = True
t.start()
for (t, output) in threads_and_output:
t.join()

View File

@ -37,13 +37,11 @@ except ImportError:
return (256, 256)
from git_command import GIT
from git_refs import R_HEADS
from project import HEAD
from git_refs import R_HEADS, HEAD
from project import Project
from project import RemoteSpec
from command import Command, MirrorSafeCommand
from error import RepoChangedException, GitError
from project import R_HEADS
from project import SyncBuffer
from progress import Progress
@ -230,8 +228,10 @@ later is required to fix a server side protocol bug.
if self.jobs == 1:
for project in projects:
pm.update()
if project.Sync_NetworkHalf(quiet=opt.quiet,
current_branch_only=opt.current_branch_only):
if project.Sync_NetworkHalf(
quiet=opt.quiet,
current_branch_only=opt.current_branch_only,
clone_bundle=not opt.no_clone_bundle):
fetched.add(project.gitdir)
else:
print >>sys.stderr, 'error: Cannot fetch %s' % project.name
@ -259,6 +259,8 @@ later is required to fix a server side protocol bug.
pm,
sem,
err_event))
# Ensure that Ctrl-C will not freeze the repo process.
t.daemon = True
threads.add(t)
t.start()
@ -402,9 +404,13 @@ uncommitted changes are present' % project.relpath
else:
print >>sys.stderr, 'error: %s' % manifest_str
sys.exit(1)
except socket.error:
print >>sys.stderr, 'error: cannot connect to manifest server %s' % (
self.manifest.manifest_server)
except (socket.error, IOError, xmlrpclib.Fault), e:
print >>sys.stderr, 'error: cannot connect to manifest server %s:\n%s' % (
self.manifest.manifest_server, e)
sys.exit(1)
except xmlrpclib.ProtocolError, e:
print >>sys.stderr, 'error: cannot connect to manifest server %s:\n%d %s' % (
self.manifest.manifest_server, e.errcode, e.errmsg)
sys.exit(1)
rp = self.manifest.repoProject

View File

@ -134,6 +134,9 @@ Gerrit Code Review: http://code.google.com/p/gerrit/
p.add_option('--cbr', '--current-branch',
dest='current_branch', action='store_true',
help='Upload current git branch.')
p.add_option('-d', '--draft',
action='store_true', dest='draft', default=False,
help='If specified, upload as a draft.')
# Options relating to upload hook. Note that verify and no-verify are NOT
# opposites of each other, which is why they store to different locations.
@ -324,7 +327,7 @@ Gerrit Code Review: http://code.google.com/p/gerrit/
key = 'review.%s.uploadtopic' % branch.project.remote.review
opt.auto_topic = branch.project.config.GetBoolean(key)
branch.UploadForReview(people, auto_topic=opt.auto_topic)
branch.UploadForReview(people, auto_topic=opt.auto_topic, draft=opt.draft)
branch.uploaded = True
except UploadError, e:
branch.error = e