Commit Graph

1346 Commits

Author SHA1 Message Date
David Pursehouse
2d5a0df798 Add support for multiple local manifests
Add support for multiple local manifests stored in the local_manifests
folder under the .repo home directory.

Local manifests will be processed in addition to local_manifest.xml.

Change-Id: Ia0569cea7e9ae0fe3208a8ffef5d9679e14db03b
2012-11-13 08:19:51 +09:00
David Pursehouse
f7fc8a95be Handle XML errors when parsing the manifest
Catch ExpatError and exit gracefully with an error message, rather
than exiting with a python traceback.

Change-Id: Ifd0a7762aab4e8de63dab8a66117170a05586866
2012-11-13 05:53:41 +09:00
Conley Owens
1ad7b555df Merge "Always show --manifest-server-* options" 2012-11-07 12:39:25 -08:00
David Pursehouse
7e6dd2dff0 Fix pylint warning W0108: Lambda may not be necessary
Remove unnecessary usage of lambda.

Change-Id: I06d41933057d60d15d307ee800cca052a44754c6
2012-11-07 08:39:57 +09:00
Conley Owens
8d070cfb25 Always show --manifest-server-* options
The --manifest-server-* flags broke the smartsync subcmd since
the corresponding variables weren't getting set.  This change
ensures that they will always be set, regardless of whether we are
using sync -s or smartsync.

Change-Id: I1b642038787f2114fa812ecbc15c64e431bbb829
2012-11-06 13:14:31 -08:00
Sarah Owens
a6053d54f1 Change usages of xrange() to range()
In Python3, range() creates a generator rather than a list.

None of the parameters in the ranges changed looked large enough
to create an impact in memory in Python2.  Note: the only use of
range() was for iteration and did not need to be changed.

This is part of a series of changes to introduce Python3 support.

Change-Id: I50b665f9296ea160a5076c71f36a65f76e47029f
2012-11-01 13:36:50 -07:00
Conley Owens
e072a92a9b Merge "Use python3 urllib when urllib2 not available" 2012-11-01 10:13:34 -07:00
Conley Owens
7601ee2608 Merge "Use 'stat' package instead of literals for mkdir()" 2012-11-01 10:01:18 -07:00
Sarah Owens
1f7627fd3c Use python3 urllib when urllib2 not available
This is part of a series of changes to introduce Python3 support.

Change-Id: I605b145791053c1f2d7bf3c907c5a68649b21d12
2012-10-31 14:26:48 -07:00
Dave Borowitz
b42b4746af project: Require git >= 1.7.2 for setting config on command line
This option causes the git call to fail, which probably indicates a
programming error; callers should check the git version and change the
call appropriately if -c is not available. Failing loudly is preferable
to failing silently in the general case.

For an example of correctly checking at the call site, see I8fd313dd.
If callers prefer to fail silently, they may set GIT_CONFIG_PARAMETERS
in the environment rather than using the config kwarg to pass
configuration.

Change-Id: I0de18153d44d3225cd3031e6ead54461430ed334
2012-10-31 12:27:27 -07:00
Dave Borowitz
e21526754b sync: Only parallelize gc for git >= 1.7.2
This minimum version is required for the -c argument to set config on
the command line. Without this option, git by default uses as many
threads per invocation as there are CPUs, so we cannot safely
parallelize without hosing a system.

Change-Id: I8fd313dd84917658162b5134b2d9aa34a96f2772
2012-10-31 12:27:17 -07:00
Sarah Owens
60798a32f6 Use 'stat' package instead of literals for mkdir()
This is part of a series of changes to introduce Python3 support.

Change-Id: Ic988ad181d32357d82dfa554e70d8525118334c0
2012-10-31 09:11:16 -07:00
David Pursehouse
1d947b3034 Even more coding style cleanup
Fixing some more pylint warnings:

W1401: Anomalous backslash in string
W0623: Redefining name 'name' from outer scope
W0702: No exception type(s) specified
E0102: name: function already defined line n

Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
2012-10-30 10:28:20 +09:00
Shawn Pearce
2d113f3546 Merge "Update minimum git version to 1.7.2" 2012-10-26 16:10:21 -07:00
Shawn O. Pearce
de7eae4826 Merge "Revert "Represent git-submodule as nested projects"" 2012-10-26 12:30:38 -07:00
Shawn O. Pearce
2fe99e8820 Merge "repo selfupdate: Fix _PostRepoUpgrade takes 2 arguments" 2012-10-26 12:27:36 -07:00
Shawn O. Pearce
cd81dd6403 Revert "Represent git-submodule as nested projects"
This reverts commit 69998b0c6f.
Broke Android's non-gitmodule use case.

Conflicts:
	project.py
	subcmds/sync.py

Change-Id: I68ceeb63d8ee3b939f85a64736bdc81dfa352aed
2012-10-26 12:24:57 -07:00
Shawn O. Pearce
80d2ceb222 repo selfupdate: Fix _PostRepoUpgrade takes 2 arguments
Change-Id: I1cf9e0674ea366ddce96c949e0bc085e3452b25a
2012-10-26 12:24:57 -07:00
Conley Owens
c5aa4d3528 Update minimum git version to 1.7.2
We now use the -c flag which was introduced in git 1.7.2.

Change-Id: I9195c0f6ac9fa63e783a03628049fe2c67d258ff
2012-10-26 11:34:11 -07:00
Conley Owens
bed45f9400 Merge "Show user about not initializing repo in current directory" 2012-10-26 09:52:16 -07:00
Conley Owens
55e4d464a7 Add a PGP key for cco3@android.com
This change adds a PGP key to allow cco3@android.com to sign releases.

Change-Id: I18a70c8b7d8f272dd1aad9d6b2e4a237ef35af33
2012-10-26 07:03:59 -07:00
Yang Zhenhui
75cc353380 Show user about not initializing repo in current directory
If the parent of current directory has an initialized repo,
for example, if the current directory is
'/home/users/harry/platform/ics', and there is an initialized repo
in harry's home directory '/home/users/harry/.repo', when user
run 'repo init' command, repo is always initialized to parent
directory in '/home/users/harry/.repo', but most of time user
intends to initialize repo in the current directory, this patch
tells user how to do it.

Change-Id: Id7a76fb18ec0af243432c29605140d60f3de85ca
2012-10-26 15:40:17 +08:00
Conley Owens
c9129d90de Update PGP keys during _PostRepoUpgrade in sync
Previously, if a key was added, a client wouldn't add the key during
the sync step.  This would cause issues if a new key were added and a
subsequent release were signed by that key.

Change-Id: I4fac317573cd9d0e8da62aa42e00faf08bfeb26c
2012-10-25 17:48:35 -07:00
Shawn Pearce
57365c98cc Merge "sync: Run gc --auto in parallel" 2012-10-25 17:38:05 -07:00
Shawn Pearce
dc96476af3 Merge "project: Support config args in git command callables" 2012-10-25 17:36:04 -07:00
Shawn Pearce
2577cec095 Merge "sync: Keep a moving average of last fetch times" 2012-10-25 17:35:15 -07:00
Shawn Pearce
e48d34659e Merge "sync: Order projects according to last fetch time" 2012-10-25 17:33:36 -07:00
Che-Liang Chiou
ab8f911a67 Fix pylint warnings introduced by the submodule patch
"69998b0 Represent git-submodule as nested projects" has introduced a
few pylint warnings.

W0612:1439,8:Project._GetSubmodules.get_submodules: Unused variable 'sub_gitdir'
W0613:1424,36:Project._GetSubmodules.get_submodules: Unused argument 'path'
W0612:1450,25:Project._GetSubmodules.parse_gitmodules: Unused variable 'e'
W0622:516,8:Sync.Execute: Redefining built-in 'all'

Change-Id: I84378e2832ed1b5ab023e394d53b22dcea799ba4
2012-10-25 13:55:49 -07:00
Conley Owens
608aff7f62 Merge "Use modern Python exception syntax" 2012-10-25 10:03:37 -07:00
Conley Owens
13657c407d Merge "Add regex matching to repo list command" 2012-10-25 10:00:42 -07:00
Conley Owens
e4ed8f65f3 Merge "Add pylint configuration and instructions" 2012-10-25 09:51:07 -07:00
Conley Owens
fdb44479f8 Merge "Change PyDev project version to "python 2.6"" 2012-10-25 09:46:38 -07:00
Dave Borowitz
188572170e sync: Run gc --auto in parallel
We can't just let this run wild with a high (or even low) -j, since
that would hose a system. Instead, limit the total number of threads
across all git gc subprocesses to the number of CPUs reported by the
multiprocessing module (available in Python 2.6 and above).

Change-Id: Icca0161a1e6116ffa5f7cfc6f5faecda510a7fb9
2012-10-25 08:12:48 -07:00
Olof Johansson
d75c669fac Add regex matching to repo list command
The repo list -r command will execute a regex search for every
argument provided on both the project name and the project
worktree path.

Useful for finding rarely used gits.

Change-Id: Iaff90dd36c240b3d5d74817d11469be22d77ae03
2012-10-25 15:49:13 +09:00
Dave Borowitz
091f893625 project: Support config args in git command callables
Change-Id: I9d4d0d2b1aeebe41a6b24a339a154d258af665eb
2012-10-24 14:52:08 -07:00
Dave Borowitz
d947858325 sync: Keep a moving average of last fetch times
Try to more accurately estimate which projects take the longest to
sync by keeping an exponentially weighted moving average (a=0.5) of
fetch times, rather than just recording the last observation. This
should discount individual outliers (e.g. an unusually large project
update) and hopefully allow truly slow repos to bubble to the top.

Change-Id: I72b2508cb1266e8a19cf15b616d8a7fc08098cb3
2012-10-24 14:52:07 -07:00
Dave Borowitz
67700e9b90 sync: Order projects according to last fetch time
Some projects may consistently take longer to fetch than others, for
example a more active project may have many more Gerrit changes than a
less active project, which take longer to transfer. Use a simple
heuristic based on the last fetch time to fetch slower projects first,
so we do not tend to spend the end of the sync fetching a small number
of outliers.

This algorithm is probably not optimal, and due to inter-run latency
variance and Python thread scheduling, we may not even have good
estimates of a project sync time.

Change-Id: I9a463f214b3ed742e4d807c42925b62cb8b1745b
2012-10-24 14:51:58 -07:00
Sarah Owens
a5be53f9c8 Use modern Python exception syntax
"except Exception as e" instead of "except Exception, e"

This is part of a transition to supporting Python 3.  Python >= 2.6
support "as" syntax.

Note: this removes Python 2.5 support.

Change-Id: I309599f3981bba2b46111c43102bee38ff132803
2012-10-23 21:35:59 -07:00
Sarah Owens
9ed12c5d9c Change PyDev project version to "python 2.6"
Repo is dropping support for Python <2.5 soon, so this updates the
PyDev configuration appropriately.

Change-Id: If327951e3a9fd9ff7513b931bfcfe6172dc8e4c5
2012-10-23 21:35:46 -07:00
David Pursehouse
4f7bdea9d2 Add pylint configuration and instructions
pylint configuration file (.pylintrc) is added, and submission
instructions are updated to include pylint usage steps.

Deprecated pylint suppression (`disable-msg`) is updated in a few
modules to make it work properly with the latest version (0.26).

Change-Id: I4ec2ef318e23557a374ecdbf40fe12645766830c
2012-10-24 10:18:13 +09:00
Che-Liang Chiou
69998b0c6f Represent git-submodule as nested projects
We need a representation of git-submodule in repo; otherwise repo will
not sync submodules, and leave workspace in a broken state.  Of course
this will not be a problem if all projects are owned by the owner of the
manifest file, who may simply choose not to use git-submodule in all
projects.  However, this is not possible in practice because manifest
file owner is unlikely to own all upstream projects.

As git submodules are simply git repositories, it is natural to treat
them as plain repo projects that live inside a repo project.  That is,
we could use recursively declared projects to denote the is-submodule
relation of git repositories.

The behavior of repo remains the same to projects that do not have a
sub-project within.  As for parent projects, repo fetches them and their
sub-projects as normal projects, and then checks out subprojects at the
commit specified in parent's commit object.  The sub-project is fetched
at a path relative to parent project's working directory; so the path
specified in manifest file should match that of .gitmodules file.

If a submodule is not registered in repo manifest, repo will derive its
properties from itself and its parent project, which might not always be
correct.  In such cases, the subproject is called a derived subproject.

To a user, a sub-project is merely a git-submodule; so all tips of
working with a git-submodule apply here, too.  For example, you should
not run `repo sync` in a parent repository if its submodule is dirty.

Change-Id: I541e9e2ac1a70304272dbe09724572aa1004eb5c
2012-10-23 16:08:58 -07:00
David Pursehouse
5c6eeac8f0 More coding style cleanup
Fixing more issues found with pylint.  Some that were supposed to
have been fixed in the previous sweep (Ie0db839e) but were missed:

C0321: More than one statement on a single line
W0622: Redefining built-in 'name'

And some more:

W0631: Using possibly undefined loop variable 'name'
W0223: Method 'name' is abstract in class 'name' but is not overridden
W0231: __init__ method from base class 'name' is not called

Change-Id: Ie119183708609d6279e973057a385fde864230c3
2012-10-22 12:30:14 +09:00
JoonCheol Park
e98607248e Support HTTP authentication using user input as fallback
If repo could not find authentication credentials from ~/.netrc, this
patch tries to get user and password from user's console input. This
could be a good choice if user doesn't want to save his plain password
in ~/.netrc or if user doesn't know about the netrc usage.

The user will be prompted only if authentication infomation does not
exist in the password manager. Since main.py firstly loads auth
infomation from ~/.netrc, this will be executed only as fallback
mechanism.

Example:
$ repo upload .
Upload project xxx/ to remote branch master:
 branch yyy ( 1 commit, ...):
 to https://review.zzz.com/gerrit/ (y/N)? y

(repo may try to access to https://review.zzz.com/gerrit/ssh_info and
will get the 401 HTTP Basic Authentication response from server. If no
authentication info in ~/.netrc, This patch will ask username/passwd)

Authorization Required (Message from Web Server)
User: pororo
Password:
....
[OK ] xxx/

Change-Id: Ia348a4609ac40060d9093c7dc8d7c2560020455a
2012-10-12 06:02:35 +09:00
Mickaël Salaün
2f6ab7f5b8 Rename "dir" variables
The variable name "dir" conflicts with the name of a Python built-in
function: http://docs.python.org/library/functions.html#dir

Change-Id: I850f3ec8df7563dc85e21f2876fe5e6550ca2d8f
2012-10-10 08:30:15 +02:00
Conley Owens
3a6cd4200e Merge "Coding style cleanup" 2012-10-09 14:29:46 -07:00
Conley Owens
25f17682ca Merge "Expand ~ to user's home directory for --reference" 2012-10-09 13:46:10 -07:00
David Pursehouse
8a68ff9605 Coding style cleanup
Fix the following issues reported by pylint:

C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
W0612: Unused variable 'name'
W0613: Unused argument 'name'
W0102: Dangerous default value 'value' as argument
W0105: String statement has no effect

Also fixed a few cases of inconsistent indentation.

Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
2012-10-09 12:45:30 +02:00
Victor Boivie
297e7c6ee6 Expand ~ to user's home directory for --reference
This allows a user to have a 'repo init' as:
  $ repo init -u ... --reference=~/mirror

Change-Id: Ib85b7c8ffca9d732132c68fe9a8d7f0ab1fa9288
2012-10-08 15:03:20 +02:00
David Pursehouse
e3b1c45aeb Remove unreachable code
Change 9bb1816b removed part of a block of code, but left the
remaining part unreachable.  Remove it.

Change-Id: Icdc6061d00e6027df32dee9a3bad3999fe7cdcbc
2012-10-05 10:34:19 +02:00
David Pursehouse
7119f94aba Update commit-msg hook to version from Gerrit v2.5-rc0
Change-Id: I0d11ac0c24cd53386e996b7dd9bd37c89c789f60
2012-10-04 10:31:09 +02:00