Commit Graph

1904 Commits

Author SHA1 Message Date
Conley Owens
8e3d355d44 Merge "Print an error message when aborted by user" 2012-11-12 17:35:47 -08:00
Conley Owens
4a4776e9ab Merge "Handle manifest parse errors in main" 2012-11-12 17:35:40 -08:00
Conley Owens
2fa715f8b5 Merge "Better handling of duplicate remotes" 2012-11-12 17:35:30 -08:00
Conley Owens
6287543e35 Merge "Change usages of xrange() to range()" 2012-11-12 17:30:55 -08:00
David Pursehouse
b0936b0e20 Print an error message when aborted by user
Change-Id: If7378c5deaace0ac6ab2be961e38644d9373557d
2012-11-13 09:56:16 +09:00
David Pursehouse
0b8df7be79 Handle manifest parse errors in main
Add handling of manifest parse errors in the main method, and
print an error.  This will prevent python tracebacks being
dumped in many cases.

Change-Id: I75e73539afa34049f73c993dbfda203f1ad33b45
2012-11-13 09:54:47 +09:00
David Pursehouse
717ece9d81 Better handling of duplicate remotes
In the current implementation, an error is raised if a remote with the
same name is defined more than once.  The check is only that the remote
has the same name as an existing remote.

With the support for multiple local manifests, it is more likely than
before that the same remote is defined in more than one manifest.

Change the check so that it only raises an error if a remote is defined
more than once with the same name, but different attributes.

Change-Id: Ic3608646cf9f40aa2bea7015d3ecd099c5f5f835
2012-11-13 09:35:37 +09:00
David Pursehouse
5566ae5dde Print deprecation warning when local_manifest.xml is used
The preferred way to specify local manifests is to drop the file(s)
in the local_manifests folder.  Print a deprecation warning when
the legacy local_manifest.xml file is used.

Change-Id: Ice85bd06fb612d6fcceeaa0755efd130556c4464
2012-11-13 08:19:51 +09:00
David Pursehouse
2d5a0df798 Add support for multiple local manifests
Add support for multiple local manifests stored in the local_manifests
folder under the .repo home directory.

Local manifests will be processed in addition to local_manifest.xml.

Change-Id: Ia0569cea7e9ae0fe3208a8ffef5d9679e14db03b
2012-11-13 08:19:51 +09:00
David Pursehouse
f7fc8a95be Handle XML errors when parsing the manifest
Catch ExpatError and exit gracefully with an error message, rather
than exiting with a python traceback.

Change-Id: Ifd0a7762aab4e8de63dab8a66117170a05586866
2012-11-13 05:53:41 +09:00
Conley Owens
1ad7b555df Merge "Always show --manifest-server-* options" 2012-11-07 12:39:25 -08:00
David Pursehouse
7e6dd2dff0 Fix pylint warning W0108: Lambda may not be necessary
Remove unnecessary usage of lambda.

Change-Id: I06d41933057d60d15d307ee800cca052a44754c6
2012-11-07 08:39:57 +09:00
Conley Owens
8d070cfb25 Always show --manifest-server-* options
The --manifest-server-* flags broke the smartsync subcmd since
the corresponding variables weren't getting set.  This change
ensures that they will always be set, regardless of whether we are
using sync -s or smartsync.

Change-Id: I1b642038787f2114fa812ecbc15c64e431bbb829
2012-11-06 13:14:31 -08:00
Sarah Owens
a6053d54f1 Change usages of xrange() to range()
In Python3, range() creates a generator rather than a list.

None of the parameters in the ranges changed looked large enough
to create an impact in memory in Python2.  Note: the only use of
range() was for iteration and did not need to be changed.

This is part of a series of changes to introduce Python3 support.

Change-Id: I50b665f9296ea160a5076c71f36a65f76e47029f
2012-11-01 13:36:50 -07:00
Conley Owens
e072a92a9b Merge "Use python3 urllib when urllib2 not available" 2012-11-01 10:13:34 -07:00
Conley Owens
7601ee2608 Merge "Use 'stat' package instead of literals for mkdir()" 2012-11-01 10:01:18 -07:00
Sarah Owens
1f7627fd3c Use python3 urllib when urllib2 not available
This is part of a series of changes to introduce Python3 support.

Change-Id: I605b145791053c1f2d7bf3c907c5a68649b21d12
2012-10-31 14:26:48 -07:00
Dave Borowitz
b42b4746af project: Require git >= 1.7.2 for setting config on command line
This option causes the git call to fail, which probably indicates a
programming error; callers should check the git version and change the
call appropriately if -c is not available. Failing loudly is preferable
to failing silently in the general case.

For an example of correctly checking at the call site, see I8fd313dd.
If callers prefer to fail silently, they may set GIT_CONFIG_PARAMETERS
in the environment rather than using the config kwarg to pass
configuration.

Change-Id: I0de18153d44d3225cd3031e6ead54461430ed334
2012-10-31 12:27:27 -07:00
Dave Borowitz
e21526754b sync: Only parallelize gc for git >= 1.7.2
This minimum version is required for the -c argument to set config on
the command line. Without this option, git by default uses as many
threads per invocation as there are CPUs, so we cannot safely
parallelize without hosing a system.

Change-Id: I8fd313dd84917658162b5134b2d9aa34a96f2772
2012-10-31 12:27:17 -07:00
Sarah Owens
60798a32f6 Use 'stat' package instead of literals for mkdir()
This is part of a series of changes to introduce Python3 support.

Change-Id: Ic988ad181d32357d82dfa554e70d8525118334c0
2012-10-31 09:11:16 -07:00
David Pursehouse
1d947b3034 Even more coding style cleanup
Fixing some more pylint warnings:

W1401: Anomalous backslash in string
W0623: Redefining name 'name' from outer scope
W0702: No exception type(s) specified
E0102: name: function already defined line n

Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
2012-10-30 10:28:20 +09:00
Shawn Pearce
2d113f3546 Merge "Update minimum git version to 1.7.2" 2012-10-26 16:10:21 -07:00
Shawn O. Pearce
de7eae4826 Merge "Revert "Represent git-submodule as nested projects"" 2012-10-26 12:30:38 -07:00
Shawn O. Pearce
2fe99e8820 Merge "repo selfupdate: Fix _PostRepoUpgrade takes 2 arguments" 2012-10-26 12:27:36 -07:00
Shawn O. Pearce
cd81dd6403 Revert "Represent git-submodule as nested projects"
This reverts commit 69998b0c6f.
Broke Android's non-gitmodule use case.

Conflicts:
	project.py
	subcmds/sync.py

Change-Id: I68ceeb63d8ee3b939f85a64736bdc81dfa352aed
2012-10-26 12:24:57 -07:00
Shawn O. Pearce
80d2ceb222 repo selfupdate: Fix _PostRepoUpgrade takes 2 arguments
Change-Id: I1cf9e0674ea366ddce96c949e0bc085e3452b25a
2012-10-26 12:24:57 -07:00
Conley Owens
c5aa4d3528 Update minimum git version to 1.7.2
We now use the -c flag which was introduced in git 1.7.2.

Change-Id: I9195c0f6ac9fa63e783a03628049fe2c67d258ff
2012-10-26 11:34:11 -07:00
Conley Owens
bed45f9400 Merge "Show user about not initializing repo in current directory" 2012-10-26 09:52:16 -07:00
Conley Owens
55e4d464a7 Add a PGP key for cco3@android.com
This change adds a PGP key to allow cco3@android.com to sign releases.

Change-Id: I18a70c8b7d8f272dd1aad9d6b2e4a237ef35af33
2012-10-26 07:03:59 -07:00
Yang Zhenhui
75cc353380 Show user about not initializing repo in current directory
If the parent of current directory has an initialized repo,
for example, if the current directory is
'/home/users/harry/platform/ics', and there is an initialized repo
in harry's home directory '/home/users/harry/.repo', when user
run 'repo init' command, repo is always initialized to parent
directory in '/home/users/harry/.repo', but most of time user
intends to initialize repo in the current directory, this patch
tells user how to do it.

Change-Id: Id7a76fb18ec0af243432c29605140d60f3de85ca
2012-10-26 15:40:17 +08:00
Conley Owens
c9129d90de Update PGP keys during _PostRepoUpgrade in sync
Previously, if a key was added, a client wouldn't add the key during
the sync step.  This would cause issues if a new key were added and a
subsequent release were signed by that key.

Change-Id: I4fac317573cd9d0e8da62aa42e00faf08bfeb26c
2012-10-25 17:48:35 -07:00
Shawn Pearce
57365c98cc Merge "sync: Run gc --auto in parallel" 2012-10-25 17:38:05 -07:00
Shawn Pearce
dc96476af3 Merge "project: Support config args in git command callables" 2012-10-25 17:36:04 -07:00
Shawn Pearce
2577cec095 Merge "sync: Keep a moving average of last fetch times" 2012-10-25 17:35:15 -07:00
Shawn Pearce
e48d34659e Merge "sync: Order projects according to last fetch time" 2012-10-25 17:33:36 -07:00
Che-Liang Chiou
ab8f911a67 Fix pylint warnings introduced by the submodule patch
"69998b0 Represent git-submodule as nested projects" has introduced a
few pylint warnings.

W0612:1439,8:Project._GetSubmodules.get_submodules: Unused variable 'sub_gitdir'
W0613:1424,36:Project._GetSubmodules.get_submodules: Unused argument 'path'
W0612:1450,25:Project._GetSubmodules.parse_gitmodules: Unused variable 'e'
W0622:516,8:Sync.Execute: Redefining built-in 'all'

Change-Id: I84378e2832ed1b5ab023e394d53b22dcea799ba4
2012-10-25 13:55:49 -07:00
Conley Owens
608aff7f62 Merge "Use modern Python exception syntax" 2012-10-25 10:03:37 -07:00
Conley Owens
13657c407d Merge "Add regex matching to repo list command" 2012-10-25 10:00:42 -07:00
Conley Owens
e4ed8f65f3 Merge "Add pylint configuration and instructions" 2012-10-25 09:51:07 -07:00
Conley Owens
fdb44479f8 Merge "Change PyDev project version to "python 2.6"" 2012-10-25 09:46:38 -07:00
Dave Borowitz
188572170e sync: Run gc --auto in parallel
We can't just let this run wild with a high (or even low) -j, since
that would hose a system. Instead, limit the total number of threads
across all git gc subprocesses to the number of CPUs reported by the
multiprocessing module (available in Python 2.6 and above).

Change-Id: Icca0161a1e6116ffa5f7cfc6f5faecda510a7fb9
2012-10-25 08:12:48 -07:00
Olof Johansson
d75c669fac Add regex matching to repo list command
The repo list -r command will execute a regex search for every
argument provided on both the project name and the project
worktree path.

Useful for finding rarely used gits.

Change-Id: Iaff90dd36c240b3d5d74817d11469be22d77ae03
2012-10-25 15:49:13 +09:00
Dave Borowitz
091f893625 project: Support config args in git command callables
Change-Id: I9d4d0d2b1aeebe41a6b24a339a154d258af665eb
2012-10-24 14:52:08 -07:00
Dave Borowitz
d947858325 sync: Keep a moving average of last fetch times
Try to more accurately estimate which projects take the longest to
sync by keeping an exponentially weighted moving average (a=0.5) of
fetch times, rather than just recording the last observation. This
should discount individual outliers (e.g. an unusually large project
update) and hopefully allow truly slow repos to bubble to the top.

Change-Id: I72b2508cb1266e8a19cf15b616d8a7fc08098cb3
2012-10-24 14:52:07 -07:00
Dave Borowitz
67700e9b90 sync: Order projects according to last fetch time
Some projects may consistently take longer to fetch than others, for
example a more active project may have many more Gerrit changes than a
less active project, which take longer to transfer. Use a simple
heuristic based on the last fetch time to fetch slower projects first,
so we do not tend to spend the end of the sync fetching a small number
of outliers.

This algorithm is probably not optimal, and due to inter-run latency
variance and Python thread scheduling, we may not even have good
estimates of a project sync time.

Change-Id: I9a463f214b3ed742e4d807c42925b62cb8b1745b
2012-10-24 14:51:58 -07:00
Sarah Owens
a5be53f9c8 Use modern Python exception syntax
"except Exception as e" instead of "except Exception, e"

This is part of a transition to supporting Python 3.  Python >= 2.6
support "as" syntax.

Note: this removes Python 2.5 support.

Change-Id: I309599f3981bba2b46111c43102bee38ff132803
2012-10-23 21:35:59 -07:00
Sarah Owens
9ed12c5d9c Change PyDev project version to "python 2.6"
Repo is dropping support for Python <2.5 soon, so this updates the
PyDev configuration appropriately.

Change-Id: If327951e3a9fd9ff7513b931bfcfe6172dc8e4c5
2012-10-23 21:35:46 -07:00
David Pursehouse
4f7bdea9d2 Add pylint configuration and instructions
pylint configuration file (.pylintrc) is added, and submission
instructions are updated to include pylint usage steps.

Deprecated pylint suppression (`disable-msg`) is updated in a few
modules to make it work properly with the latest version (0.26).

Change-Id: I4ec2ef318e23557a374ecdbf40fe12645766830c
2012-10-24 10:18:13 +09:00
Che-Liang Chiou
69998b0c6f Represent git-submodule as nested projects
We need a representation of git-submodule in repo; otherwise repo will
not sync submodules, and leave workspace in a broken state.  Of course
this will not be a problem if all projects are owned by the owner of the
manifest file, who may simply choose not to use git-submodule in all
projects.  However, this is not possible in practice because manifest
file owner is unlikely to own all upstream projects.

As git submodules are simply git repositories, it is natural to treat
them as plain repo projects that live inside a repo project.  That is,
we could use recursively declared projects to denote the is-submodule
relation of git repositories.

The behavior of repo remains the same to projects that do not have a
sub-project within.  As for parent projects, repo fetches them and their
sub-projects as normal projects, and then checks out subprojects at the
commit specified in parent's commit object.  The sub-project is fetched
at a path relative to parent project's working directory; so the path
specified in manifest file should match that of .gitmodules file.

If a submodule is not registered in repo manifest, repo will derive its
properties from itself and its parent project, which might not always be
correct.  In such cases, the subproject is called a derived subproject.

To a user, a sub-project is merely a git-submodule; so all tips of
working with a git-submodule apply here, too.  For example, you should
not run `repo sync` in a parent repository if its submodule is dirty.

Change-Id: I541e9e2ac1a70304272dbe09724572aa1004eb5c
2012-10-23 16:08:58 -07:00
David Pursehouse
5c6eeac8f0 More coding style cleanup
Fixing more issues found with pylint.  Some that were supposed to
have been fixed in the previous sweep (Ie0db839e) but were missed:

C0321: More than one statement on a single line
W0622: Redefining built-in 'name'

And some more:

W0631: Using possibly undefined loop variable 'name'
W0223: Method 'name' is abstract in class 'name' but is not overridden
W0231: __init__ method from base class 'name' is not called

Change-Id: Ie119183708609d6279e973057a385fde864230c3
2012-10-22 12:30:14 +09:00