Commit Graph

155 Commits

Author SHA1 Message Date
Simran Basi
bdb5271de3 GITC: Add repo sync support.
Add repo sync support for GITC checkouts. If the user is in the
GITC client directory they can still pull the sources as normal
if they pass in the --force-gitc argument. Otherwise the user
should call repo sync in the GITC view to update the user's
remote view. (This works because .repo in the GITC view will
link to .repo in the client config directory.)

Part of the support for this change is the refactoring of GITC
related code into gitc_utils.py.

Change-Id: I2636aaa50b450b6f091309db8dd0e8f4dbdad579
2015-08-18 11:59:10 -07:00
Conley Owens
555be54790 Merge "Emit project info in case of sync exception." 2015-07-31 17:06:23 +00:00
Dan Sandler
c5cd433daf Emit project info in case of sync exception.
Previously repo would only print the failing project path if
Sync_NetworkHalf returned false/empty, but if it threw an
exception the print() was never called.

Change-Id: I58c41de43930df5e34b21561c205e062a72e290f
2015-07-31 14:03:50 +00:00
Kevin Degi
abaa7f312f Add option to correct gitdir when syncing
In some cases, a user may wish to continue with a sync even though
it would require overwriting an existing git directory. This behavior
is not safe as a default because it could result in the loss of some
user data, but as an optional flag it allows the user more flexibility.

To support this, add a --force-sync flag to the sync command that will
attempt to overwrite the existing git dir if it is specified and the
existing git dir points to the wrong obj dir.

Change-Id: Ieddda8ad54e264a1eb4a9d54881dd6ebc8a03833
2015-07-29 14:44:46 -06:00
David Pursehouse
59b417493e sync: Remove smart sync override manifest when not in smart sync mode
When syncing with the -s or -t option, a smart_sync_override.xml file
is created. This file is left in the file system when syncing again
without the -s or -t option.

Remove the smart sync override manifest, if it exists, when not using
the -s or -t option.

Change-Id: I697a0f6405205ba5f84a4d470becf7cd23c07b4b
2015-07-06 16:18:06 +09:00
David Pursehouse
727cc3e324 sync: Improve error message when writing smart sync manifest fails
The error message only states that writing the manifest failed.

Include the exception message, so it's easier to track down the reason
that the write failed.

Change-Id: I06e942c48a19521ba45292199519dd0a8bdb1de7
2015-07-06 16:18:06 +09:00
David Pursehouse
b155354034 Add option on sync to avoid fetching from remotes for existing sha1
In 2fb6466f79 an optimisation was
added to avoid fetching from remotes if the project is fixed to
a revision and the revision is already available locally.

This causes problems for users who expect all objects to be
fetched by default.

Change the logic so that the optimized behaviour is only enabled if
an option is explicitly given to repo sync.

Change-Id: I3b2794ddd8e0071b1787e166463cd8347ca9e24f
2015-04-30 14:29:02 +00:00
David Pursehouse
f2fad61bde Add missing documentation of --current-branch option on sync command
Change-Id: I72d6e3d51241148c1df97bbad26338debb1fcb4e
2015-01-29 14:36:28 +09:00
Jeff Davidson
5cf16607d3 Allow selection of a target when using smart sync.
Change-Id: I02a24471b9b62dbba3773f22a289825bc566acd9
2014-10-02 10:17:44 -07:00
Anthony King
85b24acd6a Use JSON instead of pickle
Use JSON as it is shown to be much faster than pickle.
Also clean up the loading and saving functions.

Change-Id: I45b3dee7b4d59a1c0e0d38d4a83b543ac5839390
2014-05-07 10:46:24 +01:00
David James
89ece429fb Clean up duplicate logic in subcmds/sync.py.
The fetch logic is now shared between the jobs == 1 and
jobs > 1 cases. This refactoring also fixes a bug where
opts.force_broken was not honored when jobs > 1.

Change-Id: Ic886f3c3c00f3d8fc73a65366328fed3c44dc3be
2014-02-14 16:14:32 +00:00
Conley Owens
094cdbe090 Add wrapper module
This takes the wrapper importing code from main.py and moves it into
its own module so that other modules may import it without causing
circular imports with main.py.

Change-Id: I9402950573933ed6f14ce0bfb600f74f32727705
2014-01-30 15:17:09 -08:00
Conley Owens
f2fe2d9b86 Properly iterate through values
the value of Manifest.projects has changed from being the dictionary
to the values of the dictionary.  Here we handle this change
correctly on a PostRepoUpgrade.

From a `git diff v1.12.7 -- manifest_xml.py`:
+  @property
   def projects(self):
     self._Load()
-    return self._projects
+    return self._paths.values()

self._paths does contain the projects according to this line of
manifest_xml.py:
484      self._paths[project.relpath] = project

Change-Id: I141f8d5468ee10dfb08f99ba434004a307fed810
2014-01-29 13:57:22 -08:00
Conley Owens
e695338e21 Merge "repo: Support multiple branches for the same project." 2014-01-10 01:20:13 +00:00
Julien Campergue
335f5ef4ad Add --archive option to init to sync using git archive
This significantly reduces sync time and used brandwidth as only
a tar of each project's revision is checked out, but git is not
accessible from projects anymore.

This is relevant when git is not needed in projects but sync
speed/brandwidth may be important like on CI servers when building
several versions from scratch regularly for example.

Archive is not supported over http/https.

Change-Id: I48c3c7de2cd5a1faec33e295fcdafbc7807d0e4d
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
2013-12-10 08:27:07 +00:00
Chirayu Desai
0eb35cbe50 Fix some python3 encoding issues
* Add .decode('utf-8') where needed
* Add 'b' to `open` where needed, and remove where unnecessary

Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
2013-11-21 06:03:22 +00:00
David James
8d20116038 repo: Support multiple branches for the same project.
It is often useful to be able to include the same project more than
once, but with different branches and placed in different paths in the
workspace. Add this feature.

This CL adds the concept of an object directory. The object directory
stores objects that can be shared amongst several working trees. For
newly synced repositories, we set up the git repo now to share its
objects with an object repo.

Each worktree for a given repo shares objects, but has an independent
set of references and branches. This ensures that repo only has to
update the objects once; however the references for each worktree are
updated separately. Storing the references separately is needed to
ensure that commits to a branch on one worktree will not change the
HEAD commits of the others.

One nice side effect of sharing objects between different worktrees is
that you can easily cherry-pick changes between the two worktrees
without needing to fetch them.

Bug: Issue 141
Change-Id: I5e2f4e1a7abb56f9d3f310fa6fd0c17019330ecd
2013-10-14 15:34:32 -07:00
David Pursehouse
351fe2c793 Sync: Improved error message when manifest server RPC call fails
When the RPC call fails, the error message returned by the server
is printed, but it is not obvious that this is caused by RPC call
failure.

Prefix the error message with a descriptive message that explains
what went wrong.

Change-Id: I4b77af22aacc2e9843c4df9d06bf54e41d9692ff
2013-09-25 19:12:13 +09:00
David Pursehouse
fb99c71939 Sync: Print name of manifest server used for smart sync/smart tag
When syncing using smart sync or smart tag mode, print the url of
the manifest server that is being used.

This is useful in organisations that have multiple manifest servers
used in different manifest branches.

Change-Id: Ib5bc2de5af6f4a942d0ef735c65cbc0721059a61
2013-09-25 19:12:06 +09:00
David Pursehouse
854f2b6ef4 Merge "sync: assign manifest_name earlier" 2013-06-11 13:58:47 +00:00
Chirayu Desai
a892b1006b sync: assign manifest_name earlier
* manifest_name was never set if opt.smart_sync or opt.smart_tag is used.
* Set it earlier, so that the code handles it correctly when it is None.
* An UnboundLocalError is raised if running `repo sync` without any options:
  local variable 'manifest_name' referenced before assignment
* This fixes the above regression caused by commit
  53a6c5d93a

Change-Id: I57086670f3589beea8461ce0344f6ec47ab85b7b
2013-06-11 14:18:58 +05:30
Chirayu Desai
db2ad9dfce Fix urllib.parse (urlparse) handling
Revert "Fix "'module' object is not callable" error", and fix it properly.

* The urlparse module is renamed to urllib.parse in Python 3.
* This commit fixes the code to use "urllib.parse.urlparse"
  instead of creating a new module urlib and setting
  urlib.parse to urlparse.urlparse.
* Fixes an AttributeError:
  'function' object has no attribute 'uses_relative'

This reverts commit cd51f17c64.

Change-Id: I48490b20ecd19cf5a6edd835506ea5a467d556ac
2013-06-11 08:21:10 +00:00
Conley Owens
65b162b32f Merge "Fix "'module' object is not callable" error" 2013-06-10 14:31:30 +00:00
David Pursehouse
cd51f17c64 Fix "'module' object is not callable" error
In a couple of files the urlparse method was not being set up
correctly for python < 3 and this resulted in an error being
thrown when trying to call it.

Change-Id: I4d2040ac77101e4e228ee225862f365ae3d96cec
2013-06-08 14:50:41 +09:00
Victor Boivie
53a6c5d93a Degrade: Fix smart sync/smart tag
This was broken in b2bd91c, which updated the manifest after it had
been overridden, which made it fall back to the original file (and
not the one from the manifest server).

This builds on 0766900 and overrides the manifest by the one
downloaded from the manifest server completely.

Change-Id: Ic3972390a68919b614616631d99c9e7a63c0e0db
2013-06-08 14:31:58 +09:00
David Pursehouse
59bbb580e3 Move Python version checking to a separate module
Add a new module with methods for checking the Python version.

Instead of handling Python3 imports with try...except blocks, first
check the python version and then import the relevant modules.  This
makes the code a bit cleaner and will result in less diff when/if we
remove support for Python < 3 later.

Use the same mechanism to handle `input` vs. `raw_input` and add
suppression of pylint warnings caused by redefinition of the built-in
method `input`.

Change-Id: Ia403e525b88d77640a741ac50382146e7d635924
Also-by: Chirayu Desai <cdesai@cyanogenmod.org>
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
2013-05-23 07:28:53 +00:00
Chirayu Desai
217ea7d274 Some fixes for supporting python3
* Fix imports.
* Use python3 syntax.
* Wrap map() calls with list().
* Use list() only wherever needed.
  (Thanks Conley!)
* Fix dictionary iteration methods
  (s/iteritems/items/).
* Make use of sorted() in appropriate places
* Use iterators directly in the loop.
* Don't use .keys() wherever it isn't needed.
* Use sys.maxsize instead of sys.maxint

TODO:
* Make repo work fully with python3. :)

Some of this was done by the '2to3' tool [1], by
applying the needed fixes in a way that doesn't
break compatibility with python2.

Links:
[1]: http://docs.python.org/2/library/2to3.html

Change-Id: Ibdf3bf9a530d716db905733cb9bfef83a48820f7
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
2013-04-18 21:35:49 +05:30
Chirayu Desai
fef4ae74e2 sync: be more verbose
* Print project name if the "quiet" option is not used.

Change-Id: I99863bb50f66e4dcbaf2d170bdd05971f2a4e19a
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
2013-04-15 13:38:49 +05:30
Tim Kilbourn
07669002cb Reload the correct manifest during sync.
Fix for issue #134
https://code.google.com/p/git-repo/issues/detail?id=134

Change-Id: I94c2dea5dd63917e3f9c90cbd628921d7d61b12a
2013-03-08 16:19:03 -08:00
David Pursehouse
2f9e7e40c4 Fix: Missing spaces in printed messages
Several messages are printed with the `print` method and the message
is split across two lines, i.e.:

 print('This is a message split'
       'across two source code lines')

Which causes the message to be printed as:

 This is a message splitacross two source code lines

Add a space at the end of the first line before the line break:

 print('This is a message split '
       'across two source code lines'

Also correct a minor spelling mistake.

Change-Id: Ib98d93fcfb98d78f48025fcc428b6661380cff79
2013-03-05 17:30:59 +09:00
Mitchel Humpherys
597868b4c4 Add --no-tags option to prevent fetching of tags
Add an option to pass `--no-tags' to `git fetch'.

Change-Id: I4158cc369773e08e55a167091c38ca304a197587
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
2013-02-27 11:00:49 +09:00
Torne (Richard Coles)
7bdbde7af8 Allow sync to run even when the manifest is broken.
If the current manifest is broken then "repo sync" fails because it
can't retrieve the default value for --jobs. Use 1 in this case, in
order that you can "repo sync" to get a fixed manifest (assuming someone
fixed it upstream).

Change-Id: I4262abb59311f1e851ca2a663438a7e9f796b9f6
2012-12-05 11:01:36 +00:00
Che-Liang Chiou
b2bd91c99b Represent git-submodule as nested projects, take 2
(Previous submission of this change broke Android buildbot due to
 incorrect regular expression for parsing git-config output.  During
 investigation, we also found that Android, which pulls Chromium, has a
 workaround for Chromium's submodules; its manifest includes Chromium's
 submodules.  This new change, in addition to fixing the regex, also
 take this type of workarounds into consideration; it adds a new
 attribute that makes repo not fetch submodules unless submodules have a
 project element defined in the manifest, or this attribute is
 overridden by a parent project element or by the default element.)

We need a representation of git-submodule in repo; otherwise repo will
not sync submodules, and leave workspace in a broken state.  Of course
this will not be a problem if all projects are owned by the owner of the
manifest file, who may simply choose not to use git-submodule in all
projects.  However, this is not possible in practice because manifest
file owner is unlikely to own all upstream projects.

As git submodules are simply git repositories, it is natural to treat
them as plain repo projects that live inside a repo project.  That is,
we could use recursively declared projects to denote the is-submodule
relation of git repositories.

The behavior of repo remains the same to projects that do not have a
sub-project within.  As for parent projects, repo fetches them and their
sub-projects as normal projects, and then checks out subprojects at the
commit specified in parent's commit object.  The sub-project is fetched
at a path relative to parent project's working directory; so the path
specified in manifest file should match that of .gitmodules file.

If a submodule is not registered in repo manifest, repo will derive its
properties from itself and its parent project, which might not always be
correct.  In such cases, the subproject is called a derived subproject.

To a user, a sub-project is merely a git-submodule; so all tips of
working with a git-submodule apply here, too.  For example, you should
not run `repo sync` in a parent repository if its submodule is dirty.

Change-Id: I4b8344c1b9ccad2f58ad304573133e5d52e1faef
2012-11-19 10:45:21 -08:00
David Pursehouse
8f62fb7bd3 Tidy up code formatting a bit more
Enable the following Pylint warnings:

  C0322: Operator not preceded by a space
  C0323: Operator not followed by a space
  C0324: Comma not followed by a space

And make the necessary fixes.

Change-Id: I74d74283ad5138cbaf28d492b18614eb355ff9fe
2012-11-14 12:09:38 +09:00
David Pursehouse
c1b86a2323 Fix inconsistent indentation
The repo coding style is to indent at 2 characters, but there are
many places where this is not followed.

Enable pylint warning "W0311: Bad indentation" and make sure all
indentation is at multiples of 2 characters.

Change-Id: I68f0f64470789ce2429ab11104d15d380a63e6a8
2012-11-14 11:38:57 +09:00
Sarah Owens
cecd1d864f Change print statements to work in python3
This is part of a series of changes to introduce Python3 support.

Change-Id: I373be5de7141aa127d7debdbce1df39148dbec32
2012-11-13 17:33:56 -08:00
Conley Owens
8d070cfb25 Always show --manifest-server-* options
The --manifest-server-* flags broke the smartsync subcmd since
the corresponding variables weren't getting set.  This change
ensures that they will always be set, regardless of whether we are
using sync -s or smartsync.

Change-Id: I1b642038787f2114fa812ecbc15c64e431bbb829
2012-11-06 13:14:31 -08:00
Dave Borowitz
e21526754b sync: Only parallelize gc for git >= 1.7.2
This minimum version is required for the -c argument to set config on
the command line. Without this option, git by default uses as many
threads per invocation as there are CPUs, so we cannot safely
parallelize without hosing a system.

Change-Id: I8fd313dd84917658162b5134b2d9aa34a96f2772
2012-10-31 12:27:17 -07:00
David Pursehouse
1d947b3034 Even more coding style cleanup
Fixing some more pylint warnings:

W1401: Anomalous backslash in string
W0623: Redefining name 'name' from outer scope
W0702: No exception type(s) specified
E0102: name: function already defined line n

Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
2012-10-30 10:28:20 +09:00
Shawn O. Pearce
cd81dd6403 Revert "Represent git-submodule as nested projects"
This reverts commit 69998b0c6f.
Broke Android's non-gitmodule use case.

Conflicts:
	project.py
	subcmds/sync.py

Change-Id: I68ceeb63d8ee3b939f85a64736bdc81dfa352aed
2012-10-26 12:24:57 -07:00
Shawn O. Pearce
80d2ceb222 repo selfupdate: Fix _PostRepoUpgrade takes 2 arguments
Change-Id: I1cf9e0674ea366ddce96c949e0bc085e3452b25a
2012-10-26 12:24:57 -07:00
Conley Owens
c9129d90de Update PGP keys during _PostRepoUpgrade in sync
Previously, if a key was added, a client wouldn't add the key during
the sync step.  This would cause issues if a new key were added and a
subsequent release were signed by that key.

Change-Id: I4fac317573cd9d0e8da62aa42e00faf08bfeb26c
2012-10-25 17:48:35 -07:00
Shawn Pearce
57365c98cc Merge "sync: Run gc --auto in parallel" 2012-10-25 17:38:05 -07:00
Shawn Pearce
2577cec095 Merge "sync: Keep a moving average of last fetch times" 2012-10-25 17:35:15 -07:00
Shawn Pearce
e48d34659e Merge "sync: Order projects according to last fetch time" 2012-10-25 17:33:36 -07:00
Che-Liang Chiou
ab8f911a67 Fix pylint warnings introduced by the submodule patch
"69998b0 Represent git-submodule as nested projects" has introduced a
few pylint warnings.

W0612:1439,8:Project._GetSubmodules.get_submodules: Unused variable 'sub_gitdir'
W0613:1424,36:Project._GetSubmodules.get_submodules: Unused argument 'path'
W0612:1450,25:Project._GetSubmodules.parse_gitmodules: Unused variable 'e'
W0622:516,8:Sync.Execute: Redefining built-in 'all'

Change-Id: I84378e2832ed1b5ab023e394d53b22dcea799ba4
2012-10-25 13:55:49 -07:00
Conley Owens
608aff7f62 Merge "Use modern Python exception syntax" 2012-10-25 10:03:37 -07:00
Dave Borowitz
188572170e sync: Run gc --auto in parallel
We can't just let this run wild with a high (or even low) -j, since
that would hose a system. Instead, limit the total number of threads
across all git gc subprocesses to the number of CPUs reported by the
multiprocessing module (available in Python 2.6 and above).

Change-Id: Icca0161a1e6116ffa5f7cfc6f5faecda510a7fb9
2012-10-25 08:12:48 -07:00
Dave Borowitz
d947858325 sync: Keep a moving average of last fetch times
Try to more accurately estimate which projects take the longest to
sync by keeping an exponentially weighted moving average (a=0.5) of
fetch times, rather than just recording the last observation. This
should discount individual outliers (e.g. an unusually large project
update) and hopefully allow truly slow repos to bubble to the top.

Change-Id: I72b2508cb1266e8a19cf15b616d8a7fc08098cb3
2012-10-24 14:52:07 -07:00
Dave Borowitz
67700e9b90 sync: Order projects according to last fetch time
Some projects may consistently take longer to fetch than others, for
example a more active project may have many more Gerrit changes than a
less active project, which take longer to transfer. Use a simple
heuristic based on the last fetch time to fetch slower projects first,
so we do not tend to spend the end of the sync fetching a small number
of outliers.

This algorithm is probably not optimal, and due to inter-run latency
variance and Python thread scheduling, we may not even have good
estimates of a project sync time.

Change-Id: I9a463f214b3ed742e4d807c42925b62cb8b1745b
2012-10-24 14:51:58 -07:00
Sarah Owens
a5be53f9c8 Use modern Python exception syntax
"except Exception as e" instead of "except Exception, e"

This is part of a transition to supporting Python 3.  Python >= 2.6
support "as" syntax.

Note: this removes Python 2.5 support.

Change-Id: I309599f3981bba2b46111c43102bee38ff132803
2012-10-23 21:35:59 -07:00
Che-Liang Chiou
69998b0c6f Represent git-submodule as nested projects
We need a representation of git-submodule in repo; otherwise repo will
not sync submodules, and leave workspace in a broken state.  Of course
this will not be a problem if all projects are owned by the owner of the
manifest file, who may simply choose not to use git-submodule in all
projects.  However, this is not possible in practice because manifest
file owner is unlikely to own all upstream projects.

As git submodules are simply git repositories, it is natural to treat
them as plain repo projects that live inside a repo project.  That is,
we could use recursively declared projects to denote the is-submodule
relation of git repositories.

The behavior of repo remains the same to projects that do not have a
sub-project within.  As for parent projects, repo fetches them and their
sub-projects as normal projects, and then checks out subprojects at the
commit specified in parent's commit object.  The sub-project is fetched
at a path relative to parent project's working directory; so the path
specified in manifest file should match that of .gitmodules file.

If a submodule is not registered in repo manifest, repo will derive its
properties from itself and its parent project, which might not always be
correct.  In such cases, the subproject is called a derived subproject.

To a user, a sub-project is merely a git-submodule; so all tips of
working with a git-submodule apply here, too.  For example, you should
not run `repo sync` in a parent repository if its submodule is dirty.

Change-Id: I541e9e2ac1a70304272dbe09724572aa1004eb5c
2012-10-23 16:08:58 -07:00
Mickaël Salaün
2f6ab7f5b8 Rename "dir" variables
The variable name "dir" conflicts with the name of a Python built-in
function: http://docs.python.org/library/functions.html#dir

Change-Id: I850f3ec8df7563dc85e21f2876fe5e6550ca2d8f
2012-10-10 08:30:15 +02:00
David Pursehouse
8a68ff9605 Coding style cleanup
Fix the following issues reported by pylint:

C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
W0612: Unused variable 'name'
W0613: Unused argument 'name'
W0102: Dangerous default value 'value' as argument
W0105: String statement has no effect

Also fixed a few cases of inconsistent indentation.

Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
2012-10-09 12:45:30 +02:00
David Pursehouse
e3b1c45aeb Remove unreachable code
Change 9bb1816b removed part of a block of code, but left the
remaining part unreachable.  Remove it.

Change-Id: Icdc6061d00e6027df32dee9a3bad3999fe7cdcbc
2012-10-05 10:34:19 +02:00
David Pursehouse
cf76b1bcec sync: Support manual authentication to the manifest server
Add two new command line options, -u/--manifest-server-username and
-p/--manifest-server-password, which can be used to specify a username
and password to authenticate to the manifest server when using the
-s/--smart-sync or -t/--smart-tag option.

If -u and -p are not specified when using the -s or -t option, use
authentication credentials from the .netrc file (if there are any).

Authentication credentials from -u/-p or .netrc are not used if the
manifest server specified in the manifest file already includes
credentials.

Change-Id: I6cf9540d28f6cef64c5694e8928cfe367a71d28d
2012-09-21 11:20:59 -07:00
David Pursehouse
86d973d24e sync: Support authentication to manifest server with .netrc
When using the --smart-sync or --smart-tag option, and the specified
manifest server is hosted on a server that requires authentication,
repo sync fails with the error: HTTP 401 Unauthorized.

Add support for getting the credentials from the .netrc file.

If a .netrc file exists in the user's home directory, and it contains
credentials for the hostname of the manifest server specified in the
manifest, use the credentials to authenticate with the manifest server
using the URL syntax extension for Basic Authentication:

  http://user:password@host:port/path

Credentials from the .netrc file are only used if the manifest server
URL specified in the manifest does not already include credentials.

Change-Id: I06e6586e8849d0cd12fa9746789e8d45d5b1f848
2012-09-11 09:45:48 +02:00
David Pursehouse
d94aaef39e sync: Correct imports of R_HEADS and HEAD
`R_HEADS` is imported twice, from both the git_refs and project
modules.

It is actually defined in git_refs, and in project it is imported
from there, so the import of `R_HEADS` from project in the sync
module is redundant.  Remove it.

`HEAD` is imported from project, but like `R_HEADS` it is actually
defined in git_refs.  Import it from git_refs instead.

Change-Id: I8e2b0217d0d9f9f4ee5ef5b8cd0b026174ac52f4
2012-09-07 10:17:00 +02:00
David Pursehouse
bd489c4eaa sync: catch exceptions when connecting to the manifest server
When connecting to the manifest server, exceptions can occur but
are not caught, resulting in the repo sync exiting with a python
traceback.

Add handling of the following exceptions:

- IOError, which can be raised for example if the manifest server
URL is malformed.
- xmlrpclib.ProtocolError, which can be raised if the connection
to the manifest server fails with HTTP error.
- xmlrpclib.Fault, which can be raised if the RPC call fails for
some other reason.

Change-Id: I3a4830aef0941debadd515aac776a3932e28a943
2012-09-06 11:18:25 -07:00
David 'Digit' Turner
e2126652a3 Make "repo sync -j<count>" stop properly on Ctrl-C.
The threaded 'repo sync' implementation would very often freeze the
process when interrupted by the user with Ctrl-C. The only solution
being to kill -9 the process explicitly from another terminal.

The reason for this is best explained here:

http://snakesthatbite.blogspot.fr/2010/09/cpython-threading-interrupting.html

This patch makes all helper sync threads 'daemon', which allows the
process to terminate immediately on Ctrl-C.

Note that this will forcefully kill all threads in case of interruption; this
is generally a bad thing, but:

  1/ This is equivalent to calling kill -9 in another terminal, which
     is the _only_ thing that can currently stop the process.

  2/ There doesn't seem to be a way to tell the worker threads to
     gently stop when they are in a blocking operation anyway (even
     in the non-threaded case).

+ Do the same for "repo status -j<count>".

Change-Id: Ieaf45b0eacee36f35427f8edafd87415c2aa7be4
2012-09-05 11:38:41 -07:00
Shawn O. Pearce
5d0efdb14a sync: Honor --no-clone-bundle with -j1
Change-Id: I7c12902e386121a374d525be673092360c67c53d
2012-08-02 12:13:01 -07:00
Colin Cross
5acde75e5d Add manifest groups
Allows specifying a list of groups with a -g argument to repo init.
The groups act on a group= attribute specified on projects in the
manifest.
All projects are implicitly labelled with "default" unless they are
explicitly labelled "-default".
Prefixing a group with "-" removes matching projects from the list
of projects to sync.
If any non-inverted manifest groups are specified, the default label
is ignored.

Change-Id: I3a0dd7a93a8a1756205de1d03eee8c00906af0e5
Reviewed-on: https://gerrit-review.googlesource.com/34570
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
2012-04-13 09:46:00 -07:00
Shawn O. Pearce
e02ac0af2e sync: --no-clone-bundle disables the clone bundle support
Change-Id: Ia9ed7da8451b273c1be620c3dd0dcad777b29096
2012-03-14 15:38:28 -07:00
Chris Wolfe
e9dc3b3368 sync: Add manifest_name parameter
This parameter changes the manifest used by 'repo sync' for only
this execution. It should be useful for developers wishing to get
the repo temporarily into a known state, without clobbering their
existing manifest.

Tested by shifting Chrome OS between minilayout and full, and
between several release-builder-generated manifests.

Change-Id: I14194b665195b0e78f368d9ec8b8a83227af2627
2012-01-26 12:32:36 -05:00
Anatol Pomazau
53d6f4d17e Add a sync flag that fetches only current branch
There is also shortcuts in case if the "current branch" is
a persistent revision such as tag or sha1. We check if the
persistent revision is present locally and if it does - do
no fetch anything from the server.

This greately reduces sync time and size of the on-disk repo

Change-Id: I23c6d95185474ed6e1a03c836a47f489953b99be
2011-11-03 13:08:27 -07:00
Shawn O. Pearce
df5ee52050 Fix Python 2.4 support
Change-Id: I89521ae52fa564f0d849cc51e71fee65b3c47bab
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 14:06:11 -07:00
Shawn O. Pearce
c4657969eb sync: Update default -j flag from manifest
If the manifest is updated and the default sync-j attribute
was modified, honor it during this sync session if the user
has not supplied a -j flag on the command line.

Change-Id: I127ee5c779e2bbbb40b30bddc10ec1fa704b3bf3
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-26 09:08:44 -07:00
Shawn O. Pearce
6392c87945 sync: Allow -j to have a default in manifest
This permits manifest authors to suggest a number of parallel
fetch operations against a remote server. For example, Gerrit
Code Review servers support queuing of requests and processes
them in first-in, first-out order. Running concurrent fetches
can utilize multiple CPUs on the Gerrit server, but will also
decrease overall operation latency by having the request put
into the queue ready to execute as soon as a CPU is free.

Change-Id: I3d3904acb6f63516bae4b071c510ad57a2afab18
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-22 18:08:27 -07:00
Shawn O. Pearce
97d2b2f7a0 sync: Limit -j to file descriptors
Each worker thread requires at least 3 file descriptors to run the
forked 'git fetch' child to operate against the local repository.
Mac OS X has the RLIMIT_NOFILE set to 256 by default, which means
a sync -j128 often fails when the workers run out of pipes within
the Python parent process.

Change-Id: I2cdb14621b899424b079daf7969bc8c16b85b903
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-22 18:08:26 -07:00
Victor Boivie
08c880db18 Smart tag support
This is an evolution of 'smart-sync' that adds a new option, -t,
that allows you to specify a tag/label to use instead of the
"latest good build" on the current manifest branch which -s does.

Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
Change-Id: I8c20fd91104a6aafa0271d4d33f6c4850aade17e
2011-07-20 07:13:48 -07:00
Daniel Sandler
723c5dc3d6 Fix parallel sync on python < 2.6.
Event.isSet was renamed to is_set in 2.6, but we should
use the earlier syntax to avoid breaking compatibility
with older Python installations.

Change-Id: I41888ed38df278191d7496c1a6eed15e881733f4
2011-04-04 11:34:47 -04:00
Shawn O. Pearce
e6a0eeb80d sync: Fix syntax error on Python 2.4
Change-Id: I371d032d5a1ddde137721cbe2b24bfa38f20aaaa
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-03-22 19:04:47 -07:00
Doug Anderson
fc06ced9f9 Make 'repo sync -jN' exit with an error code in the case of sync errors.
The bug that this is fixing is described here:

http://code.google.com/p/chromium-os/issues/detail?id=6813

This fix allows the helper threads to signal the main thread that they
saw an error.  When the main thread sees the error, it will let all
existing threads finish, then exit with an error.

Change-Id: If3019bc6b0b3ab9304d49ed2eea53e9d57f3095a
2011-03-17 09:17:42 -07:00
Shawn O. Pearce
0d2b61f11d sync: Run git gc --auto after fetch
Users may wind up with a lot of loose object content in projects they
don't frequently make changes in, but that are modified by others.

Since we bypass many git code paths that would have otherwise called
out to `git gc --auto`, its possible for these projects to have
their loose object database grow out of control.  To help prevent
that, we now invoke it ourselves during the network half of sync.

Signed-off-by: Shawn O. Pearce <sop@google.com>
(cherry picked from commit 1875ddd47c)
2011-01-09 17:39:22 -08:00
Shawn O. Pearce
f18cb76173 Encode the environment variables passed to git
Windows allows the environment to have unicode values.
This will cause Python to fail to execute the command.

Change-Id: I37d922c3d7ced0d5b4883f0220346ac42defc5e9
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-01-09 16:13:56 -08:00
Doug Anderson
2b8db3ce3e Added feature to print a <notice> from manifest at the end of a sync.
This feature is used to convey information on a when a branch has
ceased development or if it is an experimental branch with a few
gotchas, etc.

You add it to your manifest XML by doing something like this:
<manifest>
  <notice>
    NOTE TO DEVELOPERS:
      If you checkin code, you have to pinky-swear that it contains no bugs.
      Anyone who breaks their promise will have tomatoes thrown at them in the
      team meeting.  Be sure to bring an extra set of clothes.
  </notice>

  <remote ... />
  ...
</manifest>

Carriage returns and indentation are relevant for the text in this tag.

This feature was requested by Anush Elangovan on the ChromiumOS team.
2010-11-01 15:08:06 -07:00
Andrei Warkentin
5df6de075e sync: Use --force-broken to continue other projects
This adds a new flag -f/--force-broken that will allow the rest of
the sync process to continue instead of bailing when a particular
project fails to sync.

Change-Id: I23680f2ee7927410f7ed930b1d469424c9aa246e
Signed-off-by: Andrei Warkentin <andreiw@motorola.com>
Signed-off-by: Shawn O. Pearce <sop@google.com>
2010-10-29 12:20:01 -07:00
Shawn O. Pearce
16614f86b3 sync --quiet: be more quiet
Change-Id: I5e8363c7b32e4546d1236cfc5a32e01c3e5ea8e6
Signed-off-by: Shawn O. Pearce <sop@google.com>
2010-10-29 12:08:57 -07:00
Roy Lee
18afd7f679 sync: support --jobs to fetch projects simultaneously
This patch does two things for being compatibile with
those Python which are built without threading support:

1. As the Python document and Shawn suggested, import dummy_threading
   when the threading is not available.

2. Reserve the single threaded code and make it default.
   In cases the --jobs does not work properly with dummy_threading,
   we still have a safe fallback.

Change-Id: I40909ef8e9b5c22f315c0a1da9be38eed8b0a2dc
2010-05-27 14:54:20 -07:00
Nico Sallembien
6623b21e10 Aliasing sync -s to 'smartsync'
This alias will let people use this command without having to
remember the option.

Change-Id: I3256d9e8e884c5be9e77f70e9cfb73e0f0c544c6
2010-05-17 09:58:55 -07:00
Nico Sallembien
719965af35 Override manifest file only after it is fully written to disk.
We called "Override()" before closing the file passed in argument.

Change-Id: I15adb99deb14297ef72fcb1b0945eb246f172fb0
2010-04-26 11:20:22 -07:00
Nico Sallembien
5732e47ebb Strip refs/heads in the branch sent to the manifest server.
The manifest server doesn't want to have refs/heads passed to it, so
we need to strip that when the branch contains it.

Change-Id: I044f8a9629220e886fd5e02e3c1ac4b4bb6020ba
2010-04-26 11:19:07 -07:00
Anthony
f3fdf823cf sync: Safely skip already deleted projects
Do not error if a project is missing on the filesystem, is deleted
from manifest.xml, but still exists in project.list.

Change-Id: I1d13e435473c83091e27e4df571504ef493282dd
2010-04-14 14:21:50 -07:00
Nico Sallembien
a1bfd2cd72 Add a 'smart sync' option to repo sync
This option allows the user to specify a manifest server to use when
syncing. This manifest server will provide a manifest pegging each
project to a known green build. This allows developers to work on a
known good tree that is known to build and pass tests, preventing
failed builds to hamper productivity.

The manifest used is not "sticky" so as to allow subsequent
'repo sync' calls to sync to the tip of the tree.

Change-Id: Id0a24ece20f5a88034ad364b416a1dd2e394226d
2010-04-13 10:20:37 -07:00
Nico Sallembien
9bb1816bdc Fixing project renaming bug.
This bug happens when a project gets added to the manifest, and
then is renamed. Users who happened to have run "repo sync" after
the project was added but before the rename happened will try to
read the data from the old project, as the manifest was only updated
after all projects were updated successfully.
2009-12-10 15:24:45 -08:00
Shawn O. Pearce
9fb29ce123 sync: Keep the project.list file sorted
Its easier to locate an entry visually if the file is sorted.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-06-04 20:41:26 -07:00
Shawn O. Pearce
3a68bb4c7f sync: Tolerate blank lines in project.list
If a line is blank in project.list, its not a relevant project path,
so skip over it.  Existing project.list files may have blank lines if
sync was run with no projects at all, and the file was created empty.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-06-04 16:21:01 -07:00
Shawn O. Pearce
cd1d7ff81e sync: Don't process project.list in a mirror
We have no working tree, so we cannot update the project.list
state file, nor should we try to delete a directory if a project is
removed from the manifest.  Clients would still need the repository
for historical records.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-06-04 16:20:02 -07:00
Jaikumar Ganesh
8135cdc53c Delete empty parent subdirs after deleting obsolete paths.
After sync, we delete obsolete project paths.
Iterate and delete parent subdirs which are empty.
Tested on projects within subdirectories.
2009-06-02 15:08:45 -07:00
Jaikumar Ganesh
4f2517ff11 Update project paths after sync.
After a repo sync, some of the project paths might need
to be removed. This changes maintains a list of project
paths from the previous sync operation and compares.
2009-06-02 11:00:53 -07:00
Shawn O. Pearce
3c8dea1f8d Change project.revision to revisionExpr and revisionId
The revisionExpr field now holds an expression from the manifest,
such as "refs/heads/master", while revisionId holds the current
commit-ish SHA-1 of the revisionExpr.  Currently that is only
filled in if the manifest points directly to a SHA-1.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-05-29 18:45:20 -07:00
Shawn O. Pearce
eb7af87bcf Document the SSH ControlMaster behavior of repo sync
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-21 08:28:06 -07:00
Shawn O. Pearce
fd89b67f5c Clarify options that control the repo executable version
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 11:28:57 -07:00
Shawn O. Pearce
f690687671 Only fetch repo once-per-day under normal 'repo sync' usage
Its unlikely that a new version of repo will be delivered in any
given day, so we now check only once every 24 hours to see if repo
has been updated.  This reduces the sync cost, as we no longer need
to contact the repo distribution servers every time we do a sync.

repo selfupdate can still be used to force a check.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 10:49:00 -07:00
Shawn O. Pearce
350cde4c4b Change repo sync to be more friendly when updating the tree
We now try to sync all projects that can be done safely first, before
we start rebasing user commits over the upstream.  This has the nice
effect of making the local tree as close to the upstream as possible
before the user has to start resolving merge conflicts, as that extra
information in other projects may aid in the conflict resolution.

Informational output is buffered and delayed until calculation for
all projects has been done, so that the user gets one concise list
of notice messages, rather than it interrupting the progress meter.

Fast-forward output is now prefixed with the project header, so the
user can see which project that update is taking place in, and make
some relation of the diffstat back to the project name.

Rebase output is now prefixed with the project header, so that if
the rebase fails, the user can see which project we were operating
on and can try to address the failure themselves.

Since rebase sits on a detached HEAD, we now look for an in-progress
rebase during sync, so we can alert the user that the given project
is in a state we cannot handle.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-16 11:21:18 -07:00
Shawn O. Pearce
e756c412e3 Add 'repo selfupdate' to upgrade only repo
Users may want to upgrade only repo to the latest release, but
leave their working tree state alone and avoid 'repo sync'.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-13 11:53:53 -07:00
Shawn O. Pearce
68194f42b0 Add a project progress meter to 'repo sync'
This way users can see how much is left during fetch.  Its
especially useful when most syncs are no-ops but there are
hundreds of repositories to poll.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-10 19:01:04 -07:00
Shawn O. Pearce
b1562faee0 Add 'repo sync -l' to only do local operations
This permits usage of 'repo sync' while offline, as we bypass the
network based portions of the code and do only the local sync.

An example use case might be:

  repo sync -n  ; # while we have network
  ... some time later ...
  repo sync -l  ; # while without network, come up to date

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-10 17:08:02 -07:00
Shawn O. Pearce
3e768c9dc7 Add 'repo sync -d' to detach projects from their current topic
The -d flag moves the project back to a detached HEAD state,
matching what is listed in the manifest.  This can be useful to
set a client to something stable (or at least well-known), such as
before a sequence of 'repo download' commands are used to get some
changes for testing.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-10 17:08:02 -07:00
Shawn O. Pearce
96fdcef9e3 Add 'repo sync -n' to only do the network transfer
This makes it easier to update all repositories, without actually
impacting the working directory, or learning about how to use
`repo forall -c 'git fetch $REPO_REMOTE' `.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-10 17:07:52 -07:00