Try to more accurately estimate which projects take the longest to
sync by keeping an exponentially weighted moving average (a=0.5) of
fetch times, rather than just recording the last observation. This
should discount individual outliers (e.g. an unusually large project
update) and hopefully allow truly slow repos to bubble to the top.
Change-Id: I72b2508cb1266e8a19cf15b616d8a7fc08098cb3
Some projects may consistently take longer to fetch than others, for
example a more active project may have many more Gerrit changes than a
less active project, which take longer to transfer. Use a simple
heuristic based on the last fetch time to fetch slower projects first,
so we do not tend to spend the end of the sync fetching a small number
of outliers.
This algorithm is probably not optimal, and due to inter-run latency
variance and Python thread scheduling, we may not even have good
estimates of a project sync time.
Change-Id: I9a463f214b3ed742e4d807c42925b62cb8b1745b
Fixing more issues found with pylint. Some that were supposed to
have been fixed in the previous sweep (Ie0db839e) but were missed:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
And some more:
W0631: Using possibly undefined loop variable 'name'
W0223: Method 'name' is abstract in class 'name' but is not overridden
W0231: __init__ method from base class 'name' is not called
Change-Id: Ie119183708609d6279e973057a385fde864230c3
If repo could not find authentication credentials from ~/.netrc, this
patch tries to get user and password from user's console input. This
could be a good choice if user doesn't want to save his plain password
in ~/.netrc or if user doesn't know about the netrc usage.
The user will be prompted only if authentication infomation does not
exist in the password manager. Since main.py firstly loads auth
infomation from ~/.netrc, this will be executed only as fallback
mechanism.
Example:
$ repo upload .
Upload project xxx/ to remote branch master:
branch yyy ( 1 commit, ...):
to https://review.zzz.com/gerrit/ (y/N)? y
(repo may try to access to https://review.zzz.com/gerrit/ssh_info and
will get the 401 HTTP Basic Authentication response from server. If no
authentication info in ~/.netrc, This patch will ask username/passwd)
Authorization Required (Message from Web Server)
User: pororo
Password:
....
[OK ] xxx/
Change-Id: Ia348a4609ac40060d9093c7dc8d7c2560020455a
Fix the following issues reported by pylint:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
W0612: Unused variable 'name'
W0613: Unused argument 'name'
W0102: Dangerous default value 'value' as argument
W0105: String statement has no effect
Also fixed a few cases of inconsistent indentation.
Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
Change 9bb1816b removed part of a block of code, but left the
remaining part unreachable. Remove it.
Change-Id: Icdc6061d00e6027df32dee9a3bad3999fe7cdcbc
The definition of `ImportError` redefines the Python built-in
class of the same name.
It is not used anywhere, so remove it.
Change-Id: I557ce28c93a3306fff72873dc6f477330fc33128
Currently when doing a sync against a revision locked manifest,
sync has no option but to fall back to sync'ing the entire refs space;
it doesn't know which ref to ask for that contains the sha1 it wants.
This sucks if we're in -c mode; thus when we generate a revision
locked manifest, record the originating branch- and try syncing that
branch first. If the sha1 is found within that branch, this saves
us having to pull down the rest of the repo- a potentially heavy
saving.
If that branch doesn't have the desired sha1, we fallback to sync'ing
everything.
Change-Id: I99a5e44fa1d792dfcada76956a2363187df94cf1
Change Details:
* Switch first default group to 'all' instead of 'default'
Change Benefits:
* More consistent with default_groups in the counterpart Save() function
* Fixes bug where command 'repo manifest' added an extra 'default'
group to every output project element groups attribute. This bug was
particularly confusing for projects which had 'groups="notdefault"'
as they were output as 'groups="notdefault,default"' by 'repo manifest'
Change-Id: I5611c027a982d3394899466248b971910bec8c6b
Add two new command line options, -u/--manifest-server-username and
-p/--manifest-server-password, which can be used to specify a username
and password to authenticate to the manifest server when using the
-s/--smart-sync or -t/--smart-tag option.
If -u and -p are not specified when using the -s or -t option, use
authentication credentials from the .netrc file (if there are any).
Authentication credentials from -u/-p or .netrc are not used if the
manifest server specified in the manifest file already includes
credentials.
Change-Id: I6cf9540d28f6cef64c5694e8928cfe367a71d28d
manifest_xml: import `HEAD` and `R_HEADS` from correct module
version: import `HEAD` from correct module
`HEAD` and `R_HEADS` should be imported from the git_refs module,
where they are originally defined, rather than from the project
module.
repo: remove unused import of readline
cherry_pick: import standard modules on separate lines
smartsync: import subcmd modules explicitly from subcmd
Use:
`import re
import sys`
and
`from subcmds.sync import Sync`
Instead of:
`import sys, re`
and
`from sync import Sync`
Change-Id: Ie10dd6832710939634c4f5c86b9ba5a9cd6fc92e
When using the --smart-sync or --smart-tag option, and the specified
manifest server is hosted on a server that requires authentication,
repo sync fails with the error: HTTP 401 Unauthorized.
Add support for getting the credentials from the .netrc file.
If a .netrc file exists in the user's home directory, and it contains
credentials for the hostname of the manifest server specified in the
manifest, use the credentials to authenticate with the manifest server
using the URL syntax extension for Basic Authentication:
http://user:password@host:port/path
Credentials from the .netrc file are only used if the manifest server
URL specified in the manifest does not already include credentials.
Change-Id: I06e6586e8849d0cd12fa9746789e8d45d5b1f848
If the first line of manifest.xml is a XML comment, root.childNodes[0]
is not a 'manifest' element node. The python minidom module will makes
a 'Comment' node as root.childNodes[0]. Since the original code only
checks whether the first child node is 'manifest', it couldn't do any
command including 'sync' due to the 'ManifestParseError' exception. This
patch could allow the comments between '<?xml ...?>' and '<manifest>' in
the manifest.xml file.
Change-Id: I0b81dea4f806965eca90f704c8aa7df49c579402
`R_HEADS` is imported twice, from both the git_refs and project
modules.
It is actually defined in git_refs, and in project it is imported
from there, so the import of `R_HEADS` from project in the sync
module is redundant. Remove it.
`HEAD` is imported from project, but like `R_HEADS` it is actually
defined in git_refs. Import it from git_refs instead.
Change-Id: I8e2b0217d0d9f9f4ee5ef5b8cd0b026174ac52f4
When connecting to the manifest server, exceptions can occur but
are not caught, resulting in the repo sync exiting with a python
traceback.
Add handling of the following exceptions:
- IOError, which can be raised for example if the manifest server
URL is malformed.
- xmlrpclib.ProtocolError, which can be raised if the connection
to the manifest server fails with HTTP error.
- xmlrpclib.Fault, which can be raised if the RPC call fails for
some other reason.
Change-Id: I3a4830aef0941debadd515aac776a3932e28a943
Catch curl failures to download clone.bundle; don't let git try to parse
the 404 page as a bundle file (was causing much user confusion).
This should eliminate false error messages from init and sync such as:
error: '.repo/manifests.git/clone.bundle' does not look like a v2 bundle file
fatal: Could not read bundle '.repo/manifests.git/clone.bundle'.
error: RPC failed; result=22, HTTP code = 400
Signed-off-by: Matt Gumbel <matthew.k.gumbel@intel.com>
Change-Id: I7994f7c0baecfb45bb5a5850c48bd2a0ffabe773
Instead of every group being in the group "default", every project
is now in the group "all". A group that should not be downloaded
by default may be added to the group "notdefault".
This allows all group names to be positive (instead of removing groups
directly in the manifest with -default) and offers a clear way of
selecting every project (--groups all).
Change-Id: I99cd70309adb1f8460db3bbc6eff46bdcd22256f
The threaded 'repo sync' implementation would very often freeze the
process when interrupted by the user with Ctrl-C. The only solution
being to kill -9 the process explicitly from another terminal.
The reason for this is best explained here:
http://snakesthatbite.blogspot.fr/2010/09/cpython-threading-interrupting.html
This patch makes all helper sync threads 'daemon', which allows the
process to terminate immediately on Ctrl-C.
Note that this will forcefully kill all threads in case of interruption; this
is generally a bad thing, but:
1/ This is equivalent to calling kill -9 in another terminal, which
is the _only_ thing that can currently stop the process.
2/ There doesn't seem to be a way to tell the worker threads to
gently stop when they are in a blocking operation anyway (even
in the non-threaded case).
+ Do the same for "repo status -j<count>".
Change-Id: Ieaf45b0eacee36f35427f8edafd87415c2aa7be4
master's original purpose was to forge ahead on using git submodules,
but this route has been abandoned.
Change-Id: I164a9efc7821bcd1b941ad76649764722046081b
One of the recent changes introduced implicit path:xxx and name:xxx groups
to every project, however they are not being stripped when generating
a manifest using "repo manifest" command resulting in clutter
Change-Id: Iec8610ba794b2fe4a6cdf0f59ca561595b66f9b5
urllib2 is not thread safe and may be causing sync to lock up or
not work correctly on various platforms. Instead use the command
line curl program.
Change-Id: I36eaf18bb4df089d26ea99d533cb015e7c616eb0
The Content-Length when resuming is the number of bytes that
remain in the file. To compute the total size as expected by
the progress meter, we must add the bytes already stored.
While we are in this method fix uses of % operator to ensure
a tuple is always supplied.
Change-Id: Ic899231b5bc0ab43b3ddb1d29845f6390e820115
The `alias` is an optional attribute in element `remote`. It can be
used to override attibute `name` to be set as the remote name in each
project's .git/config. Its value can be duplicated while attribute
`name` has to be unique across the manifest file. This helps each
project to be able to have same remote name which actually points
to different remote url.
It eases some automation scripts to be able to checkout/push to same
remote name but actually different remote url, like:
repo forall -c "git checkout -b work same_remote/work"
repo forall -c "git push same_remote work:work"
for example:
The manifest with 'alias' will look like:
<?xml version='1.0' encoding='UTF-8'?>
<manifest>
<remote alias="same_alias" fetch="git://git.external1.org/" name="ext1"
review="http://review.external1.org"/>
<remote alias="same_alias" fetch="git://git.external2.org/" name="ext2"
review="http://review.external2.org"/>
<remote alias="same_alias" fetch="ssh://git.internal.com:29418" name="int"
review="http://review.internal.com"/>
<default remote="int" revision="int-branch" sync-j="2"/>
<project name="path/to/project1" path="project1" remote="ext1"/>
<project name="path/to/project2" path="project2" remote="ext2"/>
<project name="path/to/project3" path="project3"/>
...
</manifest>
In each project, use command "git remote -v"
project1:
same_alias git://git.external1.org/project1 (fetch)
same_alias git://git.external1.org/project1 (push)
project2:
same_alias git://git.external2.org/project2 (fetch)
same_alias git://git.external2.org/project2 (push)
project3:
same_alias ssh://git.internal.com:29418/project3 (fetch)
same_alias ssh://git.internal.com:29418/project3 (push)
Change-Id: I2c48263097ff107f0c978f3e83966ae71d06cb90
The overview command shows an overview of each branch in all (or the
specified) projects. The overview lists any local commits that have
not yet been merged into the project.
The report output is inspired by the report displayed following a
"repo prune" event, with the addition of listing the one-line log
messages for each commit that is not yet merged.
The report can also be filtered to show only active branches; by
default all branches that have commits beyond the upstream HEAD will
be listed.
Change-Id: Ibe67793991ad1aa38de3bc9747de4ba64e5591aa
For CrOS, we have scenarios were people checkout a smaller version
of our manifest via groups, and enable individual repositories as
needed for their work. Previously this was via local_manifest
manipulation, which breaks via manifest-groups would require a
remove-project tag.
Via injecting the projects name into the projects groups, this
allows us to instead manipulate the configured groups allowing
the user to turn on/off projects as necessary.
Change-Id: I07b7918e16cc9dc28eb47e19a46a04dc4fd0be74
The fix for issue #46 in 5d016502eb appears to break syncing in some
situations: the branch is deleted after the point where it's been
configured, which deletes part of its configuration and causes the
config to change each time you call `repo init`, alternating between a
configuration that works and one that doesn't.
Instead of deleting the branch with git branch -D, use git update-ref -d
which just deletes the ref (to avoid the rebase) without touching the
configuration for the branch that was set up during the first repo init.
This appears to ensure the config is left in a valid state all the time
no matter what combination of repo init commands you run, without
reintroducing the rebasing issue.
Change-Id: Iaadaa6e56a46840bbc593fa5b35cb5b34cd3ce69
Git requires the values in this environment variable to be
single quoted. repo must wrap the expression into '' before
adding it to the environment.
Change-Id: I20a1fb8772f9aa6e9fd5a0516c981c2ca020ef05
This patch fixes repo behaviour when running sync -d with unmodified
topic branches.
Prior to this patch sync -d would see the latest revision is already
checked out, thus staying on the branch. Since "-d" means detach we
should follow git's behaviour and actually detach from the branch in
that case.
Basic test case - after a fresh repo init + sync -
* repo start --all testdetach
* repo sync -d
* repo status
-> status shows active topic branch "testdetach",
should show :
nothing to commit (working directory clean)
Change-Id: Ic1351e6b5721b76557a51ab09f9dd42c38a4b415
Currently repo-rebase requires that all modifications be committed
locally before it will allow the rebase. In high-velocity environments,
you may want to just pull in newer code without explicitly creating
local commits, which is typically achieved using git-stash.
If called with the --auto-stash command line argument, and it is
determined that the current index is dirty, the local modifications
are stashed, and the rebase continues. If a stash was performed, that
stash is popped once the rebase completes.
Note that there is still a possibility that the git-stash pop will
result in a merge conflict.
Change-Id: Ibe3da96f0b4486cb7ce8d040639187e26501f6af
