Syncing projects works fine the majority of the time. So rather than
dump all of that noisy output to stdout, lets capture it and only show
when things fail or in verbose mode. This tidies up the default `repo
sync` output.
Bug: https://crbug.com/gerrit/11293
Change-Id: I8314dd92e1e6aadeb26e36a8c92610da419684e6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255413
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This allows us to control sync output better by having three levels
of output: quiet (only errors), default (progress bars), verbose (all
the things). For now, we just put the chatty "already have persistent
ref" message behind the verbose level.
Bug: https://crbug.com/gerrit/11293
Change-Id: Ia61333fd8085719f3e99edb7b466cdb04031b67f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255414
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Our sync output is pretty chatty, and the stat output on fast forward
merges doesn't really help. Suppress it to tighten up the output.
Change-Id: I91e50639b3cd8db9df3d13a7da6d1aaa70d7932f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255412
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
All of the instances of this are related to Python 2 names that
don't exist in Python 3, and the warnings are raised when running
flake8 on Python 3.
All of these will go away once we completely remove support for
Python 2, so just suppress them inline. We don't globally suppress
the check so that we will still see legitimate errors if/when they
occur in new code.
Change-Id: Iccf955f50abfc9f83b371fc0af6cceb51037456f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255039
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
We've been requiring git-1.7.2 since Oct 2012, so we can safely drop
the individual checks sprinkled throughout the code base for older.
Change-Id: I1737fff7b3f27f475960b0bff9cb300aefd5d108
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253135
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
flake8 reports:
E722 do not use bare 'except'
Replace them with 'except Exception' per [1] which says:
Bare except will catch exceptions you almost certainly don't want
to catch, including KeyboardInterrupt (the user hitting Ctrl+C) and
Python-raised errors like SystemExit
If you don't have a specific exception you're expecting, at least
except Exception, which is the base type for all "Regular" exceptions.
[1] https://stackoverflow.com/a/54948581
Change-Id: Ic555ea9482645899f5b04040ddb6b24eadbf9062
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254606
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E301,E302,E303,E305,E306
Manually fix issues in project.py caused by misuse of block comments.
Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E121 continuation line under-indented for hanging indent
- E122 continuation line missing indentation or outdented
- E125 continuation line with same indent as next logical line
- E126 continuation line over-indented for hanging indent
- E127 continuation line over-indented for visual indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E121,E122,E125,E126,E127,E128,E129,E131
Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
We have a few files that we optionally symlink from the work tree
.git/ to the .repo/projects/ path. If they don't exist when we
first initialize, then we skip creating symlinks. If the files
are created later on under the work tree .git/, repo gets upset.
This can happen with the packed-refs file: if we don't have any
packed refs initially, we don't symlink it. But if git tries to
pack refs later on and creates the file, the project gets wedged.
We could create an empty file initially and then symlink it, but
for some files, it's not clear we want to always do that (e.g.
the .git/shallow setting). Instead, lets make handling of these
paths more dynamic. If they show up later on in the work tree
.git/ only, we'll take care of relocating & symlinking. This
also makes repo a little more robust and autorecovers incase a
path goes missing in one of the dirs.
Ideally we wouldn't monkey around at all here, but considering
the only option we give to users currently is to blow things
away with --force-sync, this seems a bit better.
Bug: https://crbug.com/gerrit/12324
Change-Id: Ia6960f1896ac6d890c762d7d053684a1c6ab2c87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254632
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This reverts commit 4abf8e6ef8.
The curl process for updating the cookie file is not atomic. When
fetching many bundles in parallel, we can sometimes corrupt the file
causing it to be cleared. Since users should manage gitcookies on
their own, leave it read-only.
Bug: https://crbug.com/gerrit/12300
Change-Id: Id472c99b197bc4cf8533c649f8881509f38643c1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254092
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Since tempfile.mkstemp() returns a file handle in binary mode,
make sure we turn our strings into bytes before writing.
Bug: https://crbug.com/gerrit/12043
Change-Id: I3e84d595e84b8bc12a1fbc7fd0bb3ea0ba2832b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254393
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Some Android/Nest manifests are using <linkfile> with src="." to
create stable paths to specific projects. Allow that specific
use case as it seems reasonable to support.
Bug: https://crbug.com/gerrit/11218
Change-Id: I16dbe8d9fe42ea45440afcb61404c753bff1930d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254330
Reviewed-by: Chanho Park <parkch98@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Add a helper function to unify the duplication of finding the full
path to the symbolic HEAD ref. This makes it easy to handle git
worktrees where .git is a file rather than a dir/symlink.
Bug: https://crbug.com/gerrit/11486
Change-Id: I9f794f1295ad0d98c7c13622f01ded51e4ba7846
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254074
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
These files are not directly executable, so drop the +x bits.
Change-Id: Iaf19a03a497686cc21103e7ddf08073173440dd1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254076
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* Mirrors may contain multiple projects, some of which may not
always contain the default revision.
* Only fetch the default revision explicitly if
'--current-branch' is set.
* Fixes breakage casued by
commit 6856f98467
"Fix repo mirror with --current-branch"
Bug: https://crbug.com/gerrit/12274
Change-Id: Iaafabe2992f76f3644b841f24245d3e19c9515a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253093
Reviewed-by: Kuang-che Wu <kcwu@chromium.org>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Chirayu Desai <chirayudesai1@gmail.com>
Reject paths in <copyfile> & <linkfile> that try to use symlinks or
non-file or non-dirs.
We don't fully validate <linkfile> when src is a glob as it's a bit
complicated -- any component in the src could be the glob. We make
sure the destination is a directory, and that any paths in that dir
are created as symlinks. So while this can be used to read any path,
it can't be abused to write to any paths.
Bug: https://crbug.com/gerrit/11218
Change-Id: I68b6d789b5ca4e43f569e75e8b293b3e13d3224b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233074
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Before a2cd6aeae8, "repo mirror with --current-branch" fetches git data
using command
git fetch --progress --update-head-ok cros --tags
No refspec is specified, thus it fetches default refspec, which is +refs/heads/*:refs/heads/*
After a2cd6aeae8, the fetch command became
git fetch --progress --update-head-ok cros --tags +refs/tags/*:refs/tags/*
It did not only add tags refspec, but also suppressed the fetching of default refspec.
In other words, repo mirrors doesn't work if current_branch_only=True.
This CL explicitly adds the default refspec to command line if none is
specified.
Bug: https://crbug.com/gerrit/11990
Change-Id: Iadcf7b9aa50f53c47132cfe6c53b3fb2076ebca2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/246632
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Kuang-che Wu <kcwu@chromium.org>
Series of steps:
* Create a local "b1" branch with `repo start b1` that tracks a remote
branch (totally fine)
* Manually create a local "b2" branch with `git branch --track b1 b2`
that tracks the local "b1" (uh-oh...)
* Delete the local "b1" branch manually or via `repo prune` (....)
* Try to process the "b2" branch with `repo prune`
Since b2 tracks a branch that no longer exists, everything blows up
at this point as we try to probe the non-existent ref. Instead, we
should flag this as unknown and leave it up to the user to resolve.
This probably could come up if a local branch was tracking a remote
branch that was deleted from the server, and users ran something like
`repo sync --prune` which cleaned up the remote refs.
Bug: https://crbug.com/gerrit/11485
Change-Id: I6b6b6041943944b8efa6e2ad0b8b10f13a75a5c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236793
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Kirtika Ruchandani <kirtika@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the local branch changed state since its last upload, the data
cached in .git/config related to the last uploaded CL might not be
that relevant. If we're able to fast forward merge to the latest
tree state, then let's do that. This would be akin to checking
out a detached head before syncing where we already switch state.
If we aren't able to fast forward merge, then it's not a big deal
as we'll continue on to the existing branch checking logic.
This would be easy to reproduce by doing something like:
$ repo start foo .
$ git revert HEAD
$ repo upload --cbr .
$ git reset --hard HEAD^
<CL is merged>
$ repo sync .
<we can fast forward>
Change-Id: I7d62f3d1ba5314a349d85b4dbb0ec8352eca18bb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238552
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If you run `repo sync -l foo` without first `repo sync -n foo`,
repo sets up an invalid gitdir tree that gets wedged and requires
manual recovery. Add a sanity check to abort cleanly first.
Change-Id: Iad865ea860a3f1fd2f39ce683fe66bd4380745a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244732
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Use open() as a context manager to simplify the close logic and make
the code easier to read & understand. This is also more Pythonic.
Change-Id: I579d03cca86f99b2c6c6a1f557f6e5704e2515a7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244734
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Hitting Ctrl-C in the middle of this func will leave the .git in a
bad state that requires manual recovery. The code tries to catch
all exceptions and recover by deleting the incomplete .git dir, but
it omits KeyboardInterrupt which Exception misses.
We could add that to the recovery path, but we can make this more
robust with a different approach: set up everything in .git.tmp/
and only move it to .git/ once we've fully initialized it.
Change-Id: I0f5b97f2e19fc39cffc3e5e23993a2da7220f4e3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244733
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If the repo client checkout is in an incomplete sync state, the work
git repo might be in a bad way. Turn errors parsing HEAD into None
since callers of CurrentBranch already need to account for it.
Change-Id: Ia7682e29ef4182006b1fb5f5e57800f8ab67a9f4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239239
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If `git diff` fails in any project checkout (e.g. an incomplete
sync), make sure we print that error clearly rather than blowing
up, and exit non-zero in the process.
Bug: https://crbug.com/gerrit/11613
Change-Id: I12f278427cced20f23f8047e7e3dba8f442ee25e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239236
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When displaying progress bars, we use \r to reset the cursor to the
start of the line before showing the new update. This assumes the
new line will fully erase whatever was displayed there previously.
The "done" codepath tries to handle this by including a few extra
spaces at the end of the message to "white out" what was there.
Lets replace that hack with the standard ECMA escape sequence that
clears the current line completely. This is the CSI "erase in line"
sequence that the terminal will use to delete all content. The \r
is still needed to move the cursor to the start of the line. Using
this sequence should be OK since we're already assuming the terminal
is ECMA compliant with our use of coloring sequences. We also put
the \r after the CSI sequence on the off chance the terminal can't
process it and displays a few bytes of garbage.
The other improvement is to the syncbuffer API. When it dumps its
status information, it almost always comes after a progress bar
update which leads to confusing comingled output. Something like:
Fetching projects: 100% (2/2) error: src/platform2/: branch ...
Since the progress bar is "throw away", have the syncbuffer reset
the current output to the start of the line before showing whatever
messages it has queued.
Bug: https://crbug.com/gerrit/11293
Change-Id: I6544d073fe993d98ee7e91fca5e501ba5fecfe4c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236615
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This makes it way easier to recover from forgetting to run repo start
before committing: just run `repo start -b new-branch`, instead of
all that tedious mucking around with reflogs.
Change-Id: I56d49dce5d027e28fbba0507ac10cd763ccfc36d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232712
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
There is a standard Python "trace" module, so having a local trace.py
prevents us being able to import that. Rename the module to avoid.
Change-Id: I23e29ec95a2204bb168a641323d05e76968d9b57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234832
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The GitCommand Wait helper takes care of decoding bytes to strings
for us. That means we don't have to decode stdout ourselves which
is what our local rev list, ls-remote, and generic get_attr helpers
were doing.
If we don't use Wait though to capture the output but instead go
directly to the subprocess stdout, we do have to handle decoding
ourselves. This is what the diff helpers were doing.
Bug: https://crbug.com/gerrit/10418
Change-Id: I057ca245af3ff18d6b4a074e3900887f06a5617d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233076
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
As we convert repo to support Python 3, the version of Python that we
use might not be the version that repo hooks users have written for.
Since repo upgrades are not immediate, and not easily under direct
control of end users (relative to the projects maintaining the hook
code), allow hook authors to declare the version of Python that they
want to use.
Now repo will read the shebang from the hook script and compare it
against the version of Python repo itself is running under. If they
differ, we'll try to execute a separate instance of Python and have
it load & execute the hook. If things are compatible, then we still
use the inprocess execution logic that we have today.
This allows repo hook users to upgrade on their own schedule (they
could even upgrade to Python 3 ahead of us) without having to worry
about their supported version being exactly in sync with repo's.
Bug: https://crbug.com/gerrit/10418
Change-Id: I97c7c96b64fb2ee465c39b90e9bdcc76394a146a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228432
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
A new option, --partial-clone is added to 'repo init' which tells repo
to utilize git's partial clone functionality, which reduces disk and
bandwidth usage when downloading by omitting blob downloads initially.
Different from restricting clone-depth, the user will have full access
to change history, etc., as the objects are downloaded on demand.
Change-Id: I60326744875eac16521a007bd7d5481112a98749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
This fix exception with python3 with stack-trace:
error: Cannot fetch platform_external_grpc-grpc-java.git (UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 640: invalid start byte)
[...]
File "[...]project.py", line 2255, in _IsValidBundle
if f.read(16) == '# v2 git bundle\n':
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
Even if we ask 16 characters, python buffered decoder will try to decode more in the buffer
The patch works for python2 and python3, and open the file in byte mode so that decoding is not attemped
Signed-off-by: Pierre Tardy <tardyp@gmail.com>
Change-Id: I837ae3c5cd724b34670fc2a84e853431f482b20d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/224642
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Currently we read the binary stream from the subprocess code directly
before waiting for it to finish, but there's no need to do so as we
aren't streaming the output to the user. This also means we pass up
binary data to the caller as we don't go through GitCommand's internal
logic which decodes the stream as utf-8.
Simplify the code by calling Wait first, then splitting the entire
captured output in one line.
Bug: https://crbug.com/gerrit/10418
Change-Id: I7a57904be8cb546a229980fb79c829fc3df31e7d
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.
Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
During sync, repo runs `git read-tree --reset -u -v HEAD` which causes
git-lfs's smudge filter to run, which fails because git-lfs does not
work with bare repositories.
This was fixed in I091ff37998131e2e6bbc59aa37ee352fe12d7fcd to
automatically disable this smudge filter. However, later versions of
Git (2.11.0) introduced a new filter protocol [1], to avoid spawning
a new command for each filtered file. This was implemented in Git-LFS
1.5.0 [2].
This patch fixes the issue by setting the git lfs process filter, in
addition to the smudge filter. For any projects that have LFS objects,
`git lfs pull` must still be executed manually afterwards.
[1] edcc85814c
[2] https://github.com/git-lfs/git-lfs/pull/1617
Bug: https://crbug.com/gerrit/10911
Change-Id: I277fc68fdefc91514a2412b3887e3be9106cab48
When syncing a lot of projects in parallel, it is not otherwise
clear which one of them has failed to init work tree.
Change-Id: I8edfb4955023389a499e99cfa511bdc0d2850ba2
This allows projects to include submodules inside of
projects that use repo without repo incorrectly believing
the area is dirty just because a submodule has updates.
This is in line with git porcelain commands which generally
require a commandline flag to include submodules (git add,
git rebase).
Change-Id: Ide8a292162a42ab35145b5c4ca8ca0d020cdfe81
The newly introduced "Already have persistent ref" message prevents
repo from overwriting the last line when syncing quietly. Omit the
message when syncing quietly to clean up the output and to restore
the previous behaviour.
Change-Id: Idf42751c67f95924d6de50092ba54d4c6fe93096
If a tag is rewritten on the server (which is bad), trying to recover
locally with `repo sync --force-sync` doesn't actually work. People
have to manually delete things themselves to fix syncing. While tags
should never be rewritten in practice, allow users to easily recover
from broken servers.
We updated some of these code paths already (see commit 6e53844f1e
"Allow clobbering of existing tags from remote."), but the incremental
update flow was missed.
Bug: b/120778183
Bug: chromium:932651
Test: delete local tag & recreate to diff commit, then check
`repo sync` & `repo sync --force-sync` behavior
Change-Id: I3648f7d2526732c06016b691a9a36c003157618d
Even if dots are used as separators for Git config keys, they are not
forbidden as part of submodule names. This fixes the issue of submodules
with a name like e.g. "long.js" to be skipped from checkout.
Change-Id: I77da07925ad207fa3d043067dfbbcb4a1ebdac4d
When --ne/--no-emails is added to 'repo upload' command line, gerrit
server will not generate notification emails.
project.py:Project.UploadForReview method is modified to accept a
string recognizable by gerrit to indicate different sets of destination
email addressees, but the upload command line allows only one option -
disable sending emails completely.
Default repo upload behavior is not being changed.
TEST=tried in the Chrome OS repo, observed that patches uploaded with
--ne or --no-emails indeed do not trigger any emails, while
patches uploaded without these command line options still trigger
email notifications.
Change-Id: I0301edec984907aedac277d883bd0e6d3099aedc
Pass options through the refspec for all transports, including ssh.
This means the behavior will be more consistent between the ssh and
https cases.
A downside is that this prevents passing special characters in
reviewer options. That already didn't work over https, so it seems
okay. It could be fixed by using push options instead.
Change-Id: Ia38d16e350cb8cb0de14463bfb3d9724e13bc4bf
* Add more file i/o wrappers in platform_utils to allow using
long paths (length > MAX_PATH) on Windows.
* Paths using the long path syntax ("\\?\" prefix) should never
escape the platform_utils API surface area, so that this
specific syntax is not visible to the rest of the repo code base.
* Forward many calls from os.xxx to platform_utils.xxx in various place
to ensure long paths support, specifically when repo decides to delete
obsolete directories.
* There are more places that need to be converted to support long paths,
this commit is an initial effort to unblock a few common use cases.
* Also, fix remove function to handle directory symlinks
Change-Id: If82ccc408e516e96ff7260be25f8fd2fe3f9571a
"repo init --reference" has two purposes: to decrease bandwidth used
at clone time, and to decrease disk usage afterward, by using the
reference repositories as an alternate store of objects even after
the clone. The downside is that it makes the borrowing repositories
dependent on the reference repositories, so it is easy to end up
with missing objects by mistake after a cleanup operation like "git
gc".
To prevent that, v2.3.0-rc0~30^2 (clone: --dissociate option to mark
that reference is only temporary, 2014-10-14), "git clone" gained a
--dissociate option that makes --reference reuse objects from the
reference repository at clone time but copy them over instead of
using the reference as an alternate. This is more straightforward to
use than plain --reference, at the cost of higher disk usage.
Introduce a --dissociate to "repo init" that brings the same benefits
to repo. The option is simply passed on to "git clone".
Change-Id: Ib50a549eb71e0a2b3e234aea57537923962a80d4
pylint is not used since bb5b1a0. The pyflakes cleanup mentioned in that
commit has not been done, but given that this project is no longer being
actively developed I don't think it's worth spending time doing it.
Leaving the pylint suppressions causes confusion because it leads people
to think that we are still using pylint.
Change-Id: If7d9f280a0f408c780f15915ffdb80579ae21f69
When someone does "repo download -c <project> <change>"
without specifying a patch number, by default patch 1 is
downloaded. An alternative is to look for the latest patch
and download the same when no explicit patch is given.
This commit does the same by identifying the latest patch
using "git ls-remote".
Change-Id: Ia5fa7364415f53a3d9436df4643e38f3c90ded58
Allow the 'default' and 'project' element in the manifest
file to apply "--no-tags" option equivalent.
Change-Id: I7e0f8c17a0e25cca744d45df049076d203c52ff5
Signed-off-by: YOUNG HO CHA <ganadist@gmail.com>
Put the correctly-expanded relative paths in objects/info/alternates.
From gitrepository-layout(5), this path should be "relative to the
object database, not to the repository".
Change-Id: I7b2027ae23cf7d367b80f5a187603c4cbacdb2de
Issue: when subproject url is a relative in .gitmodules
repo tool cannot handle this and cause:
"fatal: '***' does not appear to be a git repository
fatal: Could not read from remote repository."
issue.
Signed-off-by: Shouheng Zhang <shouheng.zhang@intel.com>
Change-Id: I2a24c291ea0074ba13a740b32a11c0c25975e72b
os.remove raises an exception when deleting read-only files on
Windows. Replace all calls with calls to platform_utils.remove,
which deals with deals with that issue.
Change-Id: I4dc9e0c9a36b4238880520c69f5075eca40f3e66
Without this change, '.git\HEAD' files, for examples, are sometime
read incorrectly resulting in the current branch to be reset to
"master" when running a "repo init -b xxx" on an already initialized
repository.
Change-Id: I48c7ef85ff81626edf156914329a560e14252f2a
* changes:
Port os.rename calls to work on Windows
Workaround shutil.rmtree limitation on Windows
Add support for creating symbolic links on Windows
Make "git command" and "forall" work on Windows
The output indicates that fetching happens even when it is skipped.
To avoid confusion, print a message when fetching is skipped for
an immutable ref so that the user knows when and why a fetch is skipped.
Change-Id: Id6e4812cebc5e57d379feb76a9d034af0b93043b
This option allow to bypass verification ssl certification while
establishing connection with Gerrit to upload review.
Change-Id: If2e15f5a273c18a700eb5093ca8a4d5a4cbf80cd
This reverts commit d88f53e2b9. I merged
it too hastily without paying enough attention to compatibility with
released Gerrit versions.
Change-Id: I4028d4737df1255f11e217da183a19a010597d5b
Considering that some users might expect changes created with
'-d' option are not public. Private changes may be a better
choice here than work-in-progress changes.
Change-Id: I46a8fb9ae38beb41cf96d6abe82bea6db2439669
This change adds options for git-repo tool to support private
changes and work-in-progress changes.
Change-Id: I343491f5949f06f1580d53f9cc0dee2dca09130f
The submodule argument to Sync_LocalHalf was missing in
MetaBranchSwitch, causing submodules not to get synced when the
-b/--manifest-branch argument to init is used.
Change-Id: Ie86d271abac2020725770be36ead83be3326e64b
Signed-off-by: Martin Kelly <mkelly@xevo.com>
os.rename fails on Windows if the destination exists, so replace
os.rename to platform_utils.rename which handles the platform
differences.
Change-Id: I15a86f10f65eedee5b003b80f88a0c28a3e1aa48
By default, shutil.rmtree raises an exception when deleting readonly
files on Windows.
Replace all shutil.rmtree with platform_utils.rmtree, which adds an
error handler to make files read-write when they can't be deleted.
Change-Id: I9cfea9a7b3703fb16a82cf69331540c2c179ed53
Replace all calls to os.symlink with platform_utils.symlink.
The Windows implementation calls into the CreateSymbolicLinkW Win32
API, as os.symlink is not supported.
Separate the Win32 API definitions into a separate module
platform_utils_win32 for clarity.
Change-Id: I0714c598664c2df93383734e609d948692c17ec5
Project names are stored as path using the '/' file separator, and
stored in a dictionary as keys.
Change-Id: Ide40dfe840958ac0d46caae5f77f1a49d71c9d90
repo sync can sync submodules via the --fetch-submodules option.
However, if the manifest repo has submodules, those will not be synced.
Having submodules in the manifest repo -- while not commonly done -- can
be useful for inheriting a manifest from another project using <include>
and layering changes on top of it. In this way, you can avoid having to
deal with merge conflicts between your own manifests and the other
project's manifests (for example, if you're managing an Android fork).
Add a --submodule option to init that automatically syncs the submodules
in the manifest repo whenever the manifest repo changes.
Change-Id: I45d34f04517774c1462d7f233f482d1d81a332a8
Signed-off-by: Martin Kelly <mkelly@xevo.com>
Currently, if direct fetch of a sha1 is not supported by git server and
depth option is used, we fallback on syncing the upstream branch by
ignoring depth option.
This fallback doesn't work in next 2 cases:
(1) upstream attribute is not specified in manifest
(2) depth option is passed to repo init command
(not with clone-depth attribute in manifest)
This commit do the following:
- fixes (1) by updating condition used to apply fallback
first we retry with depth set to None, then by syncing all branches
- fixes (2) by passing depth as argument of _RemoteFetch() method
thus, its value is not set again to depth value passed to repo init
command when applying fallback
Change-Id: Ifd6fffafc49ba229df624b0d7b64c83d47619d17
A recent backward incompatible change created confusion and loss of
productivity and highlighted the very limited amount of information
provided when repo sync fails; merely recommending to --force-sync
and blow-up git repos without any hint as to why. The addition of
this basic _error(...) call would have provided a clue and will in
the future.
BUG=Issue 232
TEST=simulate a breakage similar to the ones reported at
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-dev/2-0oCy_CX5s
cd .repo/projects/src/third_party/libapps.git/
file info; rm info; ln -s wronglink info
cd -
repo sync src/third_party/libapps/
# error message now shows the failure
Change-Id: Idd2f177a096f1ad686caa8c67cb361d594ccaa57
The --quiet option reduces the output to just
a list of projects with modified workspaces (and
orphans if -o is specified)
A common use case is when performing a full-workspace
merge. The integrator will kick-off a merge via:
repo forall -c git merge <some tag>
And then produce a short list of conflicted projects via:
repo status -q
The integrator can then iteratively fix and clean up all conficted
components. The merge is complete when:
repo status -q
returns no output.
Change-Id: Ibbba8713eac35befd8287c95948874e23fd5c7e2
Allow the 'remote' element in the manifest file to define an optional
'pushurl' attribute which is passed into the .git/config file.
Change-Id: If342d299d371374aedc4440645798888869c9714
Signed-off-by: Steve Rae <steve.rae@raedomain.com>
Instead of
Do you want to allow this script to run (yes/yes-never-ask-again/NO)? (yes/always/NO)?
ask
Do you want to allow this script to run (yes/always/NO)?
Change-Id: I5f5a2d0e88086a8d85e54fb8623a62d74a20956a
Signed-off-by: Jonathan Nieder <jrn@google.com>
The constant prompting when registered hooks change can be tedious and
has a large multiplication factor when the project is large (e.g. the
AOSP). It gets worse as people want to write more checks, hooks, docs,
and tests (or fix bugs), but every CL that goes in will trigger a new
prompt to approve.
Let's tweak our trust model when it comes to hooks. Since people start
off by calling `repo init` with a URL to a manifest, and that manifest
defines all the hooks, anchor trust in that. This requires that we get
the manifest over a trusted link (e.g. https or ssh) so that it can't
be MITM-ed. If the user chooses to use an untrusted link (e.g. git or
http), then we'll fallback to the existing hash based approval.
Bug: Issue 226
Change-Id: I77be9e4397383f264fcdaefb582e345ea4069a13
During sync, repo runs `git read-tree --reset -u -v HEAD` which causes
git-lfs's smudge filter to run. However this fails because git-lfs does
not work with bare repositories.
Add lfs.filter configuration to the project config as suggested in the
comments on the upstream git-lfs client issue [1]. This prevents the
smudge filter from running, and the sync completes successfully.
For any projects that have LFS objects, `git lfs pull` must be executed.
[1] https://github.com/github/git-lfs/issues/1422
Bug: Issue 224
Change-Id: I091ff37998131e2e6bbc59aa37ee352fe12d7fcd
Re-ordered to first create the symlink before checking the source
file and remove the destination if the source does not exists.
Change-Id: Iae923ba2ef0ba5a8dc1b8e42d8cc3f3708f773af
If upstream string is empty, current_branch_only variable will be assigned
to an empty string.
This is not what we expect here as this variable is a boolean.
Change-Id: Ibba935e25a74c2be1e50c88b4b403cf394ba365e
When sync-s="true" option is used, the checkout of a submodule will try
to use the revision attribute of the parent project.
If this revision is a named reference, the checkout will fail if there
is no reference with this name in the submodule.
The proposed solution is to use the git commit id as revisionExpr for
submodules.
Change-Id: Ie8390a11957fd6a9c61289c6861d13cb3fa11678
When the alias attribute is set for a remote, the RemoteSpec attached to
a Project only contains the alias name used by git, not the original
name used in the manifest. But that's not enough information to
reconstruct the manifest, so save off the original manifest name as
another RemoteSpec parameter, only used to write the manifest out.
Bug: Issue 181
Bug: Issue 219
Change-Id: Id7417dfd6ce5572e4e5fe14f22924fdf088ca4f3
The repo script allows a manifest to specify a '.' as the path the
top-level directory, which co-locates the .git and .repo directories,
and places files from the git repository at the top-level:
<project name="proj_name" path="." />
<project name="sierra.other.git" path="other" />
Most commands work correctly with this setup. Some commands, however,
fail to find the project. For instance, 'repo sync' works, and 'repo sync .'
works in a sub-project ('other' in this case) but 'repo sync .' in the
top-level directory fails with the error:
error: project . not found
There are two reasons for this:
1. The self.worktree attribute of the Project object is not normalized,
so with a '.' for path its value would be '/my/project/root/.'. This is
fine when used as a path, since it's the same path as '/my/project/root',
but when used in a string comparison it fails. This commit applies
os.path.normpath() to that value before storing it.
2. The _GetProjectByPath method in command.py was not checking the path
against manifest.topdir, so even once it was normalized the project was
not found. This commit adds a check against manifest.topdir if the
loop drops out without finding a project.
Change-Id: Ic84d053f1bbb5a357cad566805d5a326ae8246d2
We weren't copying these lists, so the += was actually changing the
underlying lists.
When a new project was added to the manifest, we run _CheckDirReference
against the manifest project with share_refs=True, which added the
working_tree_* to the shareable_* lists. Then, when we load the new
manifest and create the new project, it uses the lists that already
contain the working_tree_* files, even though we passed
share_refs=False.
This happens reliably under the above conditions, but doesn't seem to
happen when syncing a fresh tree. So we've got a mixture of links that
may need to be cleaned up later. This patch will just stop it from
happening in the future.
Change-Id: Ib7935bfad78af1e494a75e55134ec829f13c2a41
A common design pattern is to use __file__ to find the location of the
active python module to assist in output or loading of related assets.
The current hook systems runs the pre-upload.py hook in a context w/out
that set leading to runtime errors:
$ repo upload --cbr .
ERROR: Traceback (most recent call last):
File ".../repo/project.py", line 481, in _ExecuteHook
self._script_fullpath, 'exec'), context)
File ".../repohooks/pre-upload.py", line 32, in <module>
path = os.path.dirname(os.path.realpath(__file__))
NameError: name '__file__' is not defined
Define this variable in this context so code can safely use it.
Change-Id: If6331312445fa61d9351b59f83abcc1c99ae6748
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a number of violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the project.py script.
This script now has a pylint score of 10.0, and no violations reported
by pep8.
Change-Id: I1462fd84f5b6b4c0dc893052671373e7ffd838f1
We don't really use HEAD much in the bare git repositories, but there
have been reports of errors in git-symbolic-ref:
symbolic-ref: fatal: Refusing to point HEAD outside of refs/
That happen when the bare git repo is in the detached head state. It's
possible that previous operations were killed while we were pruning
branches.
Use DetachHead instead of SetHead if we're restoring the repo into a
detached head state.
Change-Id: I9062e8957bc70367d3ded399685ac026fbb421fc
If a linkfile is a broken link (destination does not exist), and it
needs to be updated, we didn't notice that it needed to be removed
first. Use lexists instead of exists to check for this condition.
Change-Id: I1f6a1f0193d3fd2b9f7a647836044997f6ab32eb
By passing --prune to the sync command, the --prune option is
given to the `git fetch`, causing refs that no longer exist on
the remote to be removed.
Change-Id: I3cedacce14276d96ac2d5aabf2d07fd05e92bc02
The source or destination attributes may have changed even if the source
didn't, so we need to make sure that these are up to date.
Change-Id: I266ef3598ddda7e8c23bc9c6a049905ddc586348
Add repo start support for GITC checkouts. If the user is in
the GITC FS view, they can now run repo start to check out
the sources and create a new working branch.
When "repo start" is called on a GITC project, the revision
tag is set to an empty string and saved in a new tag:
old-revision. This tells the GITC filesystem to display the
local copy of the sources when being viewed. The local copy
is created by pulling the project sources and the new branch
is created based off the original project revision.
Updated main.py to setup each command's gitc_manifest when
appropriate.
Updated repo sync's logic to sync opened projects and
updating the GITC manifest file for the rest.
Change-Id: I7e4809d1c4fc43c69b26f2f1bebe45aab0cae628
If a hook file has been modified locally, it will not be replaced.
Improve the message to make this clearer.
Also change it from an error to a warning.
Change-Id: I62c635390f24d2868db17717c247861b0381c99f
Use the _error method instead of directly calling `print`.
Also add a new _warn convenience method.
Change-Id: Ia332c14ef8d9d1fe2df128dbf36b5521802ccdf1
Use the same cookies and proxy that git traffic goes through for
persistent-http[s] to support authentication for smart-sync.
Change-Id: I20f4a281c259053a5a4fdbc48b1bca48e781c692
Passing the force_sync variable into the string formatting results in
the message:
"Retrying clone after deleting None"
or
"Retrying clone after deleting True".
Pass the name of the git directory instead.
Also, move the print inside the if-block so it's only displayed
when the retry is actually going to be attempted.
Change-Id: I76d9ecc176cecee4ad512d13e9d1f6bd36aacbbb
For projects that have been cloned outside of the repo command (or
cloned a long time ago), commit abaa7f312f
introduced an error message to invite the user to use --force-sync.
However, due to the risk of data loss, it's useful to know which
project's git directory is being replaced before deciding whether or not
to provide --force-sync.
This change updates the exception's associated value to include the
project's relative path and explain to the user how they can resolve the
issue. A previous version of this commit used the project name. However,
for projects that have multiple work trees, the name can be ambiguous,
while the path clearly identifies which git directory will be replaced.
Change-Id: If717e66fda4d19accc0a8e889a91f4cd4ff14dff
The existing code here makes sure that switching clone-depth from on to
off actually causes the history to be fully restored. Unfortunately, it
does this by fetching the full history every time the fetch spec
changes. Switching between two clone-depth="1" branches will fetch far
more than the top commit.
Instead, when not using clone-depth, pass --depth=2147483647 to git
fetch so that it ensures that we have the entire history. That is
slightly less efficient, so limit it to only when there are shallow
objects in the project by checking for the existance of the 'shallow'
file.
Change-Id: Iee0cfc9c6992c208344b1d9123769992412db67b
This fixes these errors:
...
File ".repo/repo/project.py", line 2371, in _ReferenceGitDir
os.symlink(os.path.relpath(src, os.path.dirname(dst)), dst)
OSError: [Errno 17] File exists
Which was happening for checkouts that were created before v1.12.8, when
project-objects was created. Nothing had yet been forcing these
checkouts to use project-objects, until the recent verification changes.
In this OSError case, we already created the symlink, so src == dst, and
the directory did not exist. This caused us to run os.makedirs the
os.symlink on the same file.
dst really should be the file in gitdir, not the target of that symlink
if it exists. So just use realpath for the dotgit portion of the path.
Change-Id: Iff5396a2093de91029c42cf38aa57131fd22981c
In some cases, a user may wish to continue with a sync even though
it would require overwriting an existing git directory. This behavior
is not safe as a default because it could result in the loss of some
user data, but as an optional flag it allows the user more flexibility.
To support this, add a --force-sync flag to the sync command that will
attempt to overwrite the existing git dir if it is specified and the
existing git dir points to the wrong obj dir.
Change-Id: Ieddda8ad54e264a1eb4a9d54881dd6ebc8a03833
If _InitGitDir fails, it leaves any progress it had made on the file
system. This can cause subsequent calls to repo sync to behave
differently. This is especially evident when _CheckDirReference() fails,
since it will not be invoked when sync is retried because both the
source and destination directories already exist.
To address this, have _InitGitDir() clean up any directories it has created
if it catches an exception. Also behave the same way for _InitWorkTree().
Change-Id: Ic16bb3feea649e115b59bd44be294e89e3692aeb
For some users it is not desirable to remove refs that don't exist
on the remote server when syncing a mirror repo.
This reverts commit b4d43b9f66.
Change-Id: Ie849b66682138ef88da6cd1a5fbb27e993197dd7
The fetch logic for the case where depth is set and revision is a
SHA1 has several failure modes that are not handled well by the
current logic.
1) 'git fetch <SHA1>' requires git version >= 1.8.3
2) 'git fetch <SHA1>' can be prevented by a configuration option on the server.
3) 'git fetch --depth=<N> <refspec>' can fail to contain a SHA1 specified by
the manifest.
Each of these cases cause infinite recursion when _RemoteFetch() tries to call
itself with current_branch_only=False because current_branch_only is set to
True when depth != None.
To try to prevent the infinite recursion, we set self.clone_depth to None
before the first retry of _RemoteFetch(). This will allow the Fetch to
eventually succeed in the case where clone-depth is specified in the manifest.
A user specified depth from the init command will still recurse infinitely.
In addition, never try to fetch a SHA1 directly if the git version being used
is not at least 1.8.3.
Change-Id: I802fc17878c0929cfd63fff611633c1d3b54ecd3
This allows a project to use globs in the linkfile src attribute. When
a glob is used in the src the dest field must be a directory. Then
_LinkFile._Link(self) calls will create symbolic links in the dest
directory to all of the entries in the src as defined by the glob
specification.
Below all of the entries in master-configs/ will have symbolic links
in <root dir>/configs directory:
<project name="helloworld.git" path="apps/helloworld">
<linkfile src="master-configs/*" dest="configs"/>
</project>
Change-Id: Idfed8fa47c83d2ca6e2b8e867731b8e2f9e2eb47
The source (target) of the symlink is specified relative to a project
within a tree, and the destination is specified relative to the top
of the tree, so it should always be possible to create a relative symlink
to the target file. Relative symlinks will allow moving an entire tree
without breaking the symlink, and copying a tree (with -p) without leaving
a symlink to the old tree.
Change-Id: I16492a8b59a137d2abe43ca78e3b212e2c835599
Pressing ctrl-c during repo sync often hangs for 30 to 45 seconds
due to the time.sleep and retry in _RemoteFetch. If git exits with
a signal, for example -2 for SIGINT triggered by ctrl-c, skip the
sleep and retry.
Change-Id: I32da12c2dcc96d9cc0b12a066e824b12ebfb52a0
There are a set of cases that can cause the git directory in
.repo/projects to point to a directory in .repo/project-objects that
is not the one specified in the manifest. This results in a tree that
is not sane, and so should cause a failure.
In order to reproduce the failure case:
1) Sync to any manifest
2) Change the 'name' of a project to a different repository. Leave the
'path' the same.
3) Resync the modified project. The project-objects directory will not
be created, and the projects directory will remain pointed at the old
project-objects.
Change-Id: Ie6711b1c773508850c5c9f748a27ff72d65e2bf2
In 2fb6466f79 an optimisation was
added to avoid fetching from remotes if the project is fixed to
a revision and the revision is already available locally.
This causes problems for users who expect all objects to be
fetched by default.
Change the logic so that the optimized behaviour is only enabled if
an option is explicitly given to repo sync.
Change-Id: I3b2794ddd8e0071b1787e166463cd8347ca9e24f
When syncing a mirror repo, add the --prune option to the fetch
command to force removal of stale refs from the mirror.
Change-Id: I4b43b2a5c86b9915627887c16f6569066f3ab978
Appending the branch to the fetch spec causes sync of a mirror to
fail for projects that don't have an explicit revision specified,
and don't have a branch of the same name as the default revision.
For example, a manifest defining a default revision:
<default revision="master">
having a project without an explicit revision:
<project name="path/to/project">
and not having a branch named "master", will cause repo sync to
fail for that project with the error:
Couldn't find remote ref refs/heads/master
Modify the logic to not append the branch onto the fetch spec when
syncing to a mirror.
Change-Id: I5c4457bd125519abf27abe682dea62ad708978c9
When running repo branch, the git merge line (in many circumstances)
is set to the revision of the project specified in the manifest. If
this is a branch name that is not fully-qualified, we will end up with
something like "merge = master" instead of "merge = refs/heads/master".
This change examines the revision if we are going to use that and
changes branch short names to fully qualified branch names.
Change-Id: Ie1be94fb8d45df8eeac44a47f729a3819a05fa81
Switch the GitCommand program to always capture the output for stdout
and stderr. And by default print the output while running.
The options capture_stdout and capture_stderr have effectively become
options to supress the printing of stdout and stderr.
Update the 'git fetch' to use '--progress' so that the progress messages
will be displayed. git checks if the output location isatty() and if it
is not a TTY it will by default not print the progress messages.
Change-Id: Ifdae138e008f80a59195f9f43c911a1a5210ec60
This reverts commit 38e4387f8e.
A "repo init" followed by "repo sync" is meant to be as safe as
"git clone". In particular it should not run arbitrary code provided
by the manifest owner.
It would still be nice to have support for manifest-defined git hooks
--- they'd just need a prompt like the upload RepoHook has. Hopefully
a later change can bring them back.
Change-Id: I5ecd90fb5c2ed64f103d856d1ffcba38a47b062d
Signed-off-by: Jonathan Nieder <jrn@google.com>
Currently, paths longer than 39 chars have no space after them so it looks
like this:
project path/branch master
Change-Id: I4c1bb13648ac099ade8a8d4ebafa04131571f842
The stderr output generated by git during a RemoteFetch was not being
printed. This information is useful so print it.
Change-Id: I6e6ce12c4a57e5ca2359f76ce14f2fcbbc37a5ef
If the repo project is synced with partial depth, then the tags
won't be fetched and users will be told the newest sha1 in the
stable branch isn't signed.
Change-Id: I107df97b4836b928c76aa33a700fa35d1705ae09
Handle the case when this error occurs:
error: some local refs could not be updated; try running
'git remote prune origin' to remove any old, conflicting branches
This is usually caused by a reference getting changed from a file to a
directory.
For example:
Initially someone creates a branch 'foo' and it is stored as:
.git/refs/remotes/origin/foo
Then later on it is decided to change the layout structure where 'foo'
is a directory with branches below it:
.git/refs/remotes/origin/foo/master
The problem occurs when someone still has
'.git/refs/remotes/origin/foo' on their system and does a repo sync.
When this occurs the error message for needing to do a
'git remote prune origin' occurs.
Now when doing a 'git fetch' if the error message from git says that a
'git remote prune' is needed, it will do the prune and then retry the
fetch.
Change-Id: I4c6f5aa6bd932f0ef7a39134400bedd52e82f633
Signed-off-by: John L. Villalovos <john.l.villalovos@intel.com>
When working within a team or corporation it is often
useful/required to use predefined git templates. This
change teaches repo to use a per-remote git hook template
structure.
The implementation is done as a continuation of the
existing projecthook functionality. The terminology is
therefore defined as projecthooks.
The downloaded projecthooks are stored in the .repo
directory as a metaproject separating them from the users
project forest.
The projecthooks are downloaded and set up when doing a
repo init and updated for each new repo init.
When downloading a mirror the projecthooks gits are
not added to the bare forest since the intention is to
ensure that the latest are used (allows for company policy
enforcement).
The projecthooks are defined in the manifest file in the
remote element as a subnode, the name refers to the
project name on the server referred to in the remote.
<remote name="myremote ..>
<projecthook name="myprojecthookgit" revision="myrevision"/>
</remote>
The hooks found in the projecthook revision supersede
the stock hooks found in repo. This removes the need for
updating the projecthook gits for repo stock hook changes.
Change-Id: I6796b7b0342c1f83c35f4b3e46782581b069a561
Signed-off-by: Patrik Ryd <patrik.ryd@stericsson.com>
Signed-off-by: Ian Kumlien <ian.kumlien@gmail.com>
Currently, we only use the depth flag when cloning. The result is that when
new project history has merges, the entire history of the merged branch is
brought in and the project becomes unshallow very quickly. --depth and
clone-depth are often used to save on space, not just network load, so this
seems less than ideal.
This change uses --depth on every fetch (when the user has depth specified),
not just the initial clone. The result is that the given project stays
consistently shallow as opposed to growing over time, especially when merges
are involved.
Change-Id: Iac706cfdad4a555c72f9d9f1119195d38d91df12
When doing a shallow checkout SHA1 pinned repos with repo init --depth=1 and
repo sync -c, repo would try to fetch only some reference and fail if the exact
SHA1 repo was missing.
Instead, when depth is set, fetch only the specific commit.
Change-Id: If3f799d0e78c03faea47f796380bb5e367b11998
We currently delete all hooks in .git/hooks for each project before
symlink'ing in the standard project hooks. This can be annoying for
users who have installed custom git hooks.
There's no reason to delete all existing hooks. Just rip out the
deletion code.
Change-Id: I5062a6cd20af700f6d6a17b11ad6c94853987c57
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The persistent proxy may choose to present a per-process cookie file
that gets cleaned up after the process exits, to help with the fact
that libcurl cannot save cookies atomically when a cookie file is
shared across processes. We were letting this cleanup happen
immediately by closing stdin as soon as we read the configuration
option, resulting in a nonexistent cookie file by the time we use the
config option.
Work around this by converting the cookie logic to a context manager
method, which closes the process only when we're done with the cookie
file.
Change-Id: I12a88b25cc19621ef8161337144c1b264264211a
The invalid clone.bundle file warning is not typically user actionable,
and can be confusing. So don't show it when -q flag is in effect.
Change-Id: If9fef4085391acf54b63c75029ec0e161c38eb86
This reverts commit 565480588d.
We are reverting this change for 2 reasons:
1) It introduced a bug for users using sync -c with a reference mirror.
2) The fetch specs have recently changed to cause git to properly fail
when we request a non-existent branch of a manifest, removing the need
for this change.
Change-Id: I0f63da9bfb40cf5ffafb7979f1b8c929a738fc7b
When there are uncommitted files in the tree, 'repo upload' stops to
ask if it is OK to continue, but does not report the actual names of
uncommitted files.
This patch adds plumbing to have the outstanding file names reported
if desired.
BUG=None
TEST=verified that 'repo upload' properly operates with the following
conditions present in the tree:
. file(s) modified locally
. file(s) added to index, but not committed
. files not known to git
. no modified files (the upload proceeds as expected)
Change-Id: If65d5f8e8bcb3300c16d85dc5d7017758545f80d
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Signed-off-by: Vadim Bendebury <vbendeb@google.com>
While not typical, some users might have an upstream that isn't in
the usual refs/heads/* namespace. There's no reason not to use
those refs as the value for the upstream attribute, so support
doing so.
Change-Id: I5b119f1135c3268c20e7c4084682e860d3ee1fb1
If a user reinits to a different manifest or the manifest updates so
that a project no longer has a fixed depth, we need to use --unshallow
when we fetch.
Change-Id: I6d3f15e5464b5eaad9205654bc24354947a78aea
Some projects use multiple remotes.
In some cases these remotes have different naming conventions.
Add an option to define a revision in the remote configuration.
The `project` revision takes precedence over `remote` and `default`.
The `remote` revision takes precedence over `default`.
The `default` revision acts as a fall back as it originally did.
Change-Id: I2b376160d45d48b0bab840c02a3eef1a1e32cf6d
iterator.next() was replaced with iterator.__next__() in Python 3.
Use next(iterator) instead which will select the correct method for
returning the next item.
Change-Id: I6d0c89c8b32e817e5897fe87332933dacf22027b
A recent optimization (2fb6466f79) skips
performing a remote fetch if we already know we have the sha1 we want.
However, that optimization skipped initialization steps that ensure HEAD
points to the correct sha1. This change makes sure not to skip those
steps.
Here is an example of how to test this change:
"""""""""
url=<manifest url>
branch1=<branch name>
branch2=<branch name>
project=<project with revision set to different sha1 in each branch>
repo init -u $url -b $branch1 --mirror
repo sync $project
first=$(cd $project.git; git rev-parse HEAD)
repo init -b $branch2
repo sync $project
second=$(cd platform/build.git; git rev-parse HEAD)
if [[ $first == $second ]]
then
echo 'problem!'
else
echo 'no problem!'
fi
"""""""""
This fixes the bug that kept clients from doing things like `git log`
in projects using the clone-depth feature.
Change-Id: Ib4024a7b82ceaa7eb7b8935b007b3e8225e0aea8
It's just like copyfile and runs at the same time as copyfile but
instead of copying it creates a symlink instead. This is needed
because copyfile copies the target of the link as opposed to the
symlink itself.
Change-Id: I7bff2aa23f0d80d9d51061045bd9c86a9b741ac5
If a manifest includes projects with a clone-depth=1 attribute, and a
workspace is initialised from that manifest using the --mirror option,
any workspaces initialised and synced from the mirror will fail with:
fatal: attempt to fetch/clone from a shallow repository
on the projects that had the clone-depth.
Ignore the clone-depth attribute when fetching from the remote to a
mirror workspace. Thus the mirror will be synched with a complete
clone of all the repositories.
Change-Id: I638b77e4894f5eda137d31fa6358eec53cf4654a
Currently, the --no-tags option is ignored if the user asks to only
fetch the current branch. There is no reason for this restriction. Fix
it.
Change-Id: Ibaaeae85ebe9955ed49325940461d630d794b990
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The old "manifest required for this command -- please run
init" is replaced by a more helpful message that lists the
command repo was trying to execute (with arguments) as well
as the str() of the NoManifestException. For example:
> error: in `sync`: [Errno 2] No such file or directory:
> 'path/to/.repo/manifests/.git/HEAD'
> error: manifest missing or unreadable -- please run init
Other failure points in basic command parsing and dispatch
are more clearly explained in the same fashion.
Change-Id: I6212e5c648bc5d57e27145d55a5391ca565e4149
In existing workspaces where the manifest specifies a commit id in the
manifest, we can avoid doing a fetch from the remote if we have the
commit locally. This substantially improves sync times for fully
specified manifests.
Change-Id: Ide216f28a545e00e0b493ce90ed0019513c61613
This command allows a deeper diff between two manifest projects.
In addition to changed projects, it displays the logs of the
commits between both revisions for each project.
Change-Id: I86d30602cfbc654f8c84db2be5d8a30cb90f1398
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
When we do an initial fetch and have not specified any branch etc,
the following fetch command will not error:
git fetch origin --tags +refs/heads/*:refs/remotes/origin/*
In this change we make sure something got fetched and if not we report
an error.
This fixes the bug that occurs when we init using a bad manifest url and
then are unable to init again (because a manifest project has been
inited with no manifest).
Change-Id: I6f8aaefc83a1837beb10b1ac90bea96dc8e61156
Fetching a new branch on a shallow client may download the entire
project history, as the depth parameter is not passed to git
fetch. Force the fetch to only download the current branch.
Change-Id: Ie17ce8eb5e3487c24d90b2cae8227319dea482c8
This significantly reduces sync time and used brandwidth as only
a tar of each project's revision is checked out, but git is not
accessible from projects anymore.
This is relevant when git is not needed in projects but sync
speed/brandwidth may be important like on CI servers when building
several versions from scratch regularly for example.
Archive is not supported over http/https.
Change-Id: I48c3c7de2cd5a1faec33e295fcdafbc7807d0e4d
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
If the top-level .repo directory is moved somewhere else (e.g. a
different drive) and replaced with a symlink, _InitHooks() will create
broken symlinks. Resolving symlinks before computing the relative path
for the symlink keeps the path within the repo tree, so the tree can
be moved anywhere.
Change-Id: Ifa5c07869e3477186ddd2c255c6c607f547bc1fe
If git-remote-persistent-https fails, we use an iter() and then
subsequently a .read() on stderr. Python doesn't like this and
gives the following error message:
ValueError: Mixing iteration and read methods would lose data
This change removes the use of iter() to avoid the issue.
Change-Id: I980659b83229e2a559c20dcc7b116f8d2476abd5
* Add .decode('utf-8') where needed
* Add 'b' to `open` where needed, and remove where unnecessary
Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
It is often useful to be able to include the same project more than
once, but with different branches and placed in different paths in the
workspace. Add this feature.
This CL adds the concept of an object directory. The object directory
stores objects that can be shared amongst several working trees. For
newly synced repositories, we set up the git repo now to share its
objects with an object repo.
Each worktree for a given repo shares objects, but has an independent
set of references and branches. This ensures that repo only has to
update the objects once; however the references for each worktree are
updated separately. Storing the references separately is needed to
ensure that commits to a branch on one worktree will not change the
HEAD commits of the others.
One nice side effect of sharing objects between different worktrees is
that you can easily cherry-pick changes between the two worktrees
without needing to fetch them.
Bug: Issue 141
Change-Id: I5e2f4e1a7abb56f9d3f310fa6fd0c17019330ecd
* Previously, it would run `git fetch <remote.name> <change refspec>
<remote.fetch>, which would fetch all the branches, even if 'sync-c'
was set to true in the manifest.
* Fix that, since all it needs to fetch is the change that was asked
for, and nothing else.
* For some more info, refer to the discussion on:
I42a9d419b51f5da03f20a640ea68993cda4b6500
Change-Id: Ibc801695d56fc16e56f999e0f61393f54461785f
git-remote-persistent-https proxy implementations may pass cookie file
configuration to git-remote-https. When fetching bundles for
persistent-http(s) URLs, use the -print_config flag (if supported) to
extract this information from the proxy binary and pass it to curl,
overriding http.cookiefile from .gitconfig.
This adds a few ms overhead per clone.bundle fetch, which should be
acceptable since it happens only on the initial clone, which takes
much longer anyway.
Change-Id: I03be8ebaf8d3005855d33998cd8ecd412e8ec287
Server auth middleware may return a 200 from a clone.bundle request
that is not a bundle file, but instead a login or access denied page.
Instead of just checking the file size, actually check the first few
bytes of the file to ensure it is a bundle file before proceeding.
Change-Id: Icea07567c568a24fd838e5cf974c58f9e4abd7c0
This adds the ability to have reviews pushed to a different branch
from the one on which changes are based. This is useful for "gateway"
systems without smartsync.
Change-Id: I3a8a0fabcaf6055e62d3fb55f89c944e2f81569f
Add a new module with methods for checking the Python version.
Instead of handling Python3 imports with try...except blocks, first
check the python version and then import the relevant modules. This
makes the code a bit cleaner and will result in less diff when/if we
remove support for Python < 3 later.
Use the same mechanism to handle `input` vs. `raw_input` and add
suppression of pylint warnings caused by redefinition of the built-in
method `input`.
Change-Id: Ia403e525b88d77640a741ac50382146e7d635924
Also-by: Chirayu Desai <cdesai@cyanogenmod.org>
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
When running 'repo init --reference=<mirror>', the mirror will be
used for all projects except the manifest project. This is because
the _InitGitDir function uses the 'repo.reference' git config
value specified in the manifest git, which has no effect when
creating the manifest git as that value will be set after the git
has been successfully cloned.
Information about where the manifest git is located on the server
is only known when performing the 'repo init', so that information
has to be provided when cloning the git in order for it to set up
a proper mapping.
Change-Id: I47a2c8b3267a4065965058718ce1def4ecb34d5a
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
Fetching all tags for a shallow git results in an
inconstent git and forces git to fetch more than
the depth specified.
This change teaches repo not to fetch any tags in a
repository initialised with the depth option.
Change-Id: I557ead9f88fa0d6a19b1cb55b23bba1100fcb8f2
Signed-off-by: Patrik Ryd <patrik.ryd@stericsson.com>
