When repo syncs a manifest that utilizes multiple branches
in the same project, then the sync will use an extra
thread for each "duplicate". For example, if
the manifest includes the project "foo" and "bar"
twice, then "repo sync -jN" will fetch with N+2 threads.
This is caused by _FetchHelper() releasing the thread semaphore
object each time it's called, even though _FetchProjectList()
may call this function multiple times within the scope of a
single thread.
Fix by moving the thread semaphore release to
_FetchProjectList(), which is only called once per thread
instance.
Change-Id: I1da78b145e09524d40457db5ca5c37d315432bd8
When there's a symlink to a directory, os.walk still lists the symlink
in dirs, even if it isn't configured to follow symlinks. This will fail
the listdirs check if the symlink is broken (either before or during the
cleanup). So instead, check for directory symlinks and remove them using
os.remove.
Bug: Issue 231
Change-Id: I0ec45a26be566613a4a39bf694a3d9c6328481c2
When there are nested projects in a manifest, like on AOSP right now:
<project path="build" name="platform/build" />
<project path="build/blueprint" name="platform/build/blueprint" />
<project path="build/kati" name="platform/build/kati" />
<project path="build/soong" name="platform/build/soong" />
And the top "build" project is removed (or renamed to remove the
nesting), repo just wipes away everything under build/ and re-creates
the projects that are still there. But it only checks to see if the
build/ project is dirty, so if there are dirty files in a nested
project, they'll just be blown away, and a fresh worktree checked out.
Instead, behave similarly to how `git clean -dxf` behaves and preserve
any subdirectories that have git repositories in them. This isn't as
strict as git -- it does not check to see if the '.git' entry is a
readable gitdir, just whether an entry named '.git' exists.
If it encounters any errors removing files, we'll print them all out to
stderr and tell the user that we were unable to clean up the obsolete
project, that they should clean it up manually, then sync again.
Change-Id: I2f6a7dd205a8e0b7590ca5369e9b0ba21d5a6f77
The shared object stores confuse git and make it throw away objects which are
still in use. We'll avoid that problem by disabling automatic pruning on those
projects, but there's nothing preventing a user from changing the config back
or pruning a repository manually.
BUG=chromium:375945
TEST=Ran repo sync on fresh ChromeOS checkout, starting with a branch of repo
with this change. Verified that the kernel projects and no others were
identified as having shared object stores, and that repo successfully disabled
automatic pruning in their configs. Re-enabled pruning and ran repo sync just
on one of the kernel directories. Verified that pruning was re-disabled as a
result.
Change-Id: I728ed5b06f0087aeb5a23ba8f5410a7cd10af5b0
The requirement to explicitly specify the local project when starting
a new repo branch is somewhat counter intuitive.
This patch uses the current directory's git tree as the default
project.
Tested by running
'repo start <name>'
observed that the result is the same as if running
'repo start <name> .'
Change-Id: If106caa801b4cd5ba70dbe8354a227d59f100aa3
When nothing is pending, most of this code is already short-circuited.
Hoist the single check up to make this more obvious/slightly faster.
Change-Id: Iec3a7e08eacd23a7c5f964900d5776bf5252c804
The constant prompting when registered hooks change can be tedious and
has a large multiplication factor when the project is large (e.g. the
AOSP). It gets worse as people want to write more checks, hooks, docs,
and tests (or fix bugs), but every CL that goes in will trigger a new
prompt to approve.
Let's tweak our trust model when it comes to hooks. Since people start
off by calling `repo init` with a URL to a manifest, and that manifest
defines all the hooks, anchor trust in that. This requires that we get
the manifest over a trusted link (e.g. https or ssh) so that it can't
be MITM-ed. If the user chooses to use an untrusted link (e.g. git or
http), then we'll fallback to the existing hash based approval.
Bug: Issue 226
Change-Id: I77be9e4397383f264fcdaefb582e345ea4069a13
Make it possible to exclude projects using regex/wildcard.
The syntax is similar to that of the -r option, e.g.:
repo forall -i ^platform/ ^device/ -c 'echo $REPO_PROJECT'
Change-Id: Id250de5665152228c044c79337d3ac15b5696484
It was only displaying 'Project list error: GitError()'
without any useful info about the project nor the error
Change-Id: Iad66cbaa03cad1053b5ae9ecc90d7772aa42ac13
As soon as we wrote the gitc manifest, the folder for that repo became
empty, causing the next GetProjects lookup to fail. Reorder the
GetProjects calls so that they all happen while we still have the
repository contents available.
If you were already in a subdir, for cases like 'repo start <branch> .',
this would still fail, since the working directory would disappear out
from under you. That's fine most of the time, since we shouldn't be
doing operations based on the local directory, but git has a realpath
function that tries to restore CWD by chdir'ing back to it. So if the
working directory no longer exists, chdir to the topdir before
continuing.
Change-Id: Ibdf6cd37ff6e5a5f8338347c3919175491f7166f
Some teams have a continuous build server that would mark certain
manifest green and safe to sync to. Then team members could repo
sync to that particular manifest file and make sure they always
sync to a green build. But if she/he has some local changes and
wants to rebase, currently it would be a manual process to find the
correct version to rebase onto. This patch helps with that use
case by automating the process to rebase onto the currently synced
manifest version.
Change-Id: I847c9eb6addf7f84fd3f5594fbf8c0bcc103f9a5
When repo sync is used with -f (--force-error) and a project fails to
sync, the sync will continue but then exit with an error status.
However if -n (--network-only) is also used, the exit code is 0, even
when a project failed.
Modify the logic to make sure the sync exits with the correct status.
Bug: Issue 214
Change-Id: I0b5d97a34642c5aa3743750ef14a42c9d5743c1d
By passing --prune to the sync command, the --prune option is
given to the `git fetch`, causing refs that no longer exist on
the remote to be removed.
Change-Id: I3cedacce14276d96ac2d5aabf2d07fd05e92bc02
Adds windows as one of the allowed platforms flags.
Fixes -p foo to append 'platform-foo', instead of each letter (list.extend
expects a list and thus appends each char in the string, rather than the
string itself).
Change-Id: I73a92127ac29a32fc31b335cc54a246302904140
With gitc-init, a gitc client may be specified using '-c'. If we're
not currently in that client, we need to change directories so that
we don't affect the local checkout, and to ensure that repo is
checked out in the new client.
This also makes '-c' optional if already in a gitc client, to match
the rest of the init options.
Change-Id: Ib514ad9fd101698060ae89bb035499800897e9bd
The .gitcookies file generated by googlesource.com does not have
the header:
# (Netscape) HTTP Cookie File
which causes python's MozillaCookieJar.load to fail with the
error:
"does not look like a Netscape format cookies file"
Prepend the expected header onto the generated cookie file.
We don't bother to check if the header already exists on the
file; repeating it does not cause any problem.
Bug: Issue 207
Change-Id: I7d39720a1d36a6aae00f70691156514ebc04e579
repo gitc-delete deletes a GITC client and all the locally
saved sources. Useful for removing unnecessary clients and
recovering disk space.
Change-Id: Idf23addcea52b8713d268c34a7b37da0c5e5cd26
These won't show up as common commands in the help text unless in a GITC
client, and will refuse to execute.
Change-Id: Iffe82adcc9d6ddde9cb4b204f83ff018042bdab0
This way any changes made to the main manifest are reflected in the gitc
manifest. It's also necessary to use both manifests to sync since the
information required to update the gitc manifest is actually in the repo
manifest.
This also fixes a few issues that came up when testing. notdefault
groups weren't being saved to the gitc manifest in a method that matched
'sync'. The merge branch wasn't always being set to the correct value
either.
Change-Id: I435235cb5622a048ffad0059affd32ecf71f1f5b
This way any changes made to the main manifest are reflected in the gitc
manifest. It's also necessary to use both manifests to sync since the
information required to update the gitc manifest is actually in the repo
manifest.
This also fixes a few issues that came up when testing. notdefault
groups weren't being saved to the gitc manifest in a method that matched
'sync'. The merge branch wasn't always being set to the correct value
either.
Change-Id: I5dbc850dd73a9fbd10ab2470ae4c40e46ff894de
Updates the repo launcher and gitc_utils to pull the manifest
directory location out of the gitc config file.
Change-Id: Id08381b8a7d61962093d5cddcb3ff6afbb13004b
The GITC filesystem does not understand relative URLs for remotes,
so now if a remote uses a relative URL, it will be be expanded to
be relative to the manifest URL.
Change-Id: Ie1210758560aeb1934da3f71496aaf19c2728214
Add repo start support for GITC checkouts. If the user is in
the GITC FS view, they can now run repo start to check out
the sources and create a new working branch.
When "repo start" is called on a GITC project, the revision
tag is set to an empty string and saved in a new tag:
old-revision. This tells the GITC filesystem to display the
local copy of the sources when being viewed. The local copy
is created by pulling the project sources and the new branch
is created based off the original project revision.
Updated main.py to setup each command's gitc_manifest when
appropriate.
Updated repo sync's logic to sync opened projects and
updating the GITC manifest file for the rest.
Change-Id: I7e4809d1c4fc43c69b26f2f1bebe45aab0cae628
Don't emit a message when the netrc file doesn't exist or couldn't
be opened.
Instead of trying to unpack the result of info.authenticators() and
catching the resulting TypeError when it's None, first store it to
a local and only unpack it if it has a value.
Also remove an unused import.
Change-Id: I5c404d91e48c261c1ab850c3e5f040c4f4c235cb
Use the same cookies and proxy that git traffic goes through for
persistent-http[s] to support authentication for smart-sync.
Change-Id: I20f4a281c259053a5a4fdbc48b1bca48e781c692
Add repo sync support for GITC checkouts. If the user is in the
GITC client directory they can still pull the sources as normal
if they pass in the --force-gitc argument. Otherwise the user
should call repo sync in the GITC view to update the user's
remote view. (This works because .repo in the GITC view will
link to .repo in the client config directory.)
Part of the support for this change is the refactoring of GITC
related code into gitc_utils.py.
Change-Id: I2636aaa50b450b6f091309db8dd0e8f4dbdad579
Adds the new gitc-init command to set up a GITC client. Gitc-init
sets up the client directory and calls repo init within it. Once
the repo is initialized, then generates a GITC manifest file
by using git ls-remote on each project and retrieving the HEAD SHA
to use as the revision attribute.
Gitc-init inherits from and has all the options as repo init.
Change-Id: Icd7e47e90eab752a77de7c80ebc98cfe16bf6de3
Previously repo would only print the failing project path if
Sync_NetworkHalf returned false/empty, but if it threw an
exception the print() was never called.
Change-Id: I58c41de43930df5e34b21561c205e062a72e290f
Enable operating against groups of repositories. As it stands, it isn't
compatible with `-r/--regex`.
`repo forall -g groupname -c pwd` will run `pwd` for all projects in
groupname.
`repo forall -g thisgroup,-butnotthisone -c pwd` will run `pwd` for all
projects in `thisgroup` but not `butnotthisone`.
`repo list -g groupname -n` will list all the names of repos in
`groupname`.
Change-Id: Ia75c50ce52541d1c8cea2874b20a4db2e0e54960
In some cases, a user may wish to continue with a sync even though
it would require overwriting an existing git directory. This behavior
is not safe as a default because it could result in the loss of some
user data, but as an optional flag it allows the user more flexibility.
To support this, add a --force-sync flag to the sync command that will
attempt to overwrite the existing git dir if it is specified and the
existing git dir points to the wrong obj dir.
Change-Id: Ieddda8ad54e264a1eb4a9d54881dd6ebc8a03833
repo info will crash when using a manifest with no default element despite
default being an optional element. Output nothing for "Manifest Branch" if no
default element exists (or if no default revision exists).
Change-Id: I7ebffa2408863837ba980f0ab6e593134400aea9
If a workspace is synced with the -s or -t option, the included projects
may be different to those in the original manifest. However, when using
the forall command, the list of the projects from the original manifest
is used.
If the smart sync manifest file exists, use it to override the original
manifest.
Change-Id: Iaefcbe148d2158ac046f158d98bbd8b5a5378ce7
When syncing with the -s or -t option, a smart_sync_override.xml file
is created. This file is left in the file system when syncing again
without the -s or -t option.
Remove the smart sync override manifest, if it exists, when not using
the -s or -t option.
Change-Id: I697a0f6405205ba5f84a4d470becf7cd23c07b4b
git rev-parse fails for projects that don't have an explicit revision
specified, and don't have a branch of the same name as the default
revision. This can be the case in a workspace synced with the smart
sync (-s) or smart tag (-t) option.
Change-Id: I19bfe9fe7396170379415d85f10f6440dc6ea08f
The error message only states that writing the manifest failed.
Include the exception message, so it's easier to track down the reason
that the write failed.
Change-Id: I06e942c48a19521ba45292199519dd0a8bdb1de7
After performing the actual cherry-pick operation, the code
in cherry_pick.py opens a pipe to 'git commit -F' to rewrite the commit
message, emits the fixed-up commit msg to the pipe, then waits
for 'git commit' to complete. The child 'git' process winds up
hanging while reading from the pipe, however, since the parent
process still has it open. To fix the hang, change the parent process
to close its end of the pipe after it has emitted the message.
Change-Id: I5929371e69a5b076f09009d00d40a2c72ac8ac33
If the generator that produces per-project worker arguments raises an
exception it triggers python bug http://bugs.python.org/issue8296.
Rewrite the generator expression as a generator function, and catch
Exceptions and KeyboardInterrupts to end the iteration.
Also add a pool worker initializer to disable SIGINT to prevent
KeyboardInterrupts inside multiprocessing.Pool in the worker threads
causing the same problem.
Fixes easy-to-reproduce hangs when hitting ctrl-c during
repo forall -c echo
Change-Id: Ie4a65b3e1e07a64ed6bb6ff20f3912c4326718ca
In 2fb6466f79 an optimisation was
added to avoid fetching from remotes if the project is fixed to
a revision and the revision is already available locally.
This causes problems for users who expect all objects to be
fetched by default.
Change the logic so that the optimized behaviour is only enabled if
an option is explicitly given to repo sync.
Change-Id: I3b2794ddd8e0071b1787e166463cd8347ca9e24f
Before, a list was generated, which is why there was a massive delay.
Using a generator will allow processes to start straight away
Change-Id: Ia325b0b340cc328c08c9bcc92a6709bbdaf6a664
buflist was being used, which isn't available in Python 3.
`Execute` was using StringIO to capture the output of `PrintWorkTreeStatus`,
only to redirect it straight to stdout.
Instead, just let `PrintWorkTreeStatus` do it's own thing directly to stdout.
for handling `_FindOrphans`, we swap StringIO for a list. Nothing was done
that needed a a file like object.
Change-Id: Ibdaae137904de66a5ffb590d84203ef0fe782d8b
This reverts commit 38e4387f8e.
A "repo init" followed by "repo sync" is meant to be as safe as
"git clone". In particular it should not run arbitrary code provided
by the manifest owner.
It would still be nice to have support for manifest-defined git hooks
--- they'd just need a prompt like the upload RepoHook has. Hopefully
a later change can bring them back.
Change-Id: I5ecd90fb5c2ed64f103d856d1ffcba38a47b062d
Signed-off-by: Jonathan Nieder <jrn@google.com>
When working within a team or corporation it is often
useful/required to use predefined git templates. This
change teaches repo to use a per-remote git hook template
structure.
The implementation is done as a continuation of the
existing projecthook functionality. The terminology is
therefore defined as projecthooks.
The downloaded projecthooks are stored in the .repo
directory as a metaproject separating them from the users
project forest.
The projecthooks are downloaded and set up when doing a
repo init and updated for each new repo init.
When downloading a mirror the projecthooks gits are
not added to the bare forest since the intention is to
ensure that the latest are used (allows for company policy
enforcement).
The projecthooks are defined in the manifest file in the
remote element as a subnode, the name refers to the
project name on the server referred to in the remote.
<remote name="myremote ..>
<projecthook name="myprojecthookgit" revision="myrevision"/>
</remote>
The hooks found in the projecthook revision supersede
the stock hooks found in repo. This removes the need for
updating the projecthook gits for repo stock hook changes.
Change-Id: I6796b7b0342c1f83c35f4b3e46782581b069a561
Signed-off-by: Patrik Ryd <patrik.ryd@stericsson.com>
Signed-off-by: Ian Kumlien <ian.kumlien@gmail.com>
When there are uncommitted files in the tree, 'repo upload' stops to
ask if it is OK to continue, but does not report the actual names of
uncommitted files.
This patch adds plumbing to have the outstanding file names reported
if desired.
BUG=None
TEST=verified that 'repo upload' properly operates with the following
conditions present in the tree:
. file(s) modified locally
. file(s) added to index, but not committed
. files not known to git
. no modified files (the upload proceeds as expected)
Change-Id: If65d5f8e8bcb3300c16d85dc5d7017758545f80d
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Signed-off-by: Vadim Bendebury <vbendeb@google.com>
When current is "split" (i.e. some projects are current while others are not):
- Disable 'not in' printout (i.e. will print out all projects)
- Disable printing of multiple projects on one line
- Print current projects in green, non-current in white
Since using color to differentiate current from non-current in "split" cases:
- In non-split cases also print out project names in color (green for current
white for non-current)
Change-Id: Ia6b826612c708447cecfe5954dc767f7b2ea2ea7
Enable '--jobs' ('-j') option in the forall subcommand. For -jn
where n > 1, the '-p' option can no longer guarantee the
continuity of console output between the project header and the
output from the worker process.
SIG_INT is sent to all worker processes upon keyboard interrupt
(Ctrl+C).
Bug: Issue 105
Change-Id: If09afa2ed639d481ede64f28b641dc80d0b89a5c
Use JSON as it is shown to be much faster than pickle.
Also clean up the loading and saving functions.
Change-Id: I45b3dee7b4d59a1c0e0d38d4a83b543ac5839390
iterator.next() was replaced with iterator.__next__() in Python 3.
Use next(iterator) instead which will select the correct method for
returning the next item.
Change-Id: I6d0c89c8b32e817e5897fe87332933dacf22027b
For long-running forall commands sometimes it's useful to know which
iteration is currently running. Add REPO_I and REPO_COUNT environment
variables to reflect the current iteration count as well as the total
number of iterations so that the user can build simple status
indicators.
Example:
$ repo forall -c 'echo $REPO_I / $REPO_COUNT; git gc'
1 / 579
Counting objects: 41, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (19/19), done.
Writing objects: 100% (41/41), done.
Total 41 (delta 21), reused 41 (delta 21)
2 / 579
Counting objects: 53410, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (10423/10423), done.
Writing objects: 100% (53410/53410), done.
Total 53410 (delta 42513), reused 53410 (delta 42513)
3 / 579
...
Change-Id: I9f28b0d8b7debe423eed3b4bc1198b23e40c0c50
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The `review.URL.autocopy` setting sends email notification to the
named reviewers, but does not add them as reviewer on the uploaded
change.
Add a new setting `review.URL.autoreviewer`. The named reviewers
will be added as reviewer on the uploaded change.
Change-Id: I3fddfb49edf346f8724fe15b84be8c39d43e7e65
Signed-off-by: bijia <bijia@xiaomi.com>
This command allows a deeper diff between two manifest projects.
In addition to changed projects, it displays the logs of the
commits between both revisions for each project.
Change-Id: I86d30602cfbc654f8c84db2be5d8a30cb90f1398
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
The fetch logic is now shared between the jobs == 1 and
jobs > 1 cases. This refactoring also fixes a bug where
opts.force_broken was not honored when jobs > 1.
Change-Id: Ic886f3c3c00f3d8fc73a65366328fed3c44dc3be
Currently if you run repo download -c on a change and the cherry-pick
runs into a merge conflict a Traceback is produced:
rob@rob-i5-lm ~/Programming/repo_test/repo1 $ repo download -c repo1 3/1
From ssh://rob-i5-lm:29418/repo1
* branch refs/changes/03/3/1 -> FETCH_HEAD
error: could not apply 0c8b474... 2
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
Traceback (most recent call last):
File "/home/rob/Programming/git-repo/main.py", line 408, in <module>
_Main(sys.argv[1:])
File "/home/rob/Programming/git-repo/main.py", line 384, in _Main
result = repo._Run(argv) or 0
File "/home/rob/Programming/git-repo/main.py", line 143, in _Run
result = cmd.Execute(copts, cargs)
File "/home/rob/Programming/git-repo/subcmds/download.py", line 90, in Execute
project._CherryPick(dl.commit)
File "/home/rob/Programming/git-repo/project.py", line 1943, in _CherryPick
raise GitError('%s cherry-pick %s ' % (self.name, rev))
error.GitError: repo1 cherry-pick 0c8b4740f876f8f8372bbaed430f02b6ba8b1898
This amount of error message is confusing to users and has the side effect
of the git message telling you the actual issue being ignored.
This change introduces a message stating that the cherry-pick couldn't
be completed removing the Traceback.
To reproduce the issue create a change that causes a conflict with one currently
in review and use repo download -c to cherry-pick the conflicting change.
Change-Id: I8ddf4e0c8ad9bd04b1af5360313f67cc053f7d6a
This takes the wrapper importing code from main.py and moves it into
its own module so that other modules may import it without causing
circular imports with main.py.
Change-Id: I9402950573933ed6f14ce0bfb600f74f32727705
the value of Manifest.projects has changed from being the dictionary
to the values of the dictionary. Here we handle this change
correctly on a PostRepoUpgrade.
From a `git diff v1.12.7 -- manifest_xml.py`:
+ @property
def projects(self):
self._Load()
- return self._projects
+ return self._paths.values()
self._paths does contain the projects according to this line of
manifest_xml.py:
484 self._paths[project.relpath] = project
Change-Id: I141f8d5468ee10dfb08f99ba434004a307fed810
The backtrace currently occurs when one uses the "--cbr" argument with
the repo upload subcommand if the current branch is not tracking an
upstream branch. There may be other cases that would backtrace as well,
but this is the only one I found so far.
Change-Id: Ie712fbb0ce3e7fe3b72769fca89cc4c0e3d2fce0
This significantly reduces sync time and used brandwidth as only
a tar of each project's revision is checked out, but git is not
accessible from projects anymore.
This is relevant when git is not needed in projects but sync
speed/brandwidth may be important like on CI servers when building
several versions from scratch regularly for example.
Archive is not supported over http/https.
Change-Id: I48c3c7de2cd5a1faec33e295fcdafbc7807d0e4d
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
* Add .decode('utf-8') where needed
* Add 'b' to `open` where needed, and remove where unnecessary
Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
git-repo uses 2 space indentation. A couple of recent changes
introduced 4 space indentation in some modules.
Change-Id: Ia4250157c1824c1b5e7d555068c4608f995be9da
It is often useful to be able to include the same project more than
once, but with different branches and placed in different paths in the
workspace. Add this feature.
This CL adds the concept of an object directory. The object directory
stores objects that can be shared amongst several working trees. For
newly synced repositories, we set up the git repo now to share its
objects with an object repo.
Each worktree for a given repo shares objects, but has an independent
set of references and branches. This ensures that repo only has to
update the objects once; however the references for each worktree are
updated separately. Storing the references separately is needed to
ensure that commits to a branch on one worktree will not change the
HEAD commits of the others.
One nice side effect of sharing objects between different worktrees is
that you can easily cherry-pick changes between the two worktrees
without needing to fetch them.
Bug: Issue 141
Change-Id: I5e2f4e1a7abb56f9d3f310fa6fd0c17019330ecd
Example:
- `repo init -b master` / sync a project
- In one project: `git checkout -b work origin/branch-thats-not-master`
- make some changes, `git commit`
- `repo upload .`
- Upload will now be skipped with a warning instead of being uploaded to
master
Change-Id: I990b36217b75fe3c8b4d776e7fefa1c7d9ab7282
