Commit Graph

354 Commits

Author SHA1 Message Date
David Riley
e0684addee sync: Add support to dump a JSON event log of all sync events.
Change-Id: Id4852968ac1b2bf0093007cf2e5ca951ddab8b3b
2017-05-29 13:39:54 +09:00
David Pursehouse
fbcbcabe98 Merge "init: add --submodules to sync manifest submodules" 2017-05-27 04:24:58 +00:00
David Pursehouse
666debc518 gitc_delete: Remove unused imports
Change-Id: I672189ba99e18dca3956e2396c921d1ef0ca2ddd
2017-05-26 21:53:34 +09:00
David Pursehouse
c354a9b922 abandon: fix usage of undefined variable
As reported by pyflakes:

  subcmds/abandon.py:84: undefined name 'p'

The name of the variable should be 'proj'.

Change-Id: Ic09eb92e8db6b510e99efce010bd0bb094d7cbfe
2017-05-26 21:52:23 +09:00
Martin Kelly
e4e94d26ae init: add --submodules to sync manifest submodules
repo sync can sync submodules via the --fetch-submodules option.
However, if the manifest repo has submodules, those will not be synced.
Having submodules in the manifest repo -- while not commonly done -- can
be useful for inheriting a manifest from another project using <include>
and layering changes on top of it.  In this way, you can avoid having to
deal with merge conflicts between your own manifests and the other
project's manifests (for example, if you're managing an Android fork).

Add a --submodule option to init that automatically syncs the submodules
in the manifest repo whenever the manifest repo changes.

Change-Id: I45d34f04517774c1462d7f233f482d1d81a332a8
Signed-off-by: Martin Kelly <mkelly@xevo.com>
2017-05-23 16:51:31 -07:00
Marc Herbert
ffb4b89099 sync.py: report the remote URL on fatal git remote errors
repo can be configured to download from any number of remote git repos.
However when one fails repo doesn't report which one. Example:
Fatal: remote error: Daily ls-remote rate limit exceeded for IP xx.xx.xx.xx

TEST=repo init -q -u https://chromium.googlesource.com/chromiumos/manifest.git
  # Apply patch in ./.repo/repo/
  # Simulate a git remote error:
  sed -i -e 's#chromiumos/docs#chromiumos/XXdocs#' .repo/manifests/full.xml
  repo sync --quiet --force-sync docs
  # error message now shows the remote URL

Optional test tip: reduce the time.sleep(random(...)) in ./.repo/repo/project.py

Change-Id: I4509383b6a43a8e66064778e8ed612d8a735c8b6
2017-04-04 22:10:34 -07:00
Naseer Ahmed
f4dda9a1be init: Add no-tags and current branch options
This avoids fetching tags and branches for huge manifests

Change-Id: I19c9724d75364440b881b297d42b906f541f73ff
2016-12-01 19:03:41 -05:00
David Pursehouse
ae81c964b6 Merge "implement optional '--all' in the abandon command" 2016-10-28 07:52:31 +00:00
Andrew Wheeler
4d5bb68d58 status: add -q/--quiet option
The --quiet option reduces the output to just
a list of projects with modified workspaces (and
orphans if -o is specified)

A common use case is when performing a full-workspace
merge.  The integrator will kick-off a merge via:

    repo forall -c git merge <some tag>

And then produce a short list of conflicted projects via:

    repo status -q

The integrator can then iteratively fix and clean up all conficted
components.  The merge is complete when:

    repo status -q

    returns no output.

Change-Id: Ibbba8713eac35befd8287c95948874e23fd5c7e2
2016-10-17 15:24:09 -05:00
Kyunam.jo
2e14792a94 implement optional '--all' in the abandon command
when you want to delete all local branches, you should be find
all branches' name, and type them behind 'repo abandon' command.

Usage:
    repo abandon --all [<project>...]

Change-Id: I4d391f37fb9d89b8095488c585468eafc1a35f31
2016-10-17 02:29:42 +00:00
David Pursehouse
82f67987a3 Merge "sync: Fix semaphore release bug that causes thread 'leaks'" 2016-10-17 01:10:16 +00:00
Kyunam.jo
699bcd40be Removed duplication code in abandon.py
code about getting argument is duplicated.
so this line is removed

Change-Id: Id321b999c7dacdb403cd986cbf35f8db62efc157
2016-10-12 10:02:30 +09:00
Andrew Wheeler
7f1ccfbb7b sync: Fix semaphore release bug that causes thread 'leaks'
When repo syncs a manifest that utilizes multiple branches
in the same project, then the sync will use an extra
thread for each "duplicate".  For example, if
the manifest includes the project "foo" and "bar"
twice, then "repo sync -jN" will fetch with N+2 threads.

This is caused by _FetchHelper() releasing the thread semaphore
object each time it's called, even though _FetchProjectList()
may call this function multiple times within the scope of a
single thread.

Fix by moving the thread semaphore release to
_FetchProjectList(), which is only called once per thread
instance.

Change-Id: I1da78b145e09524d40457db5ca5c37d315432bd8
2016-10-11 14:10:34 -05:00
Dan Willemsen
eceeb1b1f5 Support broken symlinks when cleaning obsolete paths
When there's a symlink to a directory, os.walk still lists the symlink
in dirs, even if it isn't configured to follow symlinks. This will fail
the listdirs check if the symlink is broken (either before or during the
cleanup). So instead, check for directory symlinks and remove them using
os.remove.

Bug: Issue 231
Change-Id: I0ec45a26be566613a4a39bf694a3d9c6328481c2
2016-09-27 03:05:11 +00:00
Dan Willemsen
4350791e0d On project cleanup, don't remove nested projects
When there are nested projects in a manifest, like on AOSP right now:

<project path="build" name="platform/build" />
<project path="build/blueprint" name="platform/build/blueprint" />
<project path="build/kati" name="platform/build/kati" />
<project path="build/soong" name="platform/build/soong" />

And the top "build" project is removed (or renamed to remove the
nesting), repo just wipes away everything under build/ and re-creates
the projects that are still there. But it only checks to see if the
build/ project is dirty, so if there are dirty files in a nested
project, they'll just be blown away, and a fresh worktree checked out.

Instead, behave similarly to how `git clean -dxf` behaves and preserve
any subdirectories that have git repositories in them. This isn't as
strict as git -- it does not check to see if the '.git' entry is a
readable gitdir, just whether an entry named '.git' exists.

If it encounters any errors removing files, we'll print them all out to
stderr and tell the user that we were unable to clean up the obsolete
project, that they should clean it up manually, then sync again.

Change-Id: I2f6a7dd205a8e0b7590ca5369e9b0ba21d5a6f77
2016-09-20 17:16:12 -07:00
David Pursehouse
01b7d758d5 Merge "When syncing a project with a shared object store, disable automatic pruning." 2016-09-14 20:48:24 +00:00
David Pursehouse
d9de945d8a Merge "upload: short circuit when nothing is pending" 2016-09-14 05:20:36 +00:00
Gabe Black
2ff302929c When syncing a project with a shared object store, disable automatic pruning.
The shared object stores confuse git and make it throw away objects which are
still in use. We'll avoid that problem by disabling automatic pruning on those
projects, but there's nothing preventing a user from changing the config back
or pruning a repository manually.

BUG=chromium:375945
TEST=Ran repo sync on fresh ChromeOS checkout, starting with a branch of repo
with this change. Verified that the kernel projects and no others were
identified as having shared object stores, and that repo successfully disabled
automatic pruning in their configs. Re-enabled pruning and ran repo sync just
on one of the kernel directories. Verified that pruning was re-disabled as a
result.

Change-Id: I728ed5b06f0087aeb5a23ba8f5410a7cd10af5b0
2016-09-14 00:19:44 -04:00
Vadim Bendebury
e5c0ea0a95 Consider local project to be default for 'repo start'
The requirement to explicitly specify the local project when starting
a new repo branch is somewhat counter intuitive.

This patch uses the current directory's git tree as the default
project.

Tested by running

  'repo start <name>'

observed that the result is the same as if running

  'repo start <name> .'

Change-Id: If106caa801b4cd5ba70dbe8354a227d59f100aa3
2016-09-14 00:17:45 -04:00
Mike Frysinger
163a3be18b upload: short circuit when nothing is pending
When nothing is pending, most of this code is already short-circuited.
Hoist the single check up to make this more obvious/slightly faster.

Change-Id: Iec3a7e08eacd23a7c5f964900d5776bf5252c804
2016-09-14 00:16:37 -04:00
David Pursehouse
6a0a3648f1 Merge "init: Add --no-clone-bundle option" 2016-08-17 04:57:28 +00:00
David Pursehouse
6118faa118 Merge "init: Respect --quiet option when synching manifest repository" 2016-08-17 04:04:14 +00:00
Mike Frysinger
40252c20f7 RepoHook: allow users to approve hooks via manifests
The constant prompting when registered hooks change can be tedious and
has a large multiplication factor when the project is large (e.g. the
AOSP).  It gets worse as people want to write more checks, hooks, docs,
and tests (or fix bugs), but every CL that goes in will trigger a new
prompt to approve.

Let's tweak our trust model when it comes to hooks.  Since people start
off by calling `repo init` with a URL to a manifest, and that manifest
defines all the hooks, anchor trust in that.  This requires that we get
the manifest over a trusted link (e.g. https or ssh) so that it can't
be MITM-ed.  If the user chooses to use an untrusted link (e.g. git or
http), then we'll fallback to the existing hash based approval.

Bug: Issue 226
Change-Id: I77be9e4397383f264fcdaefb582e345ea4069a13
2016-08-16 13:02:52 -04:00
Hu xiuyun
9711a98d6c init: Add --no-clone-bundle option
Bug: Issue 218
Change-Id: I42ba1f5fb9168875da0df6bdf4fe44c8d6498d54
2016-08-15 09:51:48 +09:00
David Pursehouse
438eade413 init: Respect --quiet option when synching manifest repository
Change-Id: Ib58b7dd971670e0888e6428333050700e776b0de
2016-08-15 09:51:48 +09:00
Sebastian Schuberth
7ecccf6225 diffmanifests: support custom git pretty format strings
Change-Id: I29f4f1351c421f393328514d145df1a96aed9ee2
2016-04-21 18:36:11 +00:00
David Pursehouse
79fba68e40 sync: Update help text for --smart-sync to be more specific
The --smart-sync option should return the manifest for *the latest*
known good build.

Change-Id: I2f3216b5b9e1af2ea5f9c3bf1c025813a3b77581
2016-04-13 18:03:00 +09:00
Takeshi Kanemoto
1f0564406b Add --inverse-regex option to forall subcommand
Make it possible to exclude projects using regex/wildcard.

The syntax is similar to that of the -r option, e.g.:

  repo forall -i ^platform/ ^device/ -c 'echo $REPO_PROJECT'

Change-Id: Id250de5665152228c044c79337d3ac15b5696484
2016-04-05 07:28:27 +00:00
Alexandre Garnier
4cfb6d7167 Better error display on forall
It was only displaying 'Project list error: GitError()'
without any useful info about the project nor the error

Change-Id: Iad66cbaa03cad1053b5ae9ecc90d7772aa42ac13
2016-02-18 01:29:54 +00:00
Dan Willemsen
ecf0a6c92b Merge "GITC: Fix 'repo start <branch> <repo>/<subdir>'" 2016-02-04 22:36:05 +00:00
Dan Willemsen
04197a5144 GITC: Fix 'repo start <branch> <repo>/<subdir>'
As soon as we wrote the gitc manifest, the folder for that repo became
empty, causing the next GetProjects lookup to fail. Reorder the
GetProjects calls so that they all happen while we still have the
repository contents available.

If you were already in a subdir, for cases like 'repo start <branch> .',
this would still fail, since the working directory would disappear out
from under you. That's fine most of the time, since we shouldn't be
doing operations based on the local directory, but git has a realpath
function that tries to restore CWD by chdir'ing back to it. So if the
working directory no longer exists, chdir to the topdir before
continuing.

Change-Id: Ibdf6cd37ff6e5a5f8338347c3919175491f7166f
2016-02-04 14:31:55 -08:00
Xiaohui Chen
0b4cb325c6 Add option to rebase onto project's manifest version
Some teams have a continuous build server that would mark certain
manifest green and safe to sync to.  Then team members could repo
sync to that particular manifest file and make sure they always
sync to a green build.  But if she/he has some local changes and
wants to rebase, currently it would be a manual process to find the
correct version to rebase onto.  This patch helps with that use
case by automating the process to rebase onto the currently synced
manifest version.

Change-Id: I847c9eb6addf7f84fd3f5594fbf8c0bcc103f9a5
2016-01-28 10:20:03 -08:00
Hu Xiuyun
e9becc079c Sync: Fix error exit code when both -n and -f are used
When repo sync is used with -f (--force-error) and a project fails to
sync, the sync will continue but then exit with an error status.

However if -n (--network-only) is also used, the exit code is 0, even
when a project failed.

Modify the logic to make sure the sync exits with the correct status.

Bug: Issue 214
Change-Id: I0b5d97a34642c5aa3743750ef14a42c9d5743c1d
2015-11-26 02:25:43 +00:00
David Pursehouse
74cfd2709b Sync: Add option to prune refs during sync
By passing --prune to the sync command, the --prune option is
given to the `git fetch`, causing refs that no longer exist on
the remote to be removed.

Change-Id: I3cedacce14276d96ac2d5aabf2d07fd05e92bc02
2015-10-27 03:04:17 +00:00
Pascal Muetschard
c2a64ddffd A couple of fixes to the init command's -p option.
Adds windows as one of the allowed platforms flags.
Fixes -p foo to append 'platform-foo', instead of each letter (list.extend
expects a list and thus appends each char in the string, rather than the
string itself).

Change-Id: I73a92127ac29a32fc31b335cc54a246302904140
2015-10-22 13:28:20 -07:00
Dan Willemsen
745b4ad660 Fix gitc-init behavior
With gitc-init, a gitc client may be specified using '-c'. If we're
not currently in that client, we need to change directories so that
we don't affect the local checkout, and to ensure that repo is
checked out in the new client.

This also makes '-c' optional if already in a gitc client, to match
the rest of the init options.

Change-Id: Ib514ad9fd101698060ae89bb035499800897e9bd
2015-10-07 15:43:22 -07:00
David Pursehouse
4c5f74e452 Sync: Add HTTP Cookie File header on temporary cookie file
The .gitcookies file generated by googlesource.com does not have
the header:

 # (Netscape) HTTP Cookie File

which causes python's MozillaCookieJar.load to fail with the
error:

 "does not look like a Netscape format cookies file"

Prepend the expected header onto the generated cookie file.

We don't bother to check if the header already exists on the
file; repeating it does not cause any problem.

Bug: Issue 207
Change-Id: I7d39720a1d36a6aae00f70691156514ebc04e579
2015-10-02 11:12:05 +09:00
David Pursehouse
b1ad2190a2 Sync: Don't fail when git cookies can't be loaded
If the git cookies file fails to load, use a default
cookie jar instead.

Bug: Issue 207
Change-Id: I7cb326c204f2784ab4dbd13801b3186667af5b78
2015-10-02 11:04:01 +09:00
Simran Basi
f231db11a2 GITC: Add repo gitc-delete command.
repo gitc-delete deletes a GITC client and all the locally
saved sources. Useful for removing unnecessary clients and
recovering disk space.

Change-Id: Idf23addcea52b8713d268c34a7b37da0c5e5cd26
2015-10-01 21:05:17 +00:00
Dan Willemsen
79360640f4 Add GitcClientCommand class for GITC-specific commands
These won't show up as common commands in the help text unless in a GITC
client, and will refuse to execute.

Change-Id: Iffe82adcc9d6ddde9cb4b204f83ff018042bdab0
2015-09-29 13:46:34 -07:00
Alexander Bird
3c03580607 fixed typo in gitc_init.py help output
Change-Id: I86459bf63297487457d6c4c995dfd1e63133ec53
2015-09-11 14:11:30 -04:00
Dan Willemsen
5ea32d1359 GITC: Always update the gitc manifest from the repo manifest
This way any changes made to the main manifest are reflected in the gitc
manifest. It's also necessary to use both manifests to sync since the
information required to update the gitc manifest is actually in the repo
manifest.

This also fixes a few issues that came up when testing. notdefault
groups weren't being saved to the gitc manifest in a method that matched
'sync'. The merge branch wasn't always being set to the correct value
either.

Change-Id: I435235cb5622a048ffad0059affd32ecf71f1f5b
2015-09-09 20:50:40 -07:00
Dan Willemsen
0375523331 Revert "GITC: Always update the gitc manifest from the repo manifest"
This reverts commit 250303b437.

Change-Id: I1fd8af20f802553151aacb953c913f3305ca6057
2015-09-09 21:43:32 +00:00
Dan Willemsen
250303b437 GITC: Always update the gitc manifest from the repo manifest
This way any changes made to the main manifest are reflected in the gitc
manifest. It's also necessary to use both manifests to sync since the
information required to update the gitc manifest is actually in the repo
manifest.

This also fixes a few issues that came up when testing. notdefault
groups weren't being saved to the gitc manifest in a method that matched
'sync'. The merge branch wasn't always being set to the correct value
either.

Change-Id: I5dbc850dd73a9fbd10ab2470ae4c40e46ff894de
2015-09-09 12:35:56 -07:00
Simran Basi
ba72d8301e GITC: Fix repo sync.
Fixing http://b/23785024 by calling os.getcwd() because variable
cwd no longer exists.

Change-Id: I21ff7d059e072f9f60726db76b67587a92c878ad
2015-09-03 10:47:44 -07:00
Dan Willemsen
9ff2ece6ab gitc: Improve help visibility
This improves the visiblity of gitc-init if we can get the gitc config,
and hides it otherwise.

Change-Id: I82830b0b07c311e8c74397ba79eb4c361f8b6fb5
2015-09-01 12:23:56 -07:00
Simran Basi
8ce5041596 GITC: Pull GITC Manifest Dir from the config.
Updates the repo launcher and gitc_utils to pull the manifest
directory location out of the gitc config file.

Change-Id: Id08381b8a7d61962093d5cddcb3ff6afbb13004b
2015-08-31 21:39:17 +00:00
Simran Basi
f7a51898d3 GITC: Expand relative remote URLs.
The GITC filesystem does not understand relative URLs for remotes,
so now if a remote uses a relative URL, it will be be expanded to
be relative to the manifest URL.

Change-Id: Ie1210758560aeb1934da3f71496aaf19c2728214
2015-08-28 18:09:05 +00:00
Simran Basi
b9a1b73425 GITC: Add repo start support.
Add repo start support for GITC checkouts. If the user is in
the GITC FS view, they can now run repo start to check out
the sources and create a new working branch.

When "repo start" is called on a GITC project, the revision
tag is set to an empty string and saved in a new tag:
old-revision. This tells the GITC filesystem to display the
local copy of the sources when being viewed. The local copy
is created by pulling the project sources and the new branch
is created based off the original project revision.

Updated main.py to setup each command's gitc_manifest when
appropriate.

Updated repo sync's logic to sync opened projects and
updating the GITC manifest file for the rest.

Change-Id: I7e4809d1c4fc43c69b26f2f1bebe45aab0cae628
2015-08-28 10:53:05 -07:00
Dan Willemsen
3010e5ba64 Smartsync: Don't fail if there isn't a cookiefile
Change-Id: I434a259f43ca9808e88051ac8ba865c519a24702
2015-08-20 10:29:37 -07:00