Commit Graph

291 Commits

Author SHA1 Message Date
Renaud Paquay
788e9626cc Provide more specific error message for symlinks errors on Windows
Change-Id: Ia6099beef37ae6b6143eba243fe7fbe02b74a9bb
2017-08-31 13:49:59 -07:00
Renaud Paquay
010fed7711 Replace all os.remove calls
os.remove raises an exception when deleting read-only files on
Windows. Replace all calls with calls to platform_utils.remove,
which deals with deals with that issue.

Change-Id: I4dc9e0c9a36b4238880520c69f5075eca40f3e66
2017-08-31 13:49:36 -07:00
Renaud Paquay
227ad2ef42 Implement islink, readlink and realpath using Win32 api
Change-Id: I18452cbb32d24db73601ad10485dbe6bb278731c
2017-08-31 13:49:01 -07:00
Renaud Paquay
2a4be94878 Handle Windows line endings when reading binary files
Without this change, '.git\HEAD' files, for examples, are sometime
read incorrectly resulting in the current branch to be reset to
"master" when running a "repo init -b xxx" on an already initialized
repository.

Change-Id: I48c7ef85ff81626edf156914329a560e14252f2a
2017-08-31 12:13:52 -07:00
David Pursehouse
2c57d619bc Merge "Add option '--no-cert-checks' for 'upload' sub command." 2017-08-30 15:11:10 +00:00
David Pursehouse
d1ebc89a08 Merge changes from topic "windows-support"
* changes:
  Port os.rename calls to work on Windows
  Workaround shutil.rmtree limitation on Windows
  Add support for creating symbolic links on Windows
  Make "git command" and "forall" work on Windows
2017-08-30 10:24:03 +00:00
Zac Livingston
e43322625a Print a message when fetching is skipped for an immutable ref
The output indicates that fetching happens even when it is skipped.

To avoid confusion, print a message when fetching is skipped for
an immutable ref so that the user knows when and why a fetch is skipped.

Change-Id: Id6e4812cebc5e57d379feb76a9d034af0b93043b
2017-08-25 00:00:02 +00:00
Łukasz Gardoń
bed59cec5e Add option '--no-cert-checks' for 'upload' sub command.
This option allow to bypass verification ssl certification while
establishing connection with Gerrit to upload review.

Change-Id: If2e15f5a273c18a700eb5093ca8a4d5a4cbf80cd
2017-08-23 14:06:14 +02:00
Jonathan Nieder
c94d6eb902 Revert "Migrate git-repo to create private changes rather than drafts"
This reverts commit d88f53e2b9. I merged
it too hastily without paying enough attention to compatibility with
released Gerrit versions.

Change-Id: I4028d4737df1255f11e217da183a19a010597d5b
2017-08-08 18:34:53 +00:00
Changcheng Xiao
d88f53e2b9 Migrate git-repo to create private changes rather than drafts
Considering that some users might expect changes created with
'-d' option are not public. Private changes may be a better
choice here than work-in-progress changes.

Change-Id: I46a8fb9ae38beb41cf96d6abe82bea6db2439669
2017-08-07 15:08:18 +02:00
Changcheng Xiao
87984c6db4 Add options for git-repo to support private and wip changes
This change adds options for git-repo tool to support private
changes and work-in-progress changes.

Change-Id: I343491f5949f06f1580d53f9cc0dee2dca09130f
2017-08-07 15:02:39 +02:00
Martin Kelly
224a31a765 init: add missing submodule arg
The submodule argument to Sync_LocalHalf was missing in
MetaBranchSwitch, causing submodules not to get synced when the
-b/--manifest-branch argument to init is used.

Change-Id: Ie86d271abac2020725770be36ead83be3326e64b
Signed-off-by: Martin Kelly <mkelly@xevo.com>
2017-07-10 14:50:52 -07:00
Renaud Paquay
ad1abcb556 Port os.rename calls to work on Windows
os.rename fails on Windows if the destination exists, so replace
os.rename to platform_utils.rename which handles the platform
differences.

Change-Id: I15a86f10f65eedee5b003b80f88a0c28a3e1aa48
2017-05-29 19:33:07 +09:00
Renaud Paquay
a65adf74f9 Workaround shutil.rmtree limitation on Windows
By default, shutil.rmtree raises an exception when deleting readonly
files on Windows.

Replace all shutil.rmtree with platform_utils.rmtree, which adds an
error handler to make files read-write when they can't be deleted.

Change-Id: I9cfea9a7b3703fb16a82cf69331540c2c179ed53
2017-05-29 19:32:31 +09:00
Renaud Paquay
d5cec5e752 Add support for creating symbolic links on Windows
Replace all calls to os.symlink with platform_utils.symlink.

The Windows implementation calls into the CreateSymbolicLinkW Win32
API, as os.symlink is not supported.

Separate the Win32 API definitions into a separate module
platform_utils_win32 for clarity.

Change-Id: I0714c598664c2df93383734e609d948692c17ec5
2017-05-29 19:30:34 +09:00
David Riley
e0684addee sync: Add support to dump a JSON event log of all sync events.
Change-Id: Id4852968ac1b2bf0093007cf2e5ca951ddab8b3b
2017-05-29 13:39:54 +09:00
Renaud Paquay
fef9f21b28 Fix misplaced file separator string.replace call
Project names are stored as path using the '/' file separator, and
stored in a dictionary as keys.

Change-Id: Ide40dfe840958ac0d46caae5f77f1a49d71c9d90
2017-05-28 21:18:13 +09:00
David Pursehouse
fbcbcabe98 Merge "init: add --submodules to sync manifest submodules" 2017-05-27 04:24:58 +00:00
Martin Kelly
e4e94d26ae init: add --submodules to sync manifest submodules
repo sync can sync submodules via the --fetch-submodules option.
However, if the manifest repo has submodules, those will not be synced.
Having submodules in the manifest repo -- while not commonly done -- can
be useful for inheriting a manifest from another project using <include>
and layering changes on top of it.  In this way, you can avoid having to
deal with merge conflicts between your own manifests and the other
project's manifests (for example, if you're managing an Android fork).

Add a --submodule option to init that automatically syncs the submodules
in the manifest repo whenever the manifest repo changes.

Change-Id: I45d34f04517774c1462d7f233f482d1d81a332a8
Signed-off-by: Martin Kelly <mkelly@xevo.com>
2017-05-23 16:51:31 -07:00
heping
3d7bbc9edf project.py: fix performance issue with --reference when the mirrored repository has many refs
Change-Id: Id0183903597f872eee80ca32a8050125b187a3d4
2017-04-12 20:04:47 +08:00
David Pursehouse
bb9c42cf1d Merge "Fix removing broken symlink in reference dir" 2016-12-06 07:51:01 +00:00
Dan Willemsen
8e2d1d521e Merge "Fix checkout error when depth passed to repo init and revision is a sha1" 2016-10-28 19:30:45 +00:00
Aymen Bouaziz
6c5944606a Fix checkout error when depth passed to repo init and revision is a sha1
Currently, if direct fetch of a sha1 is not supported by git server and
depth option is used, we fallback on syncing the upstream branch by
ignoring depth option.

This fallback doesn't work in next 2 cases:
(1) upstream attribute is not specified in manifest
(2) depth option is passed to repo init command
    (not with clone-depth attribute in manifest)

This commit do the following:
- fixes (1) by updating condition used to apply fallback
  first we retry with depth set to None, then by syncing all branches
- fixes (2) by passing depth as argument of _RemoteFetch() method
  thus, its value is not set again to depth value passed to repo init
  command when applying fallback

Change-Id: Ifd6fffafc49ba229df624b0d7b64c83d47619d17
2016-10-28 14:29:57 +02:00
Marc Herbert
ec287902e6 _CheckDirReference: log actual error before suggesting --force-sync
A recent backward incompatible change created confusion and loss of
productivity and highlighted the very limited amount of information
provided when repo sync fails; merely recommending to --force-sync
and blow-up git repos without any hint as to why. The addition of
this basic _error(...) call would have provided a clue and will in
the future.

BUG=Issue 232
TEST=simulate a breakage similar to the ones reported at
  https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-dev/2-0oCy_CX5s
  cd .repo/projects/src/third_party/libapps.git/
  file info; rm info; ln -s wronglink info
  cd -
  repo sync src/third_party/libapps/
  # error message now shows the failure

Change-Id: Idd2f177a096f1ad686caa8c67cb361d594ccaa57
2016-10-27 12:58:26 -07:00
Andrew Wheeler
4d5bb68d58 status: add -q/--quiet option
The --quiet option reduces the output to just
a list of projects with modified workspaces (and
orphans if -o is specified)

A common use case is when performing a full-workspace
merge.  The integrator will kick-off a merge via:

    repo forall -c git merge <some tag>

And then produce a short list of conflicted projects via:

    repo status -q

The integrator can then iteratively fix and clean up all conficted
components.  The merge is complete when:

    repo status -q

    returns no output.

Change-Id: Ibbba8713eac35befd8287c95948874e23fd5c7e2
2016-10-17 15:24:09 -05:00
Steve Rae
d648045366 implement optional 'pushurl' in the manifest file
Allow the 'remote' element in the manifest file to define an optional
'pushurl' attribute which is passed into the .git/config file.

Change-Id: If342d299d371374aedc4440645798888869c9714
Signed-off-by: Steve Rae <steve.rae@raedomain.com>
2016-09-20 15:31:20 +00:00
David Pursehouse
e01ee026e6 Merge "Fix submodule checkout error when using sync-s option" 2016-09-14 07:45:50 +00:00
Jonathan Nieder
71e4cea6de RepoHook: do not list options twice during hash based approval
Instead of

  Do you want to allow this script to run (yes/yes-never-ask-again/NO)? (yes/always/NO)?

ask

  Do you want to allow this script to run (yes/always/NO)?

Change-Id: I5f5a2d0e88086a8d85e54fb8623a62d74a20956a
Signed-off-by: Jonathan Nieder <jrn@google.com>
2016-08-18 17:06:26 -07:00
David Pursehouse
183c52ab02 Merge "project: Set config option to skip lfs smudge filter" 2016-08-17 01:00:12 +00:00
Mike Frysinger
40252c20f7 RepoHook: allow users to approve hooks via manifests
The constant prompting when registered hooks change can be tedious and
has a large multiplication factor when the project is large (e.g. the
AOSP).  It gets worse as people want to write more checks, hooks, docs,
and tests (or fix bugs), but every CL that goes in will trigger a new
prompt to approve.

Let's tweak our trust model when it comes to hooks.  Since people start
off by calling `repo init` with a URL to a manifest, and that manifest
defines all the hooks, anchor trust in that.  This requires that we get
the manifest over a trusted link (e.g. https or ssh) so that it can't
be MITM-ed.  If the user chooses to use an untrusted link (e.g. git or
http), then we'll fallback to the existing hash based approval.

Bug: Issue 226
Change-Id: I77be9e4397383f264fcdaefb582e345ea4069a13
2016-08-16 13:02:52 -04:00
David Pursehouse
76a4a9df86 project: Set config option to skip lfs smudge filter
During sync, repo runs `git read-tree --reset -u -v HEAD` which causes
git-lfs's smudge filter to run. However this fails because git-lfs does
not work with bare repositories.

Add lfs.filter configuration to the project config as suggested in the
comments on the upstream git-lfs client issue [1]. This prevents the
smudge filter from running, and the sync completes successfully.

For any projects that have LFS objects, `git lfs pull` must be executed.

[1] https://github.com/github/git-lfs/issues/1422

Bug: Issue 224
Change-Id: I091ff37998131e2e6bbc59aa37ee352fe12d7fcd
2016-08-16 21:55:36 +09:00
David Pursehouse
8016f60a46 Merge "repo: Repo does not always handle '.' parameter correctly" 2016-08-14 08:50:28 +00:00
Cheuk Leung
8ac0c96537 Fix removing broken symlink in reference dir
Re-ordered to first create the symlink before checking the source
file and remove the destination if the source does not exists.

Change-Id: Iae923ba2ef0ba5a8dc1b8e42d8cc3f3708f773af
2016-06-29 05:43:01 +00:00
David Pursehouse
a36af0767b Merge "Fix variable assignment" 2016-06-29 01:43:41 +00:00
Aymen Bouaziz
037040f73e Fix variable assignment
If upstream string is empty, current_branch_only variable will be assigned
to an empty string.

This is not what we expect here as this variable is a boolean.

Change-Id: Ibba935e25a74c2be1e50c88b4b403cf394ba365e
2016-06-28 12:31:25 +02:00
Aymen Bouaziz
2598ed06f1 Fix submodule checkout error when using sync-s option
When sync-s="true" option is used, the checkout of a submodule will try
to use the revision attribute of the parent project.

If this revision is a named reference, the checkout will fail if there
is no reference with this name in the submodule.

The proposed solution is to use the git commit id as revisionExpr for
submodules.

Change-Id: Ie8390a11957fd6a9c61289c6861d13cb3fa11678
2016-06-27 20:16:45 +02:00
Dan Willemsen
8b39fb4bc0 Merge "Fix XmlManifest.Save with remotes that have 'alias' set" 2016-04-22 18:28:57 +00:00
Dan Willemsen
96c2d65489 Fix XmlManifest.Save with remotes that have 'alias' set
When the alias attribute is set for a remote, the RemoteSpec attached to
a Project only contains the alias name used by git, not the original
name used in the manifest. But that's not enough information to
reconstruct the manifest, so save off the original manifest name as
another RemoteSpec parameter, only used to write the manifest out.

Bug: Issue 181
Bug: Issue 219
Change-Id: Id7417dfd6ce5572e4e5fe14f22924fdf088ca4f3
2016-04-22 10:32:06 +09:00
Sebastian Schuberth
7ecccf6225 diffmanifests: support custom git pretty format strings
Change-Id: I29f4f1351c421f393328514d145df1a96aed9ee2
2016-04-21 18:36:11 +00:00
Mark E. Hamilton
f9fe3e14d2 repo: Repo does not always handle '.' parameter correctly
The repo script allows a manifest to specify a '.' as the path the
top-level directory, which co-locates the .git and .repo directories,
and places files from the git repository at the top-level:

  <project name="proj_name" path="." />
  <project name="sierra.other.git" path="other" />

Most commands work correctly with this setup. Some commands, however,
fail to find the project. For instance, 'repo sync' works, and 'repo sync .'
works in a sub-project ('other' in this case) but 'repo sync .' in the
top-level directory fails with the error:

error: project . not found

There are two reasons for this:

1. The self.worktree attribute of the Project object is not normalized,
so with a '.' for path its value would be '/my/project/root/.'. This is
fine when used as a path, since it's the same path as '/my/project/root',
but when used in a string comparison it fails. This commit applies
os.path.normpath() to that value before storing it.

2. The _GetProjectByPath method in command.py was not checking the path
against manifest.topdir, so even once it was normalized the project was
not found. This commit adds a check against manifest.topdir if the
loop drops out without finding a project.

Change-Id: Ic84d053f1bbb5a357cad566805d5a326ae8246d2
2016-04-08 00:07:52 +00:00
Dan Willemsen
bdb866ea76 Fix symlinking of new projects
We weren't copying these lists, so the += was actually changing the
underlying lists.

When a new project was added to the manifest, we run _CheckDirReference
against the manifest project with share_refs=True, which added the
working_tree_* to the shareable_* lists. Then, when we load the new
manifest and create the new project, it uses the lists that already
contain the working_tree_* files, even though we passed
share_refs=False.

This happens reliably under the above conditions, but doesn't seem to
happen when syncing a fresh tree. So we've got a mixture of links that
may need to be cleaned up later. This patch will just stop it from
happening in the future.

Change-Id: Ib7935bfad78af1e494a75e55134ec829f13c2a41
2016-04-05 17:44:09 -07:00
David Pursehouse
936d6185eb Merge changes from topic 'pylint-pep8-cleanup'
* changes:
  command.py: Cleaned up pylint/pep8 violations
  project.py: Cleaned up pylint/pep8 violations
2016-03-15 00:31:12 +00:00
Mike Frysinger
4aa4b211c6 RepoHook: set __file__ when running the hook
A common design pattern is to use __file__ to find the location of the
active python module to assist in output or loading of related assets.
The current hook systems runs the pre-upload.py hook in a context w/out
that set leading to runtime errors:

$ repo upload --cbr .
ERROR: Traceback (most recent call last):
  File ".../repo/project.py", line 481, in _ExecuteHook
    self._script_fullpath, 'exec'), context)
  File ".../repohooks/pre-upload.py", line 32, in <module>
    path = os.path.dirname(os.path.realpath(__file__))
NameError: name '__file__' is not defined

Define this variable in this context so code can safely use it.

Change-Id: If6331312445fa61d9351b59f83abcc1c99ae6748
2016-03-05 21:52:31 +00:00
Mark E. Hamilton
30b0f4e022 project.py: Cleaned up pylint/pep8 violations
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a number of violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the project.py script.

This script now has a pylint score of 10.0, and no violations reported
by pep8.

Change-Id: I1462fd84f5b6b4c0dc893052671373e7ffd838f1
2016-03-02 09:05:40 -07:00
Dan Willemsen
1a799d14b7 Fix prune when bare git has detached head
We don't really use HEAD much in the bare git repositories, but there
have been reports of errors in git-symbolic-ref:

  symbolic-ref: fatal: Refusing to point HEAD outside of refs/

That happen when the bare git repo is in the detached head state. It's
possible that previous operations were killed while we were pruning
branches.

Use DetachHead instead of SetHead if we're restoring the repo into a
detached head state.

Change-Id: I9062e8957bc70367d3ded399685ac026fbb421fc
2015-12-15 14:22:40 -08:00
Dan Willemsen
e1e0bd1f75 Check for broken links when updating linkfiles
If a linkfile is a broken link (destination does not exist), and it
needs to be updated, we didn't notice that it needed to be removed
first. Use lexists instead of exists to check for this condition.

Change-Id: I1f6a1f0193d3fd2b9f7a647836044997f6ab32eb
2015-11-18 16:51:51 -08:00
David Pursehouse
74cfd2709b Sync: Add option to prune refs during sync
By passing --prune to the sync command, the --prune option is
given to the `git fetch`, causing refs that no longer exist on
the remote to be removed.

Change-Id: I3cedacce14276d96ac2d5aabf2d07fd05e92bc02
2015-10-27 03:04:17 +00:00
Dan Willemsen
029eaf3bac _CopyAndLinkFiles even if the sources haven't changed
The source or destination attributes may have changed even if the source
didn't, so we need to make sure that these are up to date.

Change-Id: I266ef3598ddda7e8c23bc9c6a049905ddc586348
2015-09-03 12:54:06 -07:00
Simran Basi
b9a1b73425 GITC: Add repo start support.
Add repo start support for GITC checkouts. If the user is in
the GITC FS view, they can now run repo start to check out
the sources and create a new working branch.

When "repo start" is called on a GITC project, the revision
tag is set to an empty string and saved in a new tag:
old-revision. This tells the GITC filesystem to display the
local copy of the sources when being viewed. The local copy
is created by pulling the project sources and the new branch
is created based off the original project revision.

Updated main.py to setup each command's gitc_manifest when
appropriate.

Updated repo sync's logic to sync opened projects and
updating the GITC manifest file for the rest.

Change-Id: I7e4809d1c4fc43c69b26f2f1bebe45aab0cae628
2015-08-28 10:53:05 -07:00
David Pursehouse
dc2545cad6 project.py: Improve message shown when hook is not replaced
If a hook file has been modified locally, it will not be replaced.

Improve the message to make this clearer.

Also change it from an error to a warning.

Change-Id: I62c635390f24d2868db17717c247861b0381c99f
2015-08-25 05:40:46 +00:00