Commit Graph

179 Commits

Author SHA1 Message Date
LaMont Jones
b308db1e2a manifest_xml: group for submanifest projects
Add all projects in a submanifest to the group
submanifest::<path_prefix> for ease in filtering.

Change-Id: Ia6f01f9445f4f8d20fda3402f4d5821c43ceaf7f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/331319
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-02-28 20:08:58 +00:00
LaMont Jones
cc879a97c3 Add multi-manifest support with <submanifest> element
To be addressed in another change:
 - a partial `repo sync` (with a list of projects/paths to sync)
   requires `--this-tree-only`.

Change-Id: I6c7400bf001540e9d7694fa70934f8f204cb5f57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322657
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-02-17 21:57:55 +00:00
LaMont Jones
87cce68b28 Move local-manifest check to manifest_xml.py
This removes the need for git_superproject to include manifest_xml, and
puts the logic for local_manifest detection in one place.

Change-Id: I4d33ded0542ceea4606a1ea24304f678de20c59e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/330499
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
2022-02-15 22:15:42 +00:00
XD Trol
630876f9e4 init: add an option --enable-git-lfs-filter
It was reported that git-lfs did not work with git-repo. Specifically,
`git read-tree -u` run by `repo sync` would fail git-lfs's smudge
filter. See https://github.com/github/git-lfs/issues/1422.

In fact, by the time `git read-tree -u` is run, the repository is not
bare. It is just that, the working directory is not the same as the
.git directory. git-lfs's filter should work. No one seems to have
delved into that issue.

Today, with newer versions of git-repo and git-lfs, that issue will
not reproduce. Tested with
- git 2.33, git-lfs 2.13 on macOS
- git 2.17, git-lfs 2.3 on ubuntu

So, it seems fine to add an option --enable-git-lfs-filter, default to
false, and stat that it may not work with older versions of git and
git-lfs in the help doc.

Bug: https://crbug.com/gerrit/14516
Change-Id: I8d21854eeeea541e072f63d6b10ad1253b1a9826
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328359
Tested-by: XD Trol <milestonejxd@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-01-26 01:47:20 +00:00
Michael Kelly
2f3c3316e4 Update revisionId if required when using extend-project
When a hard revision ID is provided in a regular project tag then the
revisionId is updated as well if it is a commit hash.  The difference
is that if the revisionExpr is a commit, git-repo needs to update
refs/remotes/m/master with update-ref not symbolic-ref, as the latter
must refer to another ref, not to a specific commit.

Change-Id: I215a62dabb30225e480ad2c731416d775fc0c750
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310963
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-28 20:12:00 +00:00
Michael Kelly
37c21c268b Add 'dest-path' to extend-project to support changing path
This allows us to move the repository to a new location in the source
tree without having to remove-project + add a new project tag.

Change-Id: I4dba6151842e57f6f2b8fe60cda260ecea68b7b4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310962
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
2021-09-28 20:12:00 +00:00
Mike Frysinger
9d96f58f5f make file removal a bit more robust
Some of the file removal calls are subject to race conditions (if
something else deletes the file), so extend our remove API to have
an option to ignore ENOENT errors.  Then update a bunch of random
call sites to use this new functionality.

Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195
Reviewed-by: Sean McAllister <smcallis@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-09-28 16:06:50 +00:00
Xin Li
e0b16a22a0 superproject: support a new revision attribute.
Tested:
$ ./run_tests

Verified that a manifest that specified superproject revision would use
the specified revision, and superproject will use the default revision.

Note that this is a slight behavior change from earlier repo versions,
which would always use the branch name of the manifest itself. However,
the new behavior would be more consisitent with regular "project"
element and would allow superproject be used even if it is not enabled
for the particular manifest branch, so we have decided to make the
change as it would provide more flexibility and better matches what
other elements would do.

Bug: [google internal] b/187868160
Change-Id: I35255ee347aff6e65179f7879d52931f168b477e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317643
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-27 06:36:05 +00:00
Jack Neus
a84f43a006 manifest: make repo-hooks more robust wrt element ordering
Currently, repo will fail to sync to a manifest if the definition
of the repo-hooks project comes after the repo-hooks element.

BUG=none
TEST=new test, run_tests

Change-Id: I0bf85625173492af6c6404d4b67543e96e670562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318520
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
2021-09-23 21:17:38 +00:00
Jack Neus
6ea0caea86 repo: properly handle remote annotations in manifest_xml
BUG=b:192664812
TEST=tests/

Change-Id: I1aa50260f4a00d3cebbd531141e1626825e70127
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312643
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-07-23 18:03:11 +00:00
Michael Kelly
c34b91c9d8 manifest: Support ignoring local manifests with 'repo manifest'
Currently users need to look up the baseline manifest by loading the
specific manifest file.  This exposes them to the internals of how the
manifest is stored which may potentially be fragile (eg: It was
switched from a symlink pointing at the file in the report to an
actual file with an 'include' tag).

Instead of doing this, we can provide an option to the 'repo manifest'
command which will emit the baseline manifest and decouple users from
the internal manifest details.

Change-Id: I12ee9160feaa591484ae71f404bc529be500ae4e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311202
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-07-20 04:56:12 +00:00
Michael Kelly
06da9987f6 Gracefully ignore bad remove-project line
Sometimes, we don't care if the remove project is referring to a
non-existing project and we can just ignore it.  This change allows us
to ignore remove-project entries if the project that they refer to
doesn't exist, making them effectively a no-op.

Because this change breaks existing configuration, we allow this to be
configuration controlled using the `optional` attribute in the
remove-project tag.

Change-Id: I6313a02983e81344eadcb4e47d7d6b037ee7420e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310964
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-07-08 16:48:21 +00:00
Jack Neus
b8c84483a5 repo: improve duplicate default check
If one default is totally empty, we don't need to fail.

BUG=b:187795796
TEST=unit tests

Change-Id: Id226a7a7cd183dbdee58f4681b84885cc9211375
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309102
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
2021-06-15 18:06:13 +00:00
Jack Neus
5ba2120362 repo: properly handle NoneType in Default/Remote equality checks
BUG=none
TEST=none

Change-Id: I4ccdbbc7ba4b6f6e20c6959db1b46fdb44ea2819
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308982
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
2021-06-11 13:54:32 +00:00
Raman Tenneti
78f4dd3138 superproject: add projects from local manifest to local::<filename> group.
With repo sync --use-superproject, don't update the commit ids of every project
that comes from local manifest.

Tested the code with the following commands.

$ ./run_tests -v

+ Test with local.xml

1. repo init --use-superproject -u persistent-https://googleplex-android.git.corp.google.com/a/platform/manifest

2. cd .repo
cp -r /google/src/head/depot/google3/wireless/android/build_tools/translations/pipeline/local_manifests local_manifests
cd ..

local$ time repo_dev sync --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../local/.repo/exp-superproject/feb2c2847da5e274f3d530d5ab438af8-superproject.git: Initial setup for superproject completed.
...

Bug: [google internal] b/189360443
Bug: [google internal] b/189139268
Bug: https://crbug.com/gerrit/14499
Change-Id: Ideaf268c294e9b500b2b9726ffbd733dd8d63004
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308822
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2021-06-10 00:16:36 +00:00
Raman Tenneti
993af5e136 superproject: Use bugurl from contactinfo in the missing commits error message.
+ In XmlManifest._Unload set 'bugurl' to Wrapper().BUG_URL.
+ contactinfo returns a namedtuple.
+ bug_url can be accessed as self._manifest.contactinfo.bugurl.

Tested the code with the following commands.

$ ./run_tests -v

Added contactinfo tag to default.xml and verified that bugurl is used.

Bug: [google internal] b/186220520.
Change-Id: Iaafd6465e072b2e47a0a0b548bf6cb608a0b0a04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/306342
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-18 15:35:54 +00:00
Raman Tenneti
1c3f57e8f1 manifest_xml: initial support for <contactinfo>
It will be used to let manifest authors self-register contact info.
This element can be repeated, and any later entries will clobber
earlier ones. This would allow manifest authors who extend
manifests to specify their own contact info.

It would have 1 required attribute: bugurl.
"bugurl" specifies the URL to file a bug against the manifest owner.

<contactinfo bugurl="bug-url"/>

TODO: This CL only implements the parsing logic and further work
will be in followup CLs.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: [google internal] b/186220520.
Change-Id: I47e765ba2dab5cdf850191129f4d4cd6b803f451
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305203
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-04 22:36:01 +00:00
Raman Tenneti
feb28914bd superproject: Don't update the commit ids of projects if remote is different.
1) Skip setting the revision id (commit id) for the projects whose
   remote doesn't match superproject's remote.
2) exp-superproject/superproject_override.xml includes local_manfiest's
   projects. When we load this XML, don't reload projects from local.xml
   (otherwise we will get duplicate projects errors).

Tested the code with the following commands.

$ ./run_tests -v

+ Test with local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml  .repo/local_manifests/local.xml

  $ repo_dev sync -c -j8

+ Test without local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
  $ repo_dev sync -c -j8

Bug: [google internal] b/186395810
Change-Id: I4e9d4ac2d94a9fc0cef0ccd787b6310758009e86
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304882
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-03 05:13:23 +00:00
Mike Frysinger
f69c7ee318 manifest_xml: ban use of newlines in paths
There should be no valid use of these anywhere, so just ban them
to make things easier for people.

Bug: https://crbug.com/gerrit/14156
Bug: https://crbug.com/gerrit/14200
Change-Id: I8d2cf988c510c98194c43a329a2b9bf313a3f0a8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304662
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-30 05:54:11 +00:00
Raman Tenneti
f32f243ff8 init: Added --partial-clone-exclude option.
partial-clone-exclude option excludes projects during
partial clone. This is a comma-delimited project names
(from manifest.xml). This option is persisted and it
is used by the sync command.

A project that has been unparital'ed will remain unpartial if
that project's name is specified in the --partial-clone-exclude
option. The project name should match exactly.

Added
$ ./run_tests -v

Bug: [google internal] b/175712967
"I can't "unpartial" my androidx-main checkout"

$ rm -rf androidx-main/
$ mkdir androidx-main/
$ cd androidx-main/
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml
$ repo_dev sync -c -j8

+ Verify a project is partial
$ cd frameworks/support/
$ git config -l | grep  'partial'

+ Unpartial a project.
$ /google/bin/releases/android/git_repack/git_unpartial

+ Verify project is unpartial
$ git config -l | grep  'partial'
$ cd ../..

+ Exclude the project from being unparial'ed after init and sync.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml

+ Verify project is unpartial
$ cd frameworks/support/
$ git config -l | grep  'partial'
$ cd ../..
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep  'partial'
$ cd ../..

+ Remove the project from exclude list and verify that project is partially cloned.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep  'partial'

Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-04-13 15:47:10 +00:00
Mike Frysinger
0458faa502 manifest: allow toplevel project checkouts
Re-allow checking out projects to the top of the repo client checkout.
We add checks to prevent checking out files under .repo/ as that path
is only managed by us, and projects cannot inject content or settings
into it.

Bug: https://crbug.com/gerrit/14156
Bug: https://crbug.com/gerrit/14200
Change-Id: Id6bf9e882f5be748442b2c35bbeaee3549410b25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299623
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-12 16:31:14 +00:00
Raman Tenneti
080877e413 superproject: pass groups to ToXml method.
Added the following methods to XmlManifest class.
+ GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower()
+ GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func.

+ Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr.
+ Used the above methods to get groups in command.py::GetProjects
  and part of init.py.

TODO: clean up these funcs to take structured group data more instead
      of passing strings around everywhere that need parsing.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL
and verified prebuilts/fullsdk-linux directory has all the folders.

Tested repo init and repo sync with --use-superproject and without
--use-superproject argument.

$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main  --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject

$ repo_dev sync -c -j32

Bug: [google internal] b/181804931
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-11 01:24:52 +00:00
Mike Frysinger
5413397204 manifest: relax include name rules for user-specified path
Allow the user to specify relative or absolute or any other funky
path that they want when using `repo init` or `repo sync`.  Our
goal is to restrict the paths in the remote manifest git repo we
cloned from the network, not protect the user from themselves.

Bug: https://crbug.com/gerrit/14156
Change-Id: I1ccfb2a6bd1dce2bd765e261bef0bbf0f8a9beb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298823
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-02 03:18:57 +00:00
Mike Frysinger
a29424ea6d manifest: validate project name & path and include name attributes
These attribute values are used to construct local filesystem paths,
so apply the existing filesystem checks to them.

Bug: https://crbug.com/gerrit/14156
Change-Id: Ibcceecd60fa74f0eb97cd9ed1a9792e139534ed4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298443
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-28 16:07:12 +00:00
Mike Frysinger
a00c5f40e7 manifest: refactor the filesystem checking logic for more reuse
This function is currently written with copyfile & linkfile in mind.
Generalize the logic & function arguments slightly so we can reuse
in more places that make sense.

This changes the validation logic slightly too in that we no longer
allow "." for the dest attribute with copyfile & linkfile, nor for
the src attribute with copyfile.  We already rejected those later on
when checking against the active filesystem, but now we reject them
a little sooner when parsing.

The empty path check isn't a new requirement exactly -- repo used to
crash on it, so it was effectively blocked, but now we diagnosis it.

Bug: https://crbug.com/gerrit/14156
Change-Id: I0fdb42a3da60ed149ff1997c5dd4b85da70eec3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-28 16:07:12 +00:00
Mike Frysinger
5d9c4972e0 use simpler super() magic
Python 3 has a simpler super() style so switch to it to make the
code a little simpler and to stop pylint warnings.

Change-Id: I1b3ccf57ae968d56a9a0bcfc1258fbd8bfa3afee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297383
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-19 20:06:20 +00:00
Mike Frysinger
c47a235bc5 trim redundant pass statements
Clean up a few linter warnings.

Change-Id: I531d0263a202435d32d83d87ec24998f4051639c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297062
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-16 19:23:00 +00:00
Raman Tenneti
b5c5a5e068 manifest: set revisionId as revision attribute it it is not being set in ToXml.
As we were testing superproject setting revisionId attribute to SHA and
reloading the manifest, we found out revisionId attribute is not being
saved. Made the change to save the revisionId if it is not being saved.

Tested the code with the following commands.

$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I95fdf655b19648ad3e9aba10b9bed8bb9439deb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296182
Reviewed-by: Ian Kasprzak <iankaz@google.com>
2021-02-07 17:13:35 +00:00
Raman Tenneti
1bb4fb222d manifest_xml: initial support for <superproject>
At most one superproject may be specified. It will be used
to specify the URL of superproject.

It would have 3 attributes: remote, name, and default.
Only "name" is required while the others have reasonable defaults.

<remote name="superproject-url" review="<url>" />
<superproject remote="superproject-url" name="platform/superproject"/>

TODO: This CL only implements the parsing logic and further work
will be in followup CLs.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I5b4bba02c8b59601c754cf6b5e4d07a1e16ce167
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292982
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-08 19:49:52 +00:00
Mike Frysinger
acf63b2892 drop pyversion & is_python3 checking
We're committed to Python 3 at this point, so purge all the
is_python3 related dynamic checks.

Bug: https://crbug.com/gerrit/10418
Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:58 +00:00
Mike Frysinger
784ccfc040 strip python2-only coding:utf-8 & print_function settings
We're committed to Python 3 at this point, so clean up boilerplate.

Bug: https://crbug.com/gerrit/10418
Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:05 +00:00
Mike Frysinger
51e39d536d manifest_xml: harmonize list fields
We allow project.groups to be whitespace or comma delimited, but
repo-hooks.enabled-list is only whitespace delimited.  This hasn't
been a big deal as it's only ever had one valid value, but if we
want to add more, we should harmonize these a bit.

Refactor the groups method to be more generic, and run the enabled-
list attribute through it.  Then add missing docs for it.

Change-Id: Iaa96a0faa9c4a68b313b49336751831b73bf855d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290743
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-12-04 17:27:11 +00:00
Fredrik de Groot
352c93b680 manifest: add support for groups in include
Attrib groups can now be added to manifest include, thus
all projects in an included manifest file can easily be tagged
with a group without modifying all projects in that manifest file.

Include groups will add and recurse, meaning included manifest
projects will carry all parent includes. Intentionally, no support
added for group remove, to keep complexity down.

Group handling for projects is untouched, meaning a group set on
a project will still append to whatever was or was not inherited
in parent manifest includes, resulting in union of groups inherited
and set for the project itself.

Test: manual multi-level manifest include structure, in serial and parallel,
      with different groups set on init
Test: added unit tests to cover the inheritance

Change-Id: Id2229aa6fd78d355ba598cc15c701b2ee71e5c6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/283587
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-11-26 09:13:14 +00:00
Mike Frysinger
8c1e9cbef1 manifest_xml: refactor manifest parsing from client management
We conflate the manifest & parsing logic with the management of the
repo client checkout in a single class.  This makes testing just one
part (the manifest parsing) hard as it requires a full checkout too.

Start splitting the two apart into separate classes to make it easy
to reason about & test.

Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-18 19:10:57 +00:00
Mike Frysinger
23411d3f9c manifest: add a --json output option
Sometimes parsing JSON is easier than parsing XML, especially when
the XML format is limited (which ours is).  Add a --json option to
the manifest command to quickly emit that form.

Bug: https://crbug.com/gerrit/11743
Change-Id: Ia2bb254a78ae2b70a851638b4545fcafe8c1a76b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280436
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-17 01:38:00 +00:00
Miguel Gaio
1f20776dbb manifest_xml: correct project revisionId for extend-project
Using sha1 manifest, project's revisionId is initialized
first by the manifest.
An update of a projet revision by extend-project node does
not apply to the revisionId which is therefore kept to the
initial value.

Resets revisionId value when revision is updated by an
extend-project node.

Change-Id: I873af283890cebaeaabde966f04b125642af929f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/275715
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Miguel Gaio <miguel.gaio@renault.com>
2020-11-12 09:00:08 +00:00
Mike Frysinger
4e1fc1013c manifest: drop support for local_manifest.xml
We deprecated this 8 years ago.  Time to drop it to simplify the code
as it'll help with refactoring in this module to not migrate it.

Change-Id: I2deae5496d1f66a4491408fcdc95cd527062f8b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280798
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
2020-09-08 17:00:06 +00:00
Mike Frysinger
cebf227026 manifest: normalize name & path when constructing fs paths
If the manifest uses a trailing slash on the name attribute, repo
will construct bad internal filesystem paths which confuses tools
later on.

For example, this manifest entry:
  <project name="aosp/platform/system/libhidl/" ...
will cause repo to use paths like:
  .repo/project-objects/aosp/platform/system/libhidl/.git/
when it really should be using:
  .repo/project-objects/aosp/platform/system/libhidl.git

Apply the normalization when we construct the local filesystem paths
as we cannot guarantee that the remote URL constructed from these
will behave the same.  A server might really want:
  https://example.com/aosp/platform/system/libhidl/
and would throw an error if we instead tried to fetch:
  https://example.com/aosp/platform/system/libhidl

Unfortunately, any existing repo client checkouts that use such a
manifest will hit a one-time sync error as the internal git location
has changed.  I'm not sure there's a way to cleanly migrate that.

Bug: https://crbug.com/1086043
Change-Id: I30bea0ffd23e478de89a035f408055e48a102658
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268742
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
2020-05-26 05:15:58 +00:00
Xin Li
d79a4bc51b Make partial clone imply no-clone-bundle by default.
For large projects, clone bundle is useful because it provided a way to
efficiently transfer a large portion of git objects through CDN, without
needing to interact with git server. However, with partial clones, the
intention is to not download most of the objects, so the use of clone
bundles would defeat the space savings normally seen with partial
clones, as they are downloaded before the first fetch.

A new option, --clone-bundle is added to override this behavior.
Add a new repo.clonebundle variable which remembers the choice if
explicitly given from command line at repo init.

Change-Id: I03638474af303a82af34579e16cd4700690b5f43
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268452
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-05-21 19:47:36 +00:00
Sean McAllister
af908cb543 When writing manifest, set the dest-branch attribute for projects
When generating a revision locked manifest, we need to know what
ref to push changes to when doing 'repo upload'.  This information
is lost when we lock the revision attribute to a particular commit
hash, so we need to expose it through the dest-branch attribute.

Bug: https://crbug.com/1005103
Test: manual execution
Change-Id: Ib31fd77ad8c9379759c4181dac1ea97de43eec35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263572
Tested-by: Sean McAllister <smcallis@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-04-20 16:35:02 +00:00
Mike Frysinger
bb8ee7f54a manifest_xml: unify bool & int parsing
We've been overly lenient with boolean parsing by ignoring invalid
values as "false" even if the user didn't intend that.  Turn all
unknown values into warnings to avoid breaking existing manifests,
and unify the parsing logic in a helper to simplify.

We've been stricter about numbers, but still copying & pasting
inconsistent code.  Add a helper for this too.  For out of range
sync-j numbers (i.e. less than 1), throw a warning for now, but
mark it for future hard failures.

Change-Id: I924162b8036e6a5f1e31b6ebb24b6a26ed63712d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256457
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-03-13 18:48:52 +00:00
Mike Frysinger
d957ec6a83 manifest_xml: skip config lookup during first init
Trying to use the config state when the git tree hasn't yet been
created hits bad side effects.  Add a check to avoid probing the
config logic during the first run.  It's not clear what's going
wrong at the lower layers, but this gets us back to the behavior
before we added worktree support, so lets settle the status quo.

Bug: https://crbug.com/gerrit/12387
Change-Id: I85b56797455f3c2e249d02c18496e060be05501d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256592
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-24 21:17:08 +00:00
Mike Frysinger
a269b1cb9d manifest_xml: change .repo/manifest.xml to a plain file
Changing this to a file instead of using a symlink serves two purposes:
* We can insert some comments & doc links to help users learn what this
  is for, discover relevant documentation, and to discourage them from
  modifying things.
* Windows requires Administrator access to use symlinks.  With this
  last change, Windows users can get repo client checkouts with the new
  --worktree option and not need symlinks anywhere at all.  Which means
  they no longer need to be an Administrator in order to `repo sync`.

Change-Id: I9bc46824fd8d4b0f446ba84bd764994ca1e597e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256313
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-22 04:38:17 +00:00
Mike Frysinger
d9254599f9 manifest/tests: get them passing under Windows
We also need to check more things in the manifest/project handlers,
and use platform_utils in a few places to address Windows behavior.

Drop Python 2.7 from Windows testing as it definitely doesn't work
and we won't be fixing it.

Change-Id: I83d00ee9f1612312bb3f7147cb9535fc61268245
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256113
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-21 05:17:05 +00:00
Mike Frysinger
979d5bdc3e add experimental git worktree support
This provides initial support for using git worktrees internally
instead of our own ad-hoc symlink tree.  It's been lightly tested
which is why it's not currently exposed via --help.

When people opt-in to worktrees in an existing repo client checkout,
no projects are migrated.  Instead, only new projects will use the
worktree method.  This allows for limited testing/opting in without
having to completely blow things away or get a second checkout.

Bug: https://crbug.com/gerrit/11486
Change-Id: Ic3ff891b30940a6ba497b406b2a387e0a8517ed8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254075
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-19 18:11:33 +00:00
David Pursehouse
819827a42d Fix blank line issues reported by flake8
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E301,E302,E303,E305,E306

Manually fix issues in project.py caused by misuse of block comments.

Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:36:40 +00:00
David Pursehouse
abdf750061 Fix indentation issues reported by flake8
- E121 continuation line under-indented for hanging indent
- E122 continuation line missing indentation or outdented
- E125 continuation line with same indent as next logical line
- E126 continuation line over-indented for hanging indent
- E127 continuation line over-indented for visual indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E121,E122,E125,E126,E127,E128,E129,E131

Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:36:22 +00:00
David Pursehouse
54a4e6007a Fix various whitespace issues reported by pyflakes
- E201 whitespace after '['
- E202 whitespace before '}'
- E221 multiple spaces before operator
- E222 multiple spaces after operator
- E225 missing whitespace around operator
- E226 missing whitespace around arithmetic operator
- E231 missing whitespace after ','
- E261 at least two spaces before inline comment
- E271 multiple spaces after keyword

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E201,E202,E221,E222,E225,E226,E231,E261,E271

Change-Id: I367113eb8c847eb460532c7c2f8643f33040308c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254601
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:00:16 +00:00
David Pursehouse
eeff3537de Fix tests for membership to use 'not in'
flake8 reports:

  E713 test for membership should be 'not in'

Change-Id: I4446be67c431b7267105b53478d2ceba2af758d7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254451
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:18:17 +00:00
David Pursehouse
e5913ae410 Fix flake8 E251 unexpected spaces around keyword / parameter equals
Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place --select E251

Change-Id: I58009e1c8c91c39745d559ac919be331d4cd9e77
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254598
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:17:08 +00:00