Commit Graph

48 Commits

Author SHA1 Message Date
Vitalii Dmitriev
449b23b698 manifest_xml: fix url normalization for inits and remotes
Before the change, repo normalizes the urls
with a following format only:

    git@github.com:foo/bar

It doesn't cover the following case:

   <remote name="org" fetch="git@github.com:org/" />
   <project name="somerepo" remote="org" />

Results to:
   error: Cannot fetch somerepo
     from ssh://git@github.com/org/git@github.com:org/somerepo

Current change fixes it by normalizing this format:

    git@github.com:foo

Test: ./run_tests tests/test_manifest_xml.py
Change-Id: I1ad0f5df0d52c0b7229ba4c9a4db4eecb5c1a003
Signed-off-by: Vitalii Dmitriev <vitalii.dmitriev@unikie.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/398337
Commit-Queue: Vitalii Dmitriev <dmit.vitalii@gmail.com>
Tested-by: Vitalii Dmitriev <dmit.vitalii@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2023-12-20 07:38:49 +00:00
Mike Frysinger
48e4137eba manifest_xml: do not allow / before : in scp-like syntax
Since git doesn't treat these as ssh:// URIs, we shouldn't either.

Bug: https://g-issues.gerritcodereview.com/issues/40010331
Change-Id: I001f49be30395187cac447d09cb5a6c29e95768b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/398517
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jason Chang <jasonnc@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2023-12-19 18:00:44 +00:00
Michael Kelly
3652b497bb Correctly handle schema-less URIs for remote fetch URL
Currently we don't deal with schema-less URIs like
`git@github.com:foo` at all resulting in a scenario where we append
them to the manifest repo URL.

In order to deal with this, we munge both the manifest URL and the
fetch URL into a format we like and proceed with that.

Bug: https://g-issues.gerritcodereview.com/issues/40010331
Change-Id: I7b79fc4ed276630fdbeb235b94e327b172f0879b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/386954
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2023-11-08 05:03:20 +00:00
Jason R. Coombs
b32ccbb66b cleanup: Update codebase to expect Python 3.6
- Bump minimum version to Python 3.6.
- Use f-strings in a lot of places.

Change-Id: I2aa70197230fcec2eff8e7c8eb754f20c08075bb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/389034
Tested-by: Jason R. Coombs <jaraco@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Jason R. Coombs <jaraco@google.com>
2023-10-31 16:03:54 +00:00
Chris Allen
7393f6bc41 manifest_xml: Fix empty project list when DOCTYPE is present
When parsing the manifest XML, the code looks for a top
level DOM node named "manifest". However, it doesn't check
that it's an element type node so if there is also an XML
document type declaration node present (which has the same
name as the root element) then it selects the wrong node
and hence you end up with no projects defined at all.

Change-Id: I8d101caffbbc2a06e56136ff21302e3f09cfc96b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/390357
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Chris Allen <chris.allen@arm.com>
Commit-Queue: Chris Allen <chris.allen@arm.com>
2023-10-20 18:22:59 +00:00
Fredrik de Groot
be71c2f80f manifest: enable remove-project using path
A something.xml that gets included by two different
files, that both remove and add same shared project
to two different locations, would not work
prior to this change.

Reason is that remove killed all name keys, even
though reuse of same repo in different locations
is allowed.

Solve by adding optional attrib path to
<remove-project name="foo" path="only_this_path" />
and tweak remove-project.

Behaves as before without path, and deletes
more selectively when remove path is supplied.

As secondary feature, a project can now also be removed
by only using path, assuming a matching project name
can be found.

Change-Id: I502d9f949f5d858ddc1503846b170473f76dc8e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375694
Tested-by: Fredrik de Groot <fredrik.de.groot@aptiv.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2023-06-21 14:50:16 +00:00
Shuchuan Zeng
3e3340d94f manifest: add support for revision in include
Attribute groups can now be added to manifest include, thus
all projects in an included manifest file can easily modify
default branch without modifying all projects in that manifest file.

For example,
the main manifest.xml has an include node contain revision attribute,
```
<include name="include.xml" revision="r1" />
```
and the include.xml has some projects,
```
<project path="project1_path" name="project1_name" revision="r2" />
<project path="project2_path" name="project2_name" />
```
With this change, the final manifest will have revision="r1" for project2.
```
<project name="project1_name" path="project1_path" revision="r2" />
<project name="project2_name" path="project2_path" revision="r1" />
```

Test: added unit tests to cover the inheritance

Change-Id: I4b8547a7198610ec3a3c6aeb2136e0c0f3557df0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/369714
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Shuchuan Zeng <zengshuchuan@allwinnertech.com>
Tested-by: Shuchuan Zeng <zengshuchuan@allwinnertech.com>
2023-05-05 03:40:28 +00:00
Gavin Mak
ea2e330e43 Format codebase with black and check formatting in CQ
Apply rules set by https://gerrit-review.googlesource.com/c/git-repo/+/362954/ across the codebase and fix any lingering errors caught
by flake8. Also check black formatting in run_tests (and CQ).

Bug: b/267675342
Change-Id: I972d77649dac351150dcfeb1cd1ad0ea2efc1956
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/363474
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2023-03-22 17:46:28 +00:00
Mike Frysinger
e815286492 tests: clean up repo_trace._TRACE_FILE patching
Patch this automatically for all tests rather than duplicating the
boilerplate in diff testcases.

Change-Id: I391d5c859974cda3d5680d34ede2ce6e9e925838
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354358
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-12-08 22:22:39 +00:00
Daniel Kutik
3593a10643 test_bad_path_name_checks: allow Windows path sep
With this change if a path ends with '/' on Linux/macOS
and ends with either '/' or '\' on Windows, the test will pass.

Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: Id7d1b134f9c0bdf7ceaf149af304bbf90cbd7b21
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353180
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-11-28 04:07:17 +00:00
Daniel Kutik
003684b6e5 test: Fix char encoding issues on windows
Some tests were failing due to Windows not using utf-8
by default when executing the tests. Enforcing usage of utf-8
resolves these issues.

Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: If42f6be2a2b688a6105ecf4fcdb541aade24519a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353179
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-11-27 17:03:41 +00:00
Daniel Kutik
0297f8312c test: fix path seperator errors on windows
Fixing multiple errors when running tests on Windows related
to path seperator being different ('\' instead of '/').

Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: I26b44d092b925edecab46a4d88e77dd9dcb8df28
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353178
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-11-27 17:03:01 +00:00
Joanna Wang
a6c52f566a Set tracing to always on and save to .repo/TRACE_FILE.
- add `--trace_to_stderr` option so stderr will include trace outputs and any other errors that get sent to stderr
- while TRACE_FILE will only include trace outputs

piggy-backing on: https://gerrit-review.googlesource.com/c/git-repo/+/349154

Change-Id: I3895a84de4b2784f17fac4325521cd5e72e645e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350114
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Joanna Wang <jojwang@google.com>
2022-11-03 21:07:07 +00:00
Peter Kjellerstedt
0d130d2da0 tests: Make the tests pass for Python < 3.8
Before Python 3.8, xml.dom.minidom sorted the attributes of an element
when writing it to a file, while later versions output the attributes
in the order they were created. Avoid these differences by sorting the
attributes for each element before comparing the generated manifests
with the expected ones.

This corresponds to commit 5d58c18, but for new tests introduced since
it was integrated.

Change-Id: I5c360656a0968e6e8d57eb068c8e87da7dfa61c1
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349917
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-10-28 17:26:48 +00:00
Erik Elmeke
4cdfdb7734 manifest: allow extend-project to override dest-branch and upstream
Bug: https://crbug.com/gerrit/16238
Change-Id: Id6eff34791525b3df690e160c911c0286331984b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/345144
Tested-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-09-20 04:25:02 +00:00
LaMont Jones
a8cf575d68 Omit local_manifest groups from superproject override.
When we create superproject_override.xml, do not include projects that
are present from local_manifests/*.  Such projects are fully under the
control of the local_manifests/ file.

Bug: b/238934278
Test: manual, ./run_tests
Change-Id: I40382ceb82d9cf7b8dc7b5f2abed3f6d4d80017e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340877
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Sam Saccone 🐐 <samccone@google.com>
2022-07-15 23:32:24 +00:00
Mike Frysinger
74737da1ab tests: switch to tempfile.TemporaryDirectory
Now that we don't need to support Python 2, we can switch to this
API for better contextmanager logic.

Change-Id: I2d03e391121886547e7808a3b5c3b470c411533f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337515
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-05-20 11:38:10 +00:00
LaMont Jones
d56e2eb421 manifest_xml: use Superproject to hold XML content
Always create Superproject when there is a <superproject> tag, and have
it hold the XML content, similar to how other manifest elements are
handled.

This also adds SetQuiet and SetPrintMessages to Superproject
consistent with manifest.SetUseLocalManifests.

Change-Id: I522bf3da542006575799f0640c67f7052704f266
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334641
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-04-12 15:46:23 +00:00
Michael Kelly
37c21c268b Add 'dest-path' to extend-project to support changing path
This allows us to move the repository to a new location in the source
tree without having to remove-project + add a new project tag.

Change-Id: I4dba6151842e57f6f2b8fe60cda260ecea68b7b4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310962
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
2021-09-28 20:12:00 +00:00
Xin Li
e0b16a22a0 superproject: support a new revision attribute.
Tested:
$ ./run_tests

Verified that a manifest that specified superproject revision would use
the specified revision, and superproject will use the default revision.

Note that this is a slight behavior change from earlier repo versions,
which would always use the branch name of the manifest itself. However,
the new behavior would be more consisitent with regular "project"
element and would allow superproject be used even if it is not enabled
for the particular manifest branch, so we have decided to make the
change as it would provide more flexibility and better matches what
other elements would do.

Bug: [google internal] b/187868160
Change-Id: I35255ee347aff6e65179f7879d52931f168b477e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317643
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-27 06:36:05 +00:00
Jack Neus
a84f43a006 manifest: make repo-hooks more robust wrt element ordering
Currently, repo will fail to sync to a manifest if the definition
of the repo-hooks project comes after the repo-hooks element.

BUG=none
TEST=new test, run_tests

Change-Id: I0bf85625173492af6c6404d4b67543e96e670562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318520
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
2021-09-23 21:17:38 +00:00
Jack Neus
6ea0caea86 repo: properly handle remote annotations in manifest_xml
BUG=b:192664812
TEST=tests/

Change-Id: I1aa50260f4a00d3cebbd531141e1626825e70127
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312643
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-07-23 18:03:11 +00:00
Michael Kelly
06da9987f6 Gracefully ignore bad remove-project line
Sometimes, we don't care if the remove project is referring to a
non-existing project and we can just ignore it.  This change allows us
to ignore remove-project entries if the project that they refer to
doesn't exist, making them effectively a no-op.

Because this change breaks existing configuration, we allow this to be
configuration controlled using the `optional` attribute in the
remove-project tag.

Change-Id: I6313a02983e81344eadcb4e47d7d6b037ee7420e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310964
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-07-08 16:48:21 +00:00
Xin Li
0e776a5837 Fix an issue when syncing with --use-superproject and clone bundles.
It is possible that a clone bundle contained the object referenced by
the branch in the manifest and in the superproject, but not the branch
itself (for example, the branch may be newly created from an existing
branch, or is not vislble to the user downloading the clone bundle).

When --use-superproject is enabled, because we are overriding
revisionExpr with the SHA1 revision provided by the superproject, the
verification step would succeed, but because the expected branch do not
exist, it would confuse git-repo at a later time, as it is expecting the
remote branch to exist in the local clone.

In project.py, fix this by making SetRevisionId() to always remember
the actual branch name and verify it in _CheckForImmutableRevision()
so that we only skip the fetch step when both objects exists locally.

Bug: [google internal] b/191974277
Change-Id: I49d3ca0667f524c8c45f416492faf95b1dd822fb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
2021-06-30 15:31:15 +00:00
Jack Neus
5ba2120362 repo: properly handle NoneType in Default/Remote equality checks
BUG=none
TEST=none

Change-Id: I4ccdbbc7ba4b6f6e20c6959db1b46fdb44ea2819
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308982
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
2021-06-11 13:54:32 +00:00
Peter Kjellerstedt
5d58c18146 tests: Make the tests pass for Python < 3.8
Before Python 3.8, xml.dom.minidom sorted the attributes of an element
when writing it to a file, while later versions output the attributes
in the order they were created. Avoid these differences by sorting the
attributes for each element before comparing the generated manifests
with the expected ones.

Bug: https://crbug.com/gerrit/14382
Change-Id: Ie2597727afcc48f9063a7261ad970e8a549f0587
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303326
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-26 13:36:20 +00:00
Raman Tenneti
993af5e136 superproject: Use bugurl from contactinfo in the missing commits error message.
+ In XmlManifest._Unload set 'bugurl' to Wrapper().BUG_URL.
+ contactinfo returns a namedtuple.
+ bug_url can be accessed as self._manifest.contactinfo.bugurl.

Tested the code with the following commands.

$ ./run_tests -v

Added contactinfo tag to default.xml and verified that bugurl is used.

Bug: [google internal] b/186220520.
Change-Id: Iaafd6465e072b2e47a0a0b548bf6cb608a0b0a04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/306342
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-18 15:35:54 +00:00
Raman Tenneti
2f8fdbecde manifest_xml: cleanup of contactinfo test for readability with f-strings.
Tested the code with the following commands.

$ ./run_tests -v

Bug: [google internal] b/186220520.
Change-Id: I1c0f8958ff4c615707eec218250e8de753ec6562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305282
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-05-05 15:59:58 +00:00
Raman Tenneti
1c3f57e8f1 manifest_xml: initial support for <contactinfo>
It will be used to let manifest authors self-register contact info.
This element can be repeated, and any later entries will clobber
earlier ones. This would allow manifest authors who extend
manifests to specify their own contact info.

It would have 1 required attribute: bugurl.
"bugurl" specifies the URL to file a bug against the manifest owner.

<contactinfo bugurl="bug-url"/>

TODO: This CL only implements the parsing logic and further work
will be in followup CLs.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: [google internal] b/186220520.
Change-Id: I47e765ba2dab5cdf850191129f4d4cd6b803f451
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305203
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-04 22:36:01 +00:00
Mike Frysinger
f69c7ee318 manifest_xml: ban use of newlines in paths
There should be no valid use of these anywhere, so just ban them
to make things easier for people.

Bug: https://crbug.com/gerrit/14156
Bug: https://crbug.com/gerrit/14200
Change-Id: I8d2cf988c510c98194c43a329a2b9bf313a3f0a8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304662
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-30 05:54:11 +00:00
Mike Frysinger
0458faa502 manifest: allow toplevel project checkouts
Re-allow checking out projects to the top of the repo client checkout.
We add checks to prevent checking out files under .repo/ as that path
is only managed by us, and projects cannot inject content or settings
into it.

Bug: https://crbug.com/gerrit/14156
Bug: https://crbug.com/gerrit/14200
Change-Id: Id6bf9e882f5be748442b2c35bbeaee3549410b25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299623
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-12 16:31:14 +00:00
Raman Tenneti
080877e413 superproject: pass groups to ToXml method.
Added the following methods to XmlManifest class.
+ GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower()
+ GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func.

+ Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr.
+ Used the above methods to get groups in command.py::GetProjects
  and part of init.py.

TODO: clean up these funcs to take structured group data more instead
      of passing strings around everywhere that need parsing.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL
and verified prebuilts/fullsdk-linux directory has all the folders.

Tested repo init and repo sync with --use-superproject and without
--use-superproject argument.

$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main  --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject

$ repo_dev sync -c -j32

Bug: [google internal] b/181804931
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-11 01:24:52 +00:00
Mike Frysinger
5413397204 manifest: relax include name rules for user-specified path
Allow the user to specify relative or absolute or any other funky
path that they want when using `repo init` or `repo sync`.  Our
goal is to restrict the paths in the remote manifest git repo we
cloned from the network, not protect the user from themselves.

Bug: https://crbug.com/gerrit/14156
Change-Id: I1ccfb2a6bd1dce2bd765e261bef0bbf0f8a9beb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298823
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-02 03:18:57 +00:00
Mike Frysinger
a29424ea6d manifest: validate project name & path and include name attributes
These attribute values are used to construct local filesystem paths,
so apply the existing filesystem checks to them.

Bug: https://crbug.com/gerrit/14156
Change-Id: Ibcceecd60fa74f0eb97cd9ed1a9792e139534ed4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298443
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-28 16:07:12 +00:00
Mike Frysinger
a00c5f40e7 manifest: refactor the filesystem checking logic for more reuse
This function is currently written with copyfile & linkfile in mind.
Generalize the logic & function arguments slightly so we can reuse
in more places that make sense.

This changes the validation logic slightly too in that we no longer
allow "." for the dest attribute with copyfile & linkfile, nor for
the src attribute with copyfile.  We already rejected those later on
when checking against the active filesystem, but now we reject them
a little sooner when parsing.

The empty path check isn't a new requirement exactly -- repo used to
crash on it, so it was effectively blocked, but now we diagnosis it.

Bug: https://crbug.com/gerrit/14156
Change-Id: I0fdb42a3da60ed149ff1997c5dd4b85da70eec3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-28 16:07:12 +00:00
Mike Frysinger
37ac3d626f tests: refactor manifest tests
The XmlManifestTests class is getting to be large and we're only
adding more to it.  Factor out the core logic into a new TestCase
so we can reuse it to better group more tests.

Change-Id: I5113444a4649a70ecfa8d83d3305959a953693f7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298222
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 17:06:56 +00:00
Raman Tenneti
b5c5a5e068 manifest: set revisionId as revision attribute it it is not being set in ToXml.
As we were testing superproject setting revisionId attribute to SHA and
reloading the manifest, we found out revisionId attribute is not being
saved. Made the change to save the revisionId if it is not being saved.

Tested the code with the following commands.

$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I95fdf655b19648ad3e9aba10b9bed8bb9439deb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296182
Reviewed-by: Ian Kasprzak <iankaz@google.com>
2021-02-07 17:13:35 +00:00
Raman Tenneti
6a872c9dae sync: Added --use-superproject option and support for superproject.
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,

.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-21 19:41:52 +00:00
Raman Tenneti
48b2d10d8f manifest_xml: - Added doc and testing of unknown tags/elements.
Added this test to verify that older versions of repo can handle
"<superproject" element. Tested by adding "<iankaz" unknown element.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I858d56f38cefcfcd14474efdd631a5a940c3ce47
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293482
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-12 21:49:13 +00:00
Raman Tenneti
1bb4fb222d manifest_xml: initial support for <superproject>
At most one superproject may be specified. It will be used
to specify the URL of superproject.

It would have 3 attributes: remote, name, and default.
Only "name" is required while the others have reasonable defaults.

<remote name="superproject-url" review="<url>" />
<superproject remote="superproject-url" name="platform/superproject"/>

TODO: This CL only implements the parsing logic and further work
will be in followup CLs.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I5b4bba02c8b59601c754cf6b5e4d07a1e16ce167
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292982
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-08 19:49:52 +00:00
Mike Frysinger
784ccfc040 strip python2-only coding:utf-8 & print_function settings
We're committed to Python 3 at this point, so clean up boilerplate.

Bug: https://crbug.com/gerrit/10418
Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:05 +00:00
Mike Frysinger
51e39d536d manifest_xml: harmonize list fields
We allow project.groups to be whitespace or comma delimited, but
repo-hooks.enabled-list is only whitespace delimited.  This hasn't
been a big deal as it's only ever had one valid value, but if we
want to add more, we should harmonize these a bit.

Refactor the groups method to be more generic, and run the enabled-
list attribute through it.  Then add missing docs for it.

Change-Id: Iaa96a0faa9c4a68b313b49336751831b73bf855d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290743
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-12-04 17:27:11 +00:00
Fredrik de Groot
352c93b680 manifest: add support for groups in include
Attrib groups can now be added to manifest include, thus
all projects in an included manifest file can easily be tagged
with a group without modifying all projects in that manifest file.

Include groups will add and recurse, meaning included manifest
projects will carry all parent includes. Intentionally, no support
added for group remove, to keep complexity down.

Group handling for projects is untouched, meaning a group set on
a project will still append to whatever was or was not inherited
in parent manifest includes, resulting in union of groups inherited
and set for the project itself.

Test: manual multi-level manifest include structure, in serial and parallel,
      with different groups set on init
Test: added unit tests to cover the inheritance

Change-Id: Id2229aa6fd78d355ba598cc15c701b2ee71e5c6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/283587
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-11-26 09:13:14 +00:00
Mike Frysinger
8c1e9cbef1 manifest_xml: refactor manifest parsing from client management
We conflate the manifest & parsing logic with the management of the
repo client checkout in a single class.  This makes testing just one
part (the manifest parsing) hard as it requires a full checkout too.

Start splitting the two apart into separate classes to make it easy
to reason about & test.

Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-18 19:10:57 +00:00
Mike Frysinger
bb8ee7f54a manifest_xml: unify bool & int parsing
We've been overly lenient with boolean parsing by ignoring invalid
values as "false" even if the user didn't intend that.  Turn all
unknown values into warnings to avoid breaking existing manifests,
and unify the parsing logic in a helper to simplify.

We've been stricter about numbers, but still copying & pasting
inconsistent code.  Add a helper for this too.  For out of range
sync-j numbers (i.e. less than 1), throw a warning for now, but
mark it for future hard failures.

Change-Id: I924162b8036e6a5f1e31b6ebb24b6a26ed63712d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256457
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-03-13 18:48:52 +00:00
Mike Frysinger
d9254599f9 manifest/tests: get them passing under Windows
We also need to check more things in the manifest/project handlers,
and use platform_utils in a few places to address Windows behavior.

Drop Python 2.7 from Windows testing as it definitely doesn't work
and we won't be fixing it.

Change-Id: I83d00ee9f1612312bb3f7147cb9535fc61268245
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256113
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-21 05:17:05 +00:00
Mike Frysinger
ae62541005 manifest_xml: allow src=. with symlinks
Some Android/Nest manifests are using <linkfile> with src="." to
create stable paths to specific projects.  Allow that specific
use case as it seems reasonable to support.

Bug: https://crbug.com/gerrit/11218
Change-Id: I5eadec257cd58ba0f8687c590ddc250a7a414a85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254276
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-10 23:19:31 +00:00
Mike Frysinger
04122b7261 manifest: add basic path checks for <copyfile> & <linkfile>
Reject paths in <copyfile> & <linkfile> that point outside of their
respective scopes.  This validates paths while parsing the manifest
as this should be quick & cheap: we don't access the filesystem as
this code runs before we've synced.

Bug: https://crbug.com/gerrit/11218
Change-Id: I8e17bb91f3f5b905a9d76391b29fbab4cb77aa58
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232932
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
2020-02-04 20:34:01 +00:00