Before the change, repo normalizes the urls
with a following format only:
git@github.com:foo/bar
It doesn't cover the following case:
<remote name="org" fetch="git@github.com:org/" />
<project name="somerepo" remote="org" />
Results to:
error: Cannot fetch somerepo
from ssh://git@github.com/org/git@github.com:org/somerepo
Current change fixes it by normalizing this format:
git@github.com:foo
Test: ./run_tests tests/test_manifest_xml.py
Change-Id: I1ad0f5df0d52c0b7229ba4c9a4db4eecb5c1a003
Signed-off-by: Vitalii Dmitriev <vitalii.dmitriev@unikie.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/398337
Commit-Queue: Vitalii Dmitriev <dmit.vitalii@gmail.com>
Tested-by: Vitalii Dmitriev <dmit.vitalii@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Currently we don't deal with schema-less URIs like
`git@github.com:foo` at all resulting in a scenario where we append
them to the manifest repo URL.
In order to deal with this, we munge both the manifest URL and the
fetch URL into a format we like and proceed with that.
Bug: https://g-issues.gerritcodereview.com/issues/40010331
Change-Id: I7b79fc4ed276630fdbeb235b94e327b172f0879b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/386954
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
- Bump minimum version to Python 3.6.
- Use f-strings in a lot of places.
Change-Id: I2aa70197230fcec2eff8e7c8eb754f20c08075bb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/389034
Tested-by: Jason R. Coombs <jaraco@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Jason R. Coombs <jaraco@google.com>
When parsing the manifest XML, the code looks for a top
level DOM node named "manifest". However, it doesn't check
that it's an element type node so if there is also an XML
document type declaration node present (which has the same
name as the root element) then it selects the wrong node
and hence you end up with no projects defined at all.
Change-Id: I8d101caffbbc2a06e56136ff21302e3f09cfc96b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/390357
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Chris Allen <chris.allen@arm.com>
Commit-Queue: Chris Allen <chris.allen@arm.com>
The logic in projectsDiff performs various operations which
suggest that a set is more appropriate than a list:
- membership lookup ("in")
- removal
Also, sorting can be performed on the the remaining elements at the
end (which will usually involve a much smaller number of elements).
(The performance gain is invisible in comparison to the time being
spent performing git operations).
Cosmetic chance:
- the definition of 'fromProj' is moved to be used in more places
- the values in diff["added"] are added with a single call to extend
Change-Id: I5ed22ba73b50650ca2d3a49a1ae81f02be3b3055
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/383434
Tested-by: Sylvain Desodt <sylvain.desodt@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Sylvain Desodt <sylvain.desodt@gmail.com>
A something.xml that gets included by two different
files, that both remove and add same shared project
to two different locations, would not work
prior to this change.
Reason is that remove killed all name keys, even
though reuse of same repo in different locations
is allowed.
Solve by adding optional attrib path to
<remove-project name="foo" path="only_this_path" />
and tweak remove-project.
Behaves as before without path, and deletes
more selectively when remove path is supplied.
As secondary feature, a project can now also be removed
by only using path, assuming a matching project name
can be found.
Change-Id: I502d9f949f5d858ddc1503846b170473f76dc8e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375694
Tested-by: Fredrik de Groot <fredrik.de.groot@aptiv.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
An investigation go/git-repo-shallow shows a number of problems
when doing a shallow git fetch/clone. This change introduces an
environment variable REPO_ALLOW_SHALLOW. When this environment variable
is set to 1 during a repo init or repo sync all shallow git fetch
commands are replaced with partial fetch commands. Any shallow
repository needing update is unshallowed. This behavior continues until
a subsequent repo sync command is run with REPO_ALLOW_SHALLOW set to 1.
Bug: b/274340522
Change-Id: I1c3188270629359e52449788897d9d4988ebf280
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374754
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Attribute groups can now be added to manifest include, thus
all projects in an included manifest file can easily modify
default branch without modifying all projects in that manifest file.
For example,
the main manifest.xml has an include node contain revision attribute,
```
<include name="include.xml" revision="r1" />
```
and the include.xml has some projects,
```
<project path="project1_path" name="project1_name" revision="r2" />
<project path="project2_path" name="project2_name" />
```
With this change, the final manifest will have revision="r1" for project2.
```
<project name="project1_name" path="project1_path" revision="r2" />
<project name="project2_name" path="project2_path" revision="r1" />
```
Test: added unit tests to cover the inheritance
Change-Id: I4b8547a7198610ec3a3c6aeb2136e0c0f3557df0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/369714
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Shuchuan Zeng <zengshuchuan@allwinnertech.com>
Tested-by: Shuchuan Zeng <zengshuchuan@allwinnertech.com>
Apply rules set by https://gerrit-review.googlesource.com/c/git-repo/+/362954/ across the codebase and fix any lingering errors caught
by flake8. Also check black formatting in run_tests (and CQ).
Bug: b/267675342
Change-Id: I972d77649dac351150dcfeb1cd1ad0ea2efc1956
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/363474
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
By default there are 4 categories in the diffmanifests
api puts the diffs in to - added, removed, changed and unreachable
Example of command - repo diffmanifests 1.xml 2.xml
added - list down the projects present in second manifest but not in
first
removed - list down the projects present in first but not in
second
changed - list down the changes and the differences for each project
unreachable - when it encounters revision value in a project is incorrect
But, when there are projects present in both manifests and could not
find in local workspace where we have cloned the repo(because of
different/subset manifest xml) - this will create unhandled exception
Now we have added a 5th category called 'missing' - where in such
cases it will handle the scenario and print the log for user
Example:
added projects :
project_2 at revision e6c8a59832c05dc4b6a68cee6bc0feb832181725
removed projects :
project_1 at revision e6c8a59832c05dc4b6a68cee6bc0feb832181725
changed projects :
project_3 changed from 3bb890e1286f04e84d505e5db48e0ada89892331 to e434b3736f11537c67590fefadfe4495895e9785
missing projects :
project_4
Change-Id: I244e8389bff7e95664c29d3dcb61e22308e3a573
Signed-off-by: Shashank Devaraj <shashankkarthik@gmail.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344774
Reviewed-by: Mike Frysinger <vapier@google.com>
We almost always use self.remote.name when calling self.GetRemote,
so make that the default to simplify the code a bit.
Change-Id: Ifdf6e1370d6b8963b44e6d384b0fac8fa5c4f2ba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343184
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
For --jobs-network, the logic is now:
* If the user specifies --jobs-network, use that.
* Else, if the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to 1.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.
For --jobs-checkout, the logic is now:
* If the user specifies --jobs-checkout, use that.
* Else, if the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to DEFAULT_LOCAL_JOBS which is based on user's ncpus.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.
For garbage collecting, the logic is now:
* If the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to the user's ncpus.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.
Having to factor in the manifest settings makes this more complicated
which is why we delay processing of defaults until after we've synced
the manifest projects.
Bug: http://b/239712300
Change-Id: Id27cda63c76c156f1d63f6a20cb2c4ceeb3d547c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/341394
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
When we create superproject_override.xml, do not include projects that
are present from local_manifests/*. Such projects are fully under the
control of the local_manifests/ file.
Bug: b/238934278
Test: manual, ./run_tests
Change-Id: I40382ceb82d9cf7b8dc7b5f2abed3f6d4d80017e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340877
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Sam Saccone 🐐 <samccone@google.com>
If the user says `--no-use-superproject`, then do not bother syncing the
superproject.
Also add/update docstrings and comments throughout.
Change-Id: I9cdad706130501bab9a22d3099a1dae605e9c194
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/338975
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
For now, this is opt-in via environment variables:
- export REPO_USE_ALTERNATES=1
The shared project logic that shares the internal .git/objects/ dir
directly between multiple projects via the project-objects/ tree has
a lot of KI with random corruption. It all boils down to projects
sharing objects/ but not refs/. Git operations that use refs to see
what objects are reachable and discard the rest can easily discard
objects that are used by other projects.
Consider this project layout:
<show fs layout>
There are unique refs in each of these trees that are not visible in
the others. This means it's not safe to run basic operations like
git prune or git gc.
Since we can't share refs (each project needs to have unique refs
like HEAD in order to function), let's change how we share objects.
The old way involved symlinking .git/objects/ to the project-objects
tree. The new way shares objects using git's info/alternates.
This means project-objects/ will only contain objects that exist in
the remote project. Local per-project objects (like when creating
branches and making changes) will never be shared. When running a
prune or gc operation in the per-project state, it will only ever
repack or discard those per-project objects. The common shared
objects would only be cleaned up when running a common operation
(i.e. by repo itself).
One downside to this for users is if they try blending unrelated
upstream projects. For example, in CrOS we have multiple kernel
projects (for diff versions) checked out. If a dev fetched the
upstream Linus tree into one of them, the objects & tags would
not be shared with the others, so they would have to fetch the
upstream state for each project. Annoying, but better than the
current corruption situation we're in now.
Also if the dev runs a manual `git fetch` in the per-project to
sync it up to newer state than the last `repo sync` they ran,
the objects would get duplicated. However, git operations later
on should eventually dedupe this.
Bug: https://crbug.com/gerrit/15553
Change-Id: I313a9b8962f9d439ef98ac0ed37ecfb9e0b3864e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328101
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
When the user does not specify any manifest groups, this allows the
parent manifest to indicate which manifest groups should be used for
syncing the submanifest.
Change-Id: I88806ed35013d13dd2ab3cd245fcd4f9061112c4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335474
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
When a manifest file is overridden, remember that and keep using the
override for the remainder of the process. If we need to revert it,
make the override name evaluate False.
Change-Id: I1eee05fec6988c1ee4a3c751c4b540d5b5d11797
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335136
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Always create Superproject when there is a <superproject> tag, and have
it hold the XML content, similar to how other manifest elements are
handled.
This also adds SetQuiet and SetPrintMessages to Superproject
consistent with manifest.SetUseLocalManifests.
Change-Id: I522bf3da542006575799f0640c67f7052704f266
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334641
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
- do not call the internal method from subcmds/sync.py.
- use the correct default groups for submanifests.
- only sync the superproject when we are told to.
Change-Id: I81e4025058f1ee564732b9e17aecc522f6b5f626
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334639
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Also fixes some typos
Change-Id: Id2ba5834ba3a74ed3f29c36d2c0030737dc63e35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334579
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Use ManifestProject properties for config values.
Change-Id: Ib4ad90b0d9a089916e35615b8058942e6d01dc04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334519
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Create RepoProject and ManifestProject, inheriting from MetaProject,
with methods separated for isolation and clarity.
Change-Id: Ic1d6efc65c99470290fea612e2abaf8670d199f4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334139
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Correctly pass the full path of the manifest file for the submanifest.
The manifest-name in the <submanifest/> element was being passed in
as given, which caused it to not be found since the current directory
never set. (b/226333721: fails when manifest-name is given.)
Also verify that the manifest_file passed to XmlManifest() is an
absolute path.
Bug: https://b.corp.google.com/issues/226333721
Change-Id: I23461078233e34562bc2eafeb732cfe8bd38ddc1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/333861
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Add all projects in a submanifest to the group
submanifest::<path_prefix> for ease in filtering.
Change-Id: Ia6f01f9445f4f8d20fda3402f4d5821c43ceaf7f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/331319
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
To be addressed in another change:
- a partial `repo sync` (with a list of projects/paths to sync)
requires `--this-tree-only`.
Change-Id: I6c7400bf001540e9d7694fa70934f8f204cb5f57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322657
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This removes the need for git_superproject to include manifest_xml, and
puts the logic for local_manifest detection in one place.
Change-Id: I4d33ded0542ceea4606a1ea24304f678de20c59e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/330499
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
It was reported that git-lfs did not work with git-repo. Specifically,
`git read-tree -u` run by `repo sync` would fail git-lfs's smudge
filter. See https://github.com/github/git-lfs/issues/1422.
In fact, by the time `git read-tree -u` is run, the repository is not
bare. It is just that, the working directory is not the same as the
.git directory. git-lfs's filter should work. No one seems to have
delved into that issue.
Today, with newer versions of git-repo and git-lfs, that issue will
not reproduce. Tested with
- git 2.33, git-lfs 2.13 on macOS
- git 2.17, git-lfs 2.3 on ubuntu
So, it seems fine to add an option --enable-git-lfs-filter, default to
false, and stat that it may not work with older versions of git and
git-lfs in the help doc.
Bug: https://crbug.com/gerrit/14516
Change-Id: I8d21854eeeea541e072f63d6b10ad1253b1a9826
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328359
Tested-by: XD Trol <milestonejxd@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
When a hard revision ID is provided in a regular project tag then the
revisionId is updated as well if it is a commit hash. The difference
is that if the revisionExpr is a commit, git-repo needs to update
refs/remotes/m/master with update-ref not symbolic-ref, as the latter
must refer to another ref, not to a specific commit.
Change-Id: I215a62dabb30225e480ad2c731416d775fc0c750
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310963
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This allows us to move the repository to a new location in the source
tree without having to remove-project + add a new project tag.
Change-Id: I4dba6151842e57f6f2b8fe60cda260ecea68b7b4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310962
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
Some of the file removal calls are subject to race conditions (if
something else deletes the file), so extend our remove API to have
an option to ignore ENOENT errors. Then update a bunch of random
call sites to use this new functionality.
Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195
Reviewed-by: Sean McAllister <smcallis@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Tested:
$ ./run_tests
Verified that a manifest that specified superproject revision would use
the specified revision, and superproject will use the default revision.
Note that this is a slight behavior change from earlier repo versions,
which would always use the branch name of the manifest itself. However,
the new behavior would be more consisitent with regular "project"
element and would allow superproject be used even if it is not enabled
for the particular manifest branch, so we have decided to make the
change as it would provide more flexibility and better matches what
other elements would do.
Bug: [google internal] b/187868160
Change-Id: I35255ee347aff6e65179f7879d52931f168b477e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317643
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Currently, repo will fail to sync to a manifest if the definition
of the repo-hooks project comes after the repo-hooks element.
BUG=none
TEST=new test, run_tests
Change-Id: I0bf85625173492af6c6404d4b67543e96e670562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318520
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
Currently users need to look up the baseline manifest by loading the
specific manifest file. This exposes them to the internals of how the
manifest is stored which may potentially be fragile (eg: It was
switched from a symlink pointing at the file in the report to an
actual file with an 'include' tag).
Instead of doing this, we can provide an option to the 'repo manifest'
command which will emit the baseline manifest and decouple users from
the internal manifest details.
Change-Id: I12ee9160feaa591484ae71f404bc529be500ae4e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311202
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Sometimes, we don't care if the remove project is referring to a
non-existing project and we can just ignore it. This change allows us
to ignore remove-project entries if the project that they refer to
doesn't exist, making them effectively a no-op.
Because this change breaks existing configuration, we allow this to be
configuration controlled using the `optional` attribute in the
remove-project tag.
Change-Id: I6313a02983e81344eadcb4e47d7d6b037ee7420e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310964
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If one default is totally empty, we don't need to fail.
BUG=b:187795796
TEST=unit tests
Change-Id: Id226a7a7cd183dbdee58f4681b84885cc9211375
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309102
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
With repo sync --use-superproject, don't update the commit ids of every project
that comes from local manifest.
Tested the code with the following commands.
$ ./run_tests -v
+ Test with local.xml
1. repo init --use-superproject -u persistent-https://googleplex-android.git.corp.google.com/a/platform/manifest
2. cd .repo
cp -r /google/src/head/depot/google3/wireless/android/build_tools/translations/pipeline/local_manifests local_manifests
cd ..
local$ time repo_dev sync --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../local/.repo/exp-superproject/feb2c2847da5e274f3d530d5ab438af8-superproject.git: Initial setup for superproject completed.
...
Bug: [google internal] b/189360443
Bug: [google internal] b/189139268
Bug: https://crbug.com/gerrit/14499
Change-Id: Ideaf268c294e9b500b2b9726ffbd733dd8d63004
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308822
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
+ In XmlManifest._Unload set 'bugurl' to Wrapper().BUG_URL.
+ contactinfo returns a namedtuple.
+ bug_url can be accessed as self._manifest.contactinfo.bugurl.
Tested the code with the following commands.
$ ./run_tests -v
Added contactinfo tag to default.xml and verified that bugurl is used.
Bug: [google internal] b/186220520.
Change-Id: Iaafd6465e072b2e47a0a0b548bf6cb608a0b0a04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/306342
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
