The ssh master logic has never worked under Windows which is why this
code always returned False when running there (including cygwin). But
the OS check was still done while holding the threading lock. While
it might be a little slower than necessary, it still worked.
The switch from the threading module to the multiprocessing module
changed global behavior subtly under Windows and broke things: the
globals previously would stay valid, but now they get cleared. So
the lock is reset to None in children workers.
We could tweak the logic to pass the lock through, but there isn't
much point when the rest of the code is still disabled in Windows.
So perform the platform check before we grab the lock. This fixes
the crash, and probably speeds things up a few nanoseconds.
This shouldn't be a problem on Linux systems as the platform fork
will duplicate the existing process memory (including globals).
Bug: https://crbug.com/gerrit/14480
Change-Id: I1d1da82c6d7bd6b8cdc1f03f640a520ecd047063
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305149
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Default the partial_clone_exclude argument to an empty set.
Fixes the following report by Emil Medve.
With this change (up to v2.14.1), on an existing "normal" clone (without partial-clone options) I'm seeing this traceback during `repo selfupdate`:
Traceback (most recent call last):
File ".../.repo/repo/main.py", line 630, in <module>
_Main(sys.argv[1:])
File ".../.repo/repo/main.py", line 604, in _Main
result = run()
File ".../.repo/repo/main.py", line 597, in <lambda>
run = lambda: repo._Run(name, gopts, argv) or 0
File ".../.repo/repo/main.py", line 261, in _Run
result = cmd.Execute(copts, cargs)
File ".../.repo/repo/subcmds/selfupdate.py", line 54, in Execute
if not rp.Sync_NetworkHalf():
File ".../.repo/repo/project.py", line 1091, in Sync_NetworkHalf
if self.name in partial_clone_exclude:
TypeError: argument of type 'NoneType' is not iterable
$ ./run_tests -v
Change-Id: I71e744e4ef2a37b13aa9ba42eba3935e78c4e40a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304082
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The merge of the repo & init parser missed this default.
When running `repo init ...` in an existing checkout but w/out the -m
option, then repo would error out complaining that -m is required when
it didn't do this before.
Change-Id: I58035d48cc413b5d373702b9dc3b9ecd3fd1e900
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303945
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Make sure we print a message whenever we retry so it's clear to the
user why repo is pausing for a long time, and why repo might have
passed even though it displayed some errors earlier.
Also unify the sleep logic so we don't have two independent methods.
This makes it easier to reason about.
Also don't sleep if we're in the last iteration of the for loop. It
doesn't make sense to and needlessly slows things down when there are
real errors.
Bug: https://crbug.com/gerrit/12494
Change-Id: Ifceace5b2dde75c2dac39ea5388527dd37376336
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303402
Reviewed-by: Sam Saccone 🐐 <samccone@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
In _FetchOne & _CheckOne, only print error.GitError exception,
but other exceptions are still thrown
Fixes the GitError exceptions from /usr/lib/python3.8/multiprocessing/pool.py
exiting the repo sync.
Tested the code with the following commands and verified repo sync
continues after fetch error because of an invalid SHA1.
$ ./run_tests -v
$ python3 ~/work/repo/git-repo/repo sync -m manifest_P21623846.xml -j32
...
error.GitError: Cannot fetch platform/vendor/google_devices/redbull/proprietary update-ref: fatal: d5a99e518f09d6abb0c0dfa899594e1ea6232459^0: not a valid SHA1
....
An error like the following when jobs=1
error.GitError: Cannot checkout platform/vendor/qcom/sdm845/proprietary/qcrilOemHook: Cannot initialize work tree for platform/vendor/qcom/sdm845/proprietary/qcrilOemHook
Bug: https://crbug.com/gerrit/14392
Change-Id: I8922ad6c07c733125419f5698b0f7e32d70c7905
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303544
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Now that we have a bunch of subcommands doing parallel execution, a
common pattern arises that we can factor out for most of them. We
leave forall alone as it's a bit too complicated atm to cut over.
Change-Id: I3617a4f7c66142bcd1ab030cb4cca698a65010ac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301942
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Since Python has deprecated the formatter module, switch to the textwrap
module instead for reflowing text. We weren't really using any other
feature anyways.
Verified by diffing the output before & after the change and making sure
it was the same.
Then made a few tweaks to tighten up the output.
Change-Id: I0be1bc2a6661a311b1a4693c80d0f8366320ba55
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303282
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Currently, list only shows projects that exist in the checkout, and
doesn't offer any way to list all projects in the manifest (based on
the current settings, or on the options passed to list). This seems
to be the opposite of what (at least some) users expect, so let's
add an option to show all of them regardless of checkout state.
Change-Id: I94bbdc5bd0ff2a411704fa215e7fc2b60fa3360e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301263
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We want progress bars in the default output mode, but not when the
user specifies --quiet. Add a setting to the Progress bar class so
it takes care of not displaying anything itself rather than having
to update every subcommand to conditionally setup & call the object.
Change-Id: I1134993bffc5437bc22e26be11a512125f10597f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303225
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Add new CommonOptions entry points to move the existing --jobs to,
and relocate all --verbose/--quiet options to that. This provides
both a consistent interface for users as well as for code.
Change-Id: Ifaf83b88872421f4749b073c472b4a67ca6c0437
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303224
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
partial-clone-exclude option excludes projects during
partial clone. This is a comma-delimited project names
(from manifest.xml). This option is persisted and it
is used by the sync command.
A project that has been unparital'ed will remain unpartial if
that project's name is specified in the --partial-clone-exclude
option. The project name should match exactly.
Added
$ ./run_tests -v
Bug: [google internal] b/175712967
"I can't "unpartial" my androidx-main checkout"
$ rm -rf androidx-main/
$ mkdir androidx-main/
$ cd androidx-main/
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml
$ repo_dev sync -c -j8
+ Verify a project is partial
$ cd frameworks/support/
$ git config -l | grep 'partial'
+ Unpartial a project.
$ /google/bin/releases/android/git_repack/git_unpartial
+ Verify project is unpartial
$ git config -l | grep 'partial'
$ cd ../..
+ Exclude the project from being unparial'ed after init and sync.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml
+ Verify project is unpartial
$ cd frameworks/support/
$ git config -l | grep 'partial'
$ cd ../..
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep 'partial'
$ cd ../..
+ Remove the project from exclude list and verify that project is partially cloned.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep 'partial'
Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The number of jobs one wants to run against the network tends to
factor differently from the number of jobs one wants to run when
checking out local projects. The former is constrained by your
internet connection & server limits while the later is constrained
by your local computer's CPU & storage I/O. People with beefier
computers probably want to keep the network/server jobs bounded a
bit lower than the local/checkout jobs.
Change-Id: Ia27ab682c62c09d244a8a1427b1c65acf0116c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302804
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The current logic has a downside in that it doesn't sync to the latest
signed version available if the latest commit itself is unsigned. This
can come up when using the "main" branch as it is sometimes signed, but
often not as it's holding the latest merged commits. When people use
the main branch, it's to get early testing on versions tagged but not
yet released, and we don't want them to get stuck indefinitely on that
old version of repo.
For example, this series of events:
* "stable" is at v2.12.
* "main" is tagged with v2.13.
* early testers use --repo-rev main to get v2.13.
* new commits are merged to "main".
* "main" is tagged with v2.14.
* new commits are merged to "main".
* devs who had synced in the past to test v2.13 are stuck on v2.13.
repo sees "main" is unsigned and so doesn't try to upgrade at all.
The only way to get unwedged is to re-run `repo init --repo-rev main`,
or to manually sync once with repo verification disabled, or for us to
leave "main" signed for a while and hope devs will sync in that window.
The new logic is that whenever changes are available, we switch to the
latest signed tag. We also replace some of the duplicated verification
code in the sync command with the newer wrapper logic. This handles a
couple of important scenarios inaddition to above:
* rollback (e.g. v2.13.8 -> v2.13.7)
* do not trash uncommitted changes (in case of ad-hoc testing)
* switch tag histories (e.g. v2.13.8 -> v2.13.8-cr1)
Change-Id: I5b45ba1dd26a7c582700ee3711f303dc7538579b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300122
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We need to pass back an int, not a CompletedProcess object. Switch to
check=False so we don't throw an exception on failure -- we're already
showing pytest's stderr, and will return the non-zero status.
Change-Id: Ib0d3862a09a3963f25025f39a8e34419cf2a54df
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299624
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This avoids GIL limitations with using threads for parallel processing.
This reworks the fetch logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.
We have to tweak the chunking logic a little here because multiprocessing
favors batching over returning immediate results when using a larger value
for chunksize. When a single job can be quite slow, this tradeoff is not
good UX.
Bug: https://crbug.com/gerrit/12389
Change-Id: I0f0512d15ad7332d1eb28aff52c29d378acc9e1d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298642
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~40sec to ~16sec with the default -j8.
The output processing does not appear to be a significant bottle
neck -- it accounts for <1sec out of the ~16sec runtime. Thus we
leave it in the main thread to simplify the code.
Change-Id: I750b72c7711b0c5d26e65d480738fbaac3a69971
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297984
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~10sec to ~4sec with the default -j8.
This only does a simple conversion over to get an easy speedup. It
is currently written to collect all results before displaying them.
If we refactored this module more, we could have it display results
as they came in.
Change-Id: I5caf4ca51df0b7f078f0db104ae5232268482c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298643
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v
Bug: [google internal] b/183232698
Bug: https://crbug.com/gerrit/13707
$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync --use-superproject
$ repo_dev sync
real 0m8.046s
user 0m2.866s
sys 0m2.457s
Second time repo sync took only 8 seconds and verified by printing that
urrent_branch_only is True in project.py's Sync_NetworkHalf function.
Change-Id: Ic48efb23ea427dfa36e12a5c49973d6ae776d818
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301182
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v
Without --depth
$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject
real 6m48.086s
user 3m27.281s
sys 1m1.386s
With --depth=1
$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject
real 2m49.637s
user 2m51.458s
sys 0m39.108s
From dwillemsen@:
"For me it's the difference between 9m28s using the current code and
16s using --depth=1 while fetching the superproject."
Bug: [google internal] b/180451672
Bug: https://crbug.com/gerrit/13707
Change-Id: I1c3e4aef4414c4e9dd259fb6e4619da0421896b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300922
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Add a new "event": "command", which is emitted at when all command
arguments have been processed.
Additional fields:
"name": Name of the primary command (ex: repo, git)
"subcommands"': List of the sub-commands once command-line arguments
are processed
Examples:
Command: repo --version
Event: {"event": "command", <common fields>,
"name": "repo",
"subcommands": ["version"]
}
Bug: [google internal] b/178507266
Testing:
- Unit tests
- Verified repo git trace2 logs had expected data
Change-Id: I825bd0ecedee45135382461a4ba10f987f09aef3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300343
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The refactor to multiprocessing broke status reporting slightly when
checking out projects. Make sure we mark the step as failed if any
of the projects failed, not just when --fail-fast is set.
Change-Id: I0efb56ce83b068b2c334046df3fef23d797599c9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299882
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We've had a limited version of this in CrOS for a long time. There's
nothing CrOS specific about it, so lets move it to the repo project so
everyone can utilize it.
Change-Id: I04cd94610c1100f3afcd2baf8c8e7ab13e589490
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299202
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the current project doesn't have any local branches, then there's
nothing to prune, so return right away. This avoids running a few
git commands when we aren't actually going to use the results, and
it avoids checking repository validity. Since we aren't going to do
anything in here, no need to check it.
Change-Id: Ie9d5c75a954e42807477299f3e5a63a92fac138b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299742
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Added the following methods to XmlManifest class.
+ GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower()
+ GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func.
+ Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr.
+ Used the above methods to get groups in command.py::GetProjects
and part of init.py.
TODO: clean up these funcs to take structured group data more instead
of passing strings around everywhere that need parsing.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL
and verified prebuilts/fullsdk-linux directory has all the folders.
Tested repo init and repo sync with --use-superproject and without
--use-superproject argument.
$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync -c -j32
Bug: [google internal] b/181804931
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Superproject objects accept the optional argument “quiet”.
The following progress messages are displayed if quiet is false.
Displayed the following message whenever we find we have to make a new
folder (aka new remote), because if you started with repo init android
and later do googleplex-android that is when it will be slow.
"<location>: Performing initial setup for superproject; this might take
several minutes.".
After fetch completion, added the following notification:
"<location>: Initial setup for superproject completed."
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
Bug: [google internal] b/181178282
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Allow the user to specify relative or absolute or any other funky
path that they want when using `repo init` or `repo sync`. Our
goal is to restrict the paths in the remote manifest git repo we
cloned from the network, not protect the user from themselves.
Bug: https://crbug.com/gerrit/14156
Change-Id: I1ccfb2a6bd1dce2bd765e261bef0bbf0f8a9beb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298823
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
With the recent commit 0501b29e7a
("status: Use multiprocessing for `repo status -j<num>` instead of
threading"), the limitation with project serialization no longer
applies. It turns out that ad-hoc logic is expensive. In the CrOS
checkout (~1000 projects w/8 jobs by default), it adds about ~7sec
overhead to all invocations. With a fast nop run:
time repo forall -j8 -c true
This goes from ~11sec to ~4sec -- more than 50% speedup.
Change-Id: Ie6bcccd21eef20440692751b7ebd36c890d5bbcc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298724
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The ProjectArgs function can be inlined which simplifies it quite a
bit. We shouldn't need the custom exception handling here either.
This also makes the next commit easier to review.
Change-Id: If3be04f58c302c36a0f20b99de0f67e78beac141
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298723
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This is in preparation for simplifying the jobs support. The nested
function is referenced in the options object which can't be pickled,
so pull it out into a static method instead.
Change-Id: I01d3c4eaabcb8b8775ddf22312a6e142c84cb77d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298722
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If a manifest checksout a project multiple times, repo download isn't
able to accurately pick the right project. We were just picking the
first result which could be a bit random for the user. If we hit that
situation, check if the cwd is one of the projects, and if it isn't,
we emit an error and tell the user it's an ambiguous request.
Bug: https://crbug.com/gerrit/13070
Change-Id: Id1059b81330229126b48c7312569b37504808383
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298702
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The clone bundle logic assumes there is a one-to-one mapping between the
projects/ and project-objects/ trees. When using shared projects (where
we checkout different branches from the same project), this would lead us
to fetching the same clone bundle multiple times. Automatically skip the
clone bundle logic if the project-objects/ dir already exists.
Bug: https://crbug.com/gerrit/10993
Change-Id: I82c6fa1faf8605fd56c104fcea2a43dd4eecbce4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
On macOS, the Finder app likes to poop .DS_Store files in every path
that the user browses. If the user pokes around the .git/ tree, it
could generate a .DS_Store file in there too. When repo goes to read
all the local refs, it tries to decode this binary file as UTF-8 and
subsequently crashes.
Since paths that begin with . are not valid refs, ignore them like we
already do with paths that end in .lock. Also bump the check up to
ignore dirs that match since that follows the git rules: they apply
to any component in its path, not just the final path (name).
We don't implement the full valid ref algorithm that git employs as
it's a bit complicated, and we only really need to focus on what will
practically show up locally.
Bug: https://crbug.com/gerrit/14162
Change-Id: I6519f990e33cc58a72fcb00c0f983ad3285ace3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298662
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
When using Git under Windows, it seems that Git doesn't always parse
GIT_DIR correctly when it uses the Windows \ form, but does when it
uses / only.
For example, when using worktrees:
$ GIT_DIR='C:\Users\vapier\Desktop\repo\breakpad\tools\test\.git' git worktree list
fatal: not a git repository: ..\..\.repo\worktrees\linux-syscall-support.git\worktrees\test
$ GIT_DIR='C:/Users/vapier/Desktop/repo/breakpad/tools/test/.git' git worktree list
C:/Users/vapier/Desktop/repo/breakpad/.repo/worktrees/linux-syscall-support.git fd00dbbd0c06 (detached HEAD)
..\..\..\..\..\src\src\third_party\lss\.git fd00dbbd0c06 (detached HEAD)
..\..\..\..\..\tools\test\.git fd00dbbd0c06 (detached HEAD)
Change-Id: I666c03ae845ecb55d7f9800731ea6987d3e7f401
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298622
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This function is currently written with copyfile & linkfile in mind.
Generalize the logic & function arguments slightly so we can reuse
in more places that make sense.
This changes the validation logic slightly too in that we no longer
allow "." for the dest attribute with copyfile & linkfile, nor for
the src attribute with copyfile. We already rejected those later on
when checking against the active filesystem, but now we reject them
a little sooner when parsing.
The empty path check isn't a new requirement exactly -- repo used to
crash on it, so it was effectively blocked, but now we diagnosis it.
Bug: https://crbug.com/gerrit/14156
Change-Id: I0fdb42a3da60ed149ff1997c5dd4b85da70eec3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can speed things up. Across 1000 repos, it goes from
~9sec to ~5sec with the default -j8.
Change-Id: Ida6dd565db78ff7bac0ecb25d2805e8a1bf78048
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297982
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This avoids GIL limitations with using threads for parallel processing.
In a CrOS checkout with ~1000 repos, the nop case goes from ~6 sec down
to ~4 sec with -j8. Not a big deal, but shows that this actually works
to speed things up unlike the threading model.
This reworks the checkout logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.
Bug: https://crbug.com/gerrit/12389
Change-Id: I143e5e3f7158e83ea67e2d14e5552153a874248a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298063
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~30sec to ~3sec with the default -j8.
Change-Id: I0dc62d704c022dd02cac0bd67fe79224f4e34095
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297484
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
+ superproject will be fetched into a directory with the name
“<remote name>-superproject.git” instead of the current
“superproject.git” folder.
+ Deleted _Clone method and added _Init method.
+ _Init method will do “git init --bare <remote>-superproject.git”.
It will create the folder and set up a bare repository in
<remote>-superproject.git folder.
+ _Fetch method, will pass <remote url>, <branch> arguments.
Moved the --filter argument from “git clone” to “git fetch”.
_Fetch method will execute the following command to fetch
superproject. Added --no-tags argument.
master: git fetch <remote url> --force --no-tags --filter blob:none
branch: git fetch <remote url> --force --no-tags --filter blob:none \
<branch>:<branch>
+ Performance improvements for aosp-master
++ repo init performance improved from 35 seconds to 17 seconds.
++ repo init --use-superproject is around 5 to 7 secsonds slower.
++ repo sync --use-superproject is around 3 to 4 minutes faster.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real 0m20.648s
user 0m8.046s
sys 0m3.271s
+ Without superproject
$ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main
real 0m13.078s
user 0m9.783s
sys 0m2.528s
$ time repo_dev sync -c -j32 --use-superproject
...
real 15m7.072s
user 110m7.216s
sys 20m17.559s
+ Without superproject
$ time repo sync -c -j32
...
real 19m25.644s
user 91m56.331s
sys 20m59.170s
Bug: [google internal] b/180492484
Bug: [google internal] b/179470886
Bug: [google internal] b/180124069
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
This is in preparation for adding jobs support. The nested function
is referenced in the options object which can't be pickled, so pull
it out into a static method instead.
Change-Id: I280ed2bf26390a0203925517a0d17c13053becaa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297983
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The idea for skipping some progress updates was to avoid spending
too much time on the progress bar itself. Unfortunately, for large
projects (100s if not 1000s) of repos, we get into the situation
with large/slow checkouts that we skip showing updates when a repo
finishes, but not enough repos finished to increase the percent.
Since the progress bar should be relatively fast compared to the
actual network & local dick operations, have it show an update
whenever the caller requests it. A test with ~1000 repos shows
that the progress bar in total adds <100ms.
Bug: https://crbug.com/gerrit/11293
Change-Id: I708a0c4bd923c59c7691a5b48ae33eb6fca4cd14
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297903
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When people switch to non-default branches, they sometimes want to
switch back to the default, but don't know the exact name for that
branch. Add a -b HEAD shortcut for that.
Change-Id: I090230da25f9f5a169608115d483f660f555624f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297843
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The XmlManifestTests class is getting to be large and we're only
adding more to it. Factor out the core logic into a new TestCase
so we can reuse it to better group more tests.
Change-Id: I5113444a4649a70ecfa8d83d3305959a953693f7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298222
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Now that we've converted the few users of this over to subprocess APIs,
we don't need this anymore. It's been a bit hairy to maintain across
different operating systems, so there's no desire to bring it back.
Using multiprocessing Pool to batch things has been working better in
general anyways.
Change-Id: I10769e96f60ecf27a80d8cc2aa0d1b199085252e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Now that these code paths are all synchronous, there's no need to run
our own poll loop to read & pass thru/save output. Delete all of that
and just let the subprocess module take care of it all.
Change-Id: Ic27fe71b6f964905cf280ce2b183bb7ee46f4a0d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297422
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This fixes intermingling of parallel jobs and simplifies the code
by switching to subprocess.run. This also provides stable output
in the order of projects by returning the output as a string that
the main loop outputs.
This drops support for interactive commands, but it's unclear if
anyone was relying on that, and the default behavior (-j2) made
that unreliable. If it turns out someone still wants this, we can
look at readding it.
Change-Id: I7555b4e7a15aad336667292614f730fb7a90bd26
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297482
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The status command runs a bunch of jobs in parallel, and each one
is responsible for writing to stdout directly. When running many
noisy jobs in parallel, output can get intermingled. Pass down a
StringIO buffer for writing to so we can return the entire output
as a string so the main job can handle displaying it. This fixes
interleaved output as well as making the output stable: we always
display results in the same project order now. By switching from
map to imap, this ends up not really adding any overhead.
Bug: https://crbug.com/gerrit/12231
Change-Id: Ic18b07c8074c046ff36e306eb8d392fb34fb6eca
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297242
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Extend the Command class to support adding the --jobs option to the
parser if the command declares it supports running in parallel. Also
pull the default value used for the number of local jobs into the
command module so local commands can share it.
Change-Id: I22b0f8d2cf69875013cec657b8e6c4385549ccac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297024
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Every use of GitCommand in the tree just calls Wait as soon as it's
instantiated. Move the bulk of the logic into the init path to make
the call synchronous to simplify. We'll cleanup the users of the
Wait API to follup commits -- having this split makes it easier to
track down regressions.
Change-Id: I1e8c519efa912da723749ff7663558c04c1f491c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297244
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Since the --manifest-url flag is always required when creating a new
checkout, allow the url to be specified via a positional argument.
This brings it a little closer to the `git clone` UI.
Change-Id: Iaf18e794ae2fa38b20579243d067205cae5fae2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297322
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
This is the only code in the tree that uses GitCommand asynchronously.
Rewrite it to use multiprocessing.Pool as it makes the code a little
bit easier to understand and simpler.
Change-Id: I3ed3b037f24aa1e9dfe8eec9ec21815cdda7678a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297143
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
These are the only users in the tree that process the output as it's
produced. All others capture all the output first and then process
the results. However, these functions still don't fully return until
it's finished processing, and these funcs are in turn used in other
synchronous code paths. So it's unclear whether anyone will notice
that it's slightly slower or less interactive. Let's try it out and
see if users report issues.
This will allow us to simplify our custom GitCommand code and move it
over to Python's subprocess.run, and will help fix interleaved output
when running multiple commands in parallel (e.g. `repo diff -j8`).
Change-Id: Ida16fafc47119d30a629a8783babeba890515de0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297144
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We only provide input to GitCommand in one place, so inline the logic
to be more synchronous and similar to subprocess.run. This makes the
code simpler and easier to understand.
Change-Id: Ibe498fedf608774bae1f807fc301eb67841c468b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297142
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Added --no-use-superproject to repo and init.py to disable use of
manifest superprojects.
Replaced the term "sha" with "commit id".
Added _GetBranch method to Superproject object.
Moved shared code between init and sync into SyncSuperproject function.
This function either does git clone or git fetch. If git fetch fails
it does git clone.
Changed Superproject constructor to accept manifest, repodir and branch
to avoid passing them to multiple functions as argument.
Changed functions that were raising exceptions to return either True
or False.
Saved the --use-superproject option in config as repo.superproject.
Updated internal-fs-layout.md document.
Updated the tests to work with the new API changes in Superproject.
Performance for the first time sync has improved from 20 minutes to
around 15 minutes.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo init took around 20 seconds longer because of cloning of superproject.
$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real 0m35.919s
user 0m21.947s
sys 0m8.977s
First run
$ time repo sync --use-superproject
...
real 16m41.982s
user 100m6.916s
sys 19m18.753s
No difference in repo sync time after the first run.
Bug: [google internal] b/179090734
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Changed "git pull" to "git fetch" as we are using --bare option. Used the
following command to fetch:
git fetch origin +refs/heads/*:refs/heads/* --prune
Pass --branch argument to Superproject's UpdateProjectsRevisionId function.
Returned False/None when directories don't exist instead of raise
GitError exception from _Fetch and _LsTree functions. The caller of Fetch
does Clone if Fetch fails.
Tested the code with the following commands.
$ ./run_tests -v
Tested the init and sync code by copying all the repo changes into my Android
AOSP checkout and running repo sync with --use-superproject option.
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222
Reviewed-by: Mike Frysinger <vapier@google.com>
After updating all project’s revsionIds with the SHAs from superproject,
write the updated manifest into superproject_override.xml file. Reload
that file for future Reloads. This file is created in exp-superproject
directory.
Moved most of the code that is superproject specific into
git_superproject.py and wrote test code.
If git pull fails, did a git clone of the superproject.
We saw performance gains for consecutive repo sync's. The time to sync
went down from around 120 secs to 40 secs when repo sync is executed
consecutively.
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
First run
$ time repo sync --use-superproject
...
real 21m3.745s
user 97m59.380s
sys 19m11.286s
After two consecutive sync runs
$ time repo sync -c -j8 --use-superproject
real 0m39.626s
user 0m29.937s
sys 0m38.155s
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
_CheckForImmutableRevision is used to see if repo can
skip fetching a project, but 'git rev-parse' with partial
clone does a data fetch to accomplish this.
Changed to use: 'git rev-list -1 --missing=allow-any <SHA>^0' which
checks the local ref without fetching from the server first.
Bug: [google internal] b/179477822
Testing:
- Unit tests
- Verified init/sync working on aosp-master
- Verified wwith a pinned manifest that local ref check works (no fetch)
Change-Id: If327b893c6658421f41df1f58c337f53b4c60ce6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296142
Reviewed-by: Dan Willemsen <dwillemsen@google.com>
Tested-by: Ian Kasprzak <iankaz@google.com>
+ This is without --depth option. This is done for reachability.
Server doesn't know what you know about in the history so it always
sends you the whole thing Which is very slow.
If we have the full history it can send you incremental update history
which is very small and fast.
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
.../WORKING_DIRECTORY$ repo sync --use-superproject
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I239de6d8f1c2ed6b4c69e7a78b8aa95338fa838c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/295362
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
.../WORKING_DIRECTORY$ repo sync --use-superproject
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I7e4b1e51ca1d18b836a5fa8d139a0765262ba500
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
.../WORKING_DIRECTORY$ repo sync --use-superproject
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Ieea31445ca89ba1d217e779ec7a7a2ebe81ac518
repo diffmanifests saves git commit messages in buf and uses default
utf-8 decoding, in some scenarios git commit message can itself contain
a non UTF-8 character due to a typo or incorrect i18n.commitEncoding.
e.g.
d354d9afe923 [PATCH] fbcon: don\xb4t call set_par() in fbcon_init() if vc_mode == KD_GRAPHICS
Convert the buf containing git commits to string if decoding to utf-8
encounters an error.
Signed-off-by: Gaurav Pathak <gaurav.pathak@pantacor.com>
Change-Id: If818562f0faaa5062c765fbea11dc0e1c86a24d7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294742
Reviewed-by: Mike Frysinger <vapier@google.com>
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.
Tested the code with the following commands.
$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,
.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []
Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
Currently we don't have a way for the checked out repo version to
declare the version of tools it needs before we start running it.
For somethings, like git, it's not a big deal as it can handle all
the asserts itself. But for things like Python, it's impossible
to reliably check before executing.
We're in this state now:
- we've been allowing Python 3.4, so the launcher accepts it
- the repo codebase starts using Python 3.6 features
- launcher tries to import us but hits syntax errors
- user is left confused and assuming new repo is broken because
they're seeing syntax errors
This scenario is playing out with old launchers that still accept
Python 2, and will continue to play out as time goes on and we want
to require newer versions of Python 3.
Lets create a JSON file to declare all these system requirements.
That file format is extremely stable, so loading & parsing from
even ancient versions of Python shouldn't be a problem. Then the
launcher can read these settings and check the system state before
attempting to execute any code. If the tools are too old, it can
clearly diagnose & display information to the user as to the real
problem (and not emit tracebacks or syntax errors).
We have a couple of different tool version checks already (git,
python, ssh) and can harmonize them in a single place.
This also allows us to assert a reverse dependency if the need
ever comes up: force the user to upgrade their `repo` launcher
before we'll let them run us. Even though the launcher warns
whenever a newer release is available, some users seem to ignore
that, or they don't use repo that often (on the scale of years),
and their upgrade jump is so dramatic that they fall back into
the syntax error pit.
Hopefully by the end of the year we can assume enough people
have upgraded their launcher such that we can delete all of the
duplicate version checks in the codebase. But until then, we'll
keep them to maintain coverage.
Change-Id: I5c12bbffdfd0a8ce978f39aa7f4674026fe9f4f8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293003
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
At most one superproject may be specified. It will be used
to specify the URL of superproject.
It would have 3 attributes: remote, name, and default.
Only "name" is required while the others have reasonable defaults.
<remote name="superproject-url" review="<url>" />
<superproject remote="superproject-url" name="platform/superproject"/>
TODO: This CL only implements the parsing logic and further work
will be in followup CLs.
Tested the code with the following commands.
$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v
Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I5b4bba02c8b59601c754cf6b5e4d07a1e16ce167
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292982
Reviewed-by: Mike Frysinger <vapier@google.com>
We've been warning about Python 3.4 for almost a year. This drops
support for these systems:
* Ubuntu Trusty: released Apr 2014, EOL Apr 2022
* Debian Jessie: released Apr 2015, EOL Jun 2020
So the min required distros would now be:
* Ubuntu Xenial: released Sep 2015 w/Python 3.5
* Debian Stretch: released Jun 2017 w/Python 3.6
I don't think we're quite ready to drop Python 3.5 which would affect
Ubuntu Xenial -- we'd have to update to Ubuntu Bionic from Apr 2018.
Let's see how much the community reacts to loss of Python 3.4 first.
Bug: https://crbug.com/gerrit/10418
Change-Id: Ib24a57818fdca49e23db53e1bdd1f4c76b4963f7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291502
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The codebase still supports Python 3.5, so allow use of that instead
of requiring Python 3.6+. Supporting this mode well is a bit tricky
as we want to first scan for newer versions before falling back to
older ones. And we have to avoid infinite loops in the process.
Bug: https://crbug.com/gerrit/13795
Change-Id: I47949a173899bfa9ab20d3fefa1a97bf002659f6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When checkout is done with Git worktrees then the HEAD in the
bare-git repositories point to the initialized default (e.g.
'refs/heads/master'). This default branch does not exist
locally and is not automatically created.
When a user now creates a branch in any git repository named
'master' then it is no longer possible to get rid of this branch,
neither is it possible to switch to another branch and switch
back to this master branch. Git concludes the 'master' branch is
already checked out (in the bare Git) and that results in a
lockdown of this master branch.
To repoduce this issue, run these commands in a repo tree
checked out with --worktree:
- git checkout master # assuming the remote repo has a master branch,
# a local tracking branch master is created here
- git checkout -b temp
- git checkout master # This one now fails
- git branch -d master # fails too
The failure is caused by Git assuming the master branch is checked out
by the bare git repository since HEAD is pointing towards it.
To workaround this, we always detach HEAD in the bare-git when
syncing. We don't need it to point to a ref in general, but we
would like it to be valid so git tools "just work" if they're run
in here.
Signed-off-by: Remy Bohmer <oss@bohmer.net>
Change-Id: I15c96604363c41f0d01c42f533174393097daeb5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290985
Reviewed-by: Mike Frysinger <vapier@google.com>
Spread the operation of querying which local branches exist across a
pool of processes and build the name map of projects -> branches as
these tasks finish rather than blocking on the entire query. The search
operations are submitted in batches to reduce the overhead of interprocess
communication. The `chunksize` argument used to control this batch size
was selected by incrementing through powers of two until it stopped being
faster.
Change-Id: Ie3d7f799ee8e83e5058536caf53e2979175408b7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291342
Tested-by: Chris Mcdonald <cjmcdonald@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Rather than pull the client dir out to construct the manifest
filename which the manifest itself already has, pull the filename
out and use that.
Change-Id: I33991084dcb3205f819bb841084e3c48d6ccb284
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291264
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When updating the tracking ref to whatever the user requested,
make sure we reset state completely rather than trying to update
the ref to it. This avoids confusing git as to the current state
of the tree, and is more inline with user intentions: if they made
a local change to the checkout, but ran repo init with a specific
rev, we shouldn't stay wedged forever until they manually clean it
all up.
Bug: https://crbug.com/gerrit/12801
Change-Id: Ieba8d9c15781b4d0649bf01c7460694da63387b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290923
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We allow project.groups to be whitespace or comma delimited, but
repo-hooks.enabled-list is only whitespace delimited. This hasn't
been a big deal as it's only ever had one valid value, but if we
want to add more, we should harmonize these a bit.
Refactor the groups method to be more generic, and run the enabled-
list attribute through it. Then add missing docs for it.
Change-Id: Iaa96a0faa9c4a68b313b49336751831b73bf855d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290743
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Tests worked fine if init.defaultBranch main was used,
but failed due to git branch reasons if master was still used.
Since we can only use init.defaultBranch if git version >= 2.28,
I also went with a template dir HEAD main tweak if lower so tests
now pass regardless of client git default branch and version.
Test: Ran tests with ~/.gitconfig:init.defaultBranch=master
Test: Ran tests with ~/.gitconfig:init.defaultBranch=main
Test: Ran tests for both code branches of git require
Change-Id: I49fa1e4ae45b8aec16a093132ee9fa466cbc11ec
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290404
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Attrib groups can now be added to manifest include, thus
all projects in an included manifest file can easily be tagged
with a group without modifying all projects in that manifest file.
Include groups will add and recurse, meaning included manifest
projects will carry all parent includes. Intentionally, no support
added for group remove, to keep complexity down.
Group handling for projects is untouched, meaning a group set on
a project will still append to whatever was or was not inherited
in parent manifest includes, resulting in union of groups inherited
and set for the project itself.
Test: manual multi-level manifest include structure, in serial and parallel,
with different groups set on init
Test: added unit tests to cover the inheritance
Change-Id: Id2229aa6fd78d355ba598cc15c701b2ee71e5c6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/283587
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
The knowledge about running hooks and all its exception handling
is scattered over multiple files. This makes the code harder
to read, but also it requires duplication of logic in case
other RepoHooks are added to different commands.
This refactoring also creates uniform behavior of the hooks
across multiple commands and it guarantees the re-use of the same
arguments on all of them.
Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: Ia4d90eab429e4af00943306e89faec8db35ba29d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/277562
Tested-by: Remy Bohmer <oss@bohmer.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
When intializing a new repo with the --reference option on Windows 10
the objects/info/alternates in each git repository is created with
Windows line endings (\r\n), leading to the following error:
error: object directory C:/<PATH_TO_MIRROR>/<REPO_NAME>.git/objects?
does not exist; check .git/objects/info/alternates
This can be fixed by simply using unix line endings on both
Windows and unix platforms.
Reported-by: Francisco Javier Alvarez Garcia <javier.alvarez.garcia.17@gmail.com>
Follow-up-from: I268fe029ede68802c21037b0f2ae8a95afb85e48
Bug: https://crbug.com/gerrit/13208
Change-Id: I6da60c4ca957778b3c42ab6b9ad85c40483f0042
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289431
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <oss@bohmer.net>
Worktree .git and gitdir reference files are written by Git with
Unix line ending, even on Windows & macOS. The conversion to
relative paths makes these files end with DOS line endings in
Windows. The Git integration in Visual Studio 2019 cannot deal
with these DOS line endings and considers these worktrees invalid.
Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: I088cfd994f3cc31db4e0ca7791fa0a4ee3ac222f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289310
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <linux@bohmer.net>
We conflate the manifest & parsing logic with the management of the
repo client checkout in a single class. This makes testing just one
part (the manifest parsing) hard as it requires a full checkout too.
Start splitting the two apart into separate classes to make it easy
to reason about & test.
Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Using sha1 manifest, project's revisionId is initialized
first by the manifest.
An update of a projet revision by extend-project node does
not apply to the revisionId which is therefore kept to the
initial value.
Resets revisionId value when revision is updated by an
extend-project node.
Change-Id: I873af283890cebaeaabde966f04b125642af929f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/275715
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Miguel Gaio <miguel.gaio@renault.com>
In the .repo discovery loop
while curdir != '/' and curdir != olddir:
... break if we found .repo ...
olddir = curdir
curdir = os.path.dirname(curdir)
the "while" condition is meant to avoid searching forever if we do not
find .repo before reaching the top-level directory of the filesystem.
For that purpose, the first half of the condition is redundant; once
we reach "/", the parent directory will be "/" again and the curdir !=
olddir check would suffice to terminate the search. Simplify by
removing the redundant first half of the check.
Noticed by code inspection. The first half of the check was retained
when introducing the second half in df14a70c ("Make path references OS
independent", 2011-01-09), in an excess of caution.
This also improves consistency a little: if I start with curdir =
'/home/me', then with the redundant check in place we search
/home/me
/home
before hitting / and giving up. On Windows, if I start with
'c:/users/me', then we search
c:/users/me
c:/users
c:/
before hitting a repetition and giving up. Fortunately it is not
common for people to set up repo clients at the top level of
filesystems, but consistently following the latter behavior should
make debugging a little easier in case it comes up.
Link: https://gerrit-review.googlesource.com/id/Ib9e830e3b9adfb1c4e56f3bcfba4746c401fb84f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/286002
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jonathan Nieder <jrn@google.com>
Instead of hardcoding "master" as our default, use the remote server's
default branch instead. For most people, this should be the same as
"master" already. For projects moving to "main", it means we'll use
the new name automatically rather than forcing people to use -b main.
For repositories that never set up a default HEAD, we should still use
the historical "master" default.
Bug: https://crbug.com/gerrit/13339
Change-Id: I4117c81a760c9495f98dbb1111a3e6c127f45eba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280799
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This change increases the speed of the command with parallelization with
processes. The parallelization with threads doesn't work well, and
increasing the number of jobs to many (8 threads ~) didn't increase the speed.
Possibly, the global interpreter lock of Python affects.
Bug: https://crbug.com/gerrit/12389
Change-Id: Icbe5df8ba037dd91422b96f4e43708068d7be924
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279936
Tested-by: Kimiyuki Onaka <kimiyuki@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If you pass args to `repo init` when first creating a checkout, the
repo launcher throws an error. But the init subcommand that runs in
an existing checkout silently ignores them. Throw a proper error.
Change-Id: I433bfcc73902d25f6b6a2974e77f6a977a75ed16
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279696
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the manifest uses a trailing slash on the name attribute, repo
will construct bad internal filesystem paths which confuses tools
later on.
For example, this manifest entry:
<project name="aosp/platform/system/libhidl/" ...
will cause repo to use paths like:
.repo/project-objects/aosp/platform/system/libhidl/.git/
when it really should be using:
.repo/project-objects/aosp/platform/system/libhidl.git
Apply the normalization when we construct the local filesystem paths
as we cannot guarantee that the remote URL constructed from these
will behave the same. A server might really want:
https://example.com/aosp/platform/system/libhidl/
and would throw an error if we instead tried to fetch:
https://example.com/aosp/platform/system/libhidl
Unfortunately, any existing repo client checkouts that use such a
manifest will hit a one-time sync error as the internal git location
has changed. I'm not sure there's a way to cleanly migrate that.
Bug: https://crbug.com/1086043
Change-Id: I30bea0ffd23e478de89a035f408055e48a102658
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268742
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
For large projects, clone bundle is useful because it provided a way to
efficiently transfer a large portion of git objects through CDN, without
needing to interact with git server. However, with partial clones, the
intention is to not download most of the objects, so the use of clone
bundles would defeat the space savings normally seen with partial
clones, as they are downloaded before the first fetch.
A new option, --clone-bundle is added to override this behavior.
Add a new repo.clonebundle variable which remembers the choice if
explicitly given from command line at repo init.
Change-Id: I03638474af303a82af34579e16cd4700690b5f43
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268452
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
If the dest-branch attribute is set in the project manifest, then
we need to push to that branch. Previously, we would unconditionally
pre-pend the refs/heads prefix to it. The dest-branch attribute is
allowed to be a ref expression though, so it may already have it.
Simple fix is to check if it already has the prefix before adding it.
Bug: crbug.com/gerrit/12770
Change-Id: I45d6107ed6cf305cf223023b0ddad4278f7f4146
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268152
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
When running repo info -d an error would be thrown saying:
fatal: bad revision 'refs/remotes/m/refs/heads/master..'
Using the short branch name here instead, like 'refs/remotes/m/master..'
resolves this issue.
Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: I50ea92c45c011b2c3e3a63803decb88e7837a380
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/266578
Reviewed-by: Mike Frysinger <vapier@google.com>
The intention of the check is to verify whether the target
file name contains a wild card. The code, however, assumes
that if the file is non-existent - it contains a wild card.
This has the side effect that a target file that does not
exist at the moment of the check is considered to contain a
wild card, this leads itself to softlink not being created.
Change-Id: I4e4cd7b5e1b8ce2e4b2edc9abf5a1147cd86242f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/265736
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Angel Petkov <apetkov86@gmail.com>
Git config files may have an include tag pointing to another file.
The included file is not parsed unless “git config --list” is
explicitly told to follow includes by adding the argument ”--includes”.
This change add the "--includes" when parsing the global gitconfig file.
Change-Id: I892c9a3a748754c1eb8c9e220578305ca5850dd5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/264759
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Ulrik Laurén <ulrik.lauren@gmail.com>
exit if no repo_main can be found right before executing the command.
This happens for instance when 'repo init' is run on root path
(for example in a container). Without this counter measure the tool
will crash at exec_command with
TypeError: sequence item 1: expected str instance, NoneType found
Change-Id: Ia8480cfe2151c3b35c9572789ad8cb619288cce1
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263457
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
The launcher already raised itself up to use Python 3 on the fly, and
the main.py script uses a plain `python` shebang. So make sure we use
the active interpreter when re-execing ourselves to avoid falling back
down to Python 2 (which then triggers warnings).
Change-Id: Ic53c07dead3bc9233e4089a0a422f83bb5ac2f91
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263272
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
When generating a revision locked manifest, we need to know what
ref to push changes to when doing 'repo upload'. This information
is lost when we lock the revision attribute to a particular commit
hash, so we need to expose it through the dest-branch attribute.
Bug: https://crbug.com/1005103
Test: manual execution
Change-Id: Ib31fd77ad8c9379759c4181dac1ea97de43eec35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263572
Tested-by: Sean McAllister <smcallis@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Recent changes in ChromeOS Infra to ensure we're reading from
snapshot manifests properly have exposed several bugs in our
assumptions about manifest files. Mainly that the revision field
for a project does _not_ have to refer to a ref, it can just be
a commit hash.
Several places assume that the revision field can be parsed as a
ref to get the branch the project is on, which isn't true. To fix
this we need to be able to look at the upstream and dest-branch
attributes of the repo, so we expose them through the environment
variables set in `repo forall`.
Test: manual 'repo forall' run
Bug: https://crbug.com/1032441
Change-Id: I2c039e0f4b2e0f430602932e91b782edb6f9b1ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263132
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
The generated socket path can be too long, if your FQDN is very long...
Typical error message from ssh client:
unix_listener: path "/tmp/ssh-fqduawon/master-USER@HOST:PORT.qfCZ51OAZgTzVLbg" too long for Unix domain socket
Use a hashed version instead, to keep within the socket file path limit.
This requires OpenSSH_6.7p1, or later.
Change-Id: Ia4bb9ae8aac6c4ee31d5a458f917f3753f40001b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255632
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Anders Björklund <anders.bjorklund.2@volvocars.com>
The SetupGnuPG test tries to test the full setup, including the
creation of the directories. In order to do that, it create a
temporary directory, and redefines the home_dot_repo to point there.
When a home_dot_repo directory does not exist, it should be created.
The gpg_dir, which should exist inside home_dot_repo, also needs to be
created if it does not exist. However, since the gpg_dir path is
relative to home_dot_repo, once we redefine one, we need to redifine
the other.
The failure of this test might have gone unnoticed so far, since in
only fails if you do not have a ~/.repoconfig/gnupg/ on the
environment you are running the tests on.
Change-Id: Ic69d59e56137eea43349a61b5cf81f215c6a7f9a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/262573
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Marcos Marado <mindboosternoori@gmail.com>
In older versions of Gerrit the Change-Id field was inserted at the
start of the trailers. Commit 68296f71804feab2e0ae18ae33f834a8a41621e4
simplified the trailers code by using git trailers instead of custom
code but now inserts Change-Id at the end of the trailers section.
A consequence of this is that folks who sign-off their commits using
`git commit -s` now has the sign-off appear first followed by
Change-Id. If the user then runs `git commit -s --amend` to update
the change because the Sign-off-by line is not last, git inserts
a 2nd duplicate Signed-off-by line.
This patch simply restores the previous behaviour of the Gerrit
commit-msg hook where Change-Id would be inserted before the
Sign-off-by line to avoid this issue.
Backported from [1] by Thanh Ha.
[1] https://gerrit-review.googlesource.com/c/gerrit/+/262072
Bug: https://crbug.com/12546
Change-Id: I1406c763a3935761247f6771f55e02367f698e6e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/262352
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
When developing repo itself, it helps to run repo directly out of it
and to run bisection tools. The current _SetDefaultsTo logic fails
in that situation though as it wants a branch, but the source isn't
checked out to one. Now that we support tracking commits via the
--repo-rev setting, fall back to using the current HEAD commit.
Change-Id: I37d79fd9f7bea87d212421ebed6c8267ec95145f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260192
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We respect this option when running the first `repo init`, but then
silently ignore it once the initial sync is done. Make sure users
are able to change things on the fly.
We refactor the wrapper API to allow reuse between the two init's.
Bug: https://crbug.com/gerrit/11045
Change-Id: Icb89a8cddca32f39a760a6283152457810b2392d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260032
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
While the help/usage suggested that revisions would work, they never
actually did, and just throw confusing errors. Now that we warn if
the checkout isn't tracking a branch, allow people to specify commits
or tags explicitly. Hopefully our nags will be sufficient to keep
most people on the right path.
Bug: https://crbug.com/gerrit/11045
Change-Id: I6ea32c677912185f55ab20faaa23c6c0a4c483b3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259492
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We gracefully handle cherry-pick errors, but none of the others
which means people get confusing Python tracebacks. Move the
main logic in a single GitError try block so we can show pretty
error messages for all of them.
Change-Id: I52cdf6468d21a98de7f65b86d5267b3caabd5af8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259854
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The git cherry-pick already supports this, so plumb the existing repo
option down. Otherwise it's confusing when people use -c --ff and it
doesn't use that behavior.
Change-Id: Id68932ffa09204bb30b92a21aff185c00394a520
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259852
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When the launcher handles the init subcommand, it takes care of
setting the repo url & branch itself when cloning. So we don't
need to pass them down to the checked out init subcommand.
Further, the init subcommand has never actually done anything
with those options, so there's no point in passing them.
We'll be changing the latter behavior so that init will reset
the url/branch when specified with an existing repo checkout
which means passing them through adds overhead: the launcher
will checkout to the right value, then chain to the sub-init
which will then reset the checkout to the same value.
Bug: https://crbug.com/gerrit/11045
Change-Id: Ia2a4ab9d86febc470aea4abd73d75bb10e848b56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259312
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The current subcmds design has singletons in all_commands. This isn't
exactly unusual, but the fact that our main & help subcommand will then
attach members to the classes before invoking them is. This makes it
hard to keep track of what members a command has access to, and the two
code paths (main & help) attach different members depending on what APIs
they then invoke.
Lets pull this back a step by storing classes in all_commands and leave
the instantiation step to when they're used. This doesn't fully clean
up the confusion, but gets us closer.
Change-Id: I6a768ff97fe541e6f3228358dba04ed66c4b070a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259154
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The branch->branches alias is setup in the main module when that
really belongs in the existing all_commands setup.
For help, rather than monkey patching all_commands to the class,
switch it to use the state directly from the module. This makes
it a bit more obvious where it's coming from rather than this one
subcommand having a |commands| member added externally to it.
Change-Id: I0200def09bf4774cad8012af0f4ae60ea3089dc0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259153
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
When we download git sources, we get a progress bar (good) and we get
a dump of all the refs we downloaded (bad) as it can easily be 100+ if
not 1000+ depending on the project (for each git repo!). Lets rework
the output behavior so that:
* quiet: Only errors.
* default: Progress bars (if on a tty).
* verbose: Full output (progress bars & downloaded refs).
Bug: https://crbug.com/gerrit/11293
Change-Id: I87a380075e79de6805f91095876dd1b37d32873a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256456
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
We've been overly lenient with boolean parsing by ignoring invalid
values as "false" even if the user didn't intend that. Turn all
unknown values into warnings to avoid breaking existing manifests,
and unify the parsing logic in a helper to simplify.
We've been stricter about numbers, but still copying & pasting
inconsistent code. Add a helper for this too. For out of range
sync-j numbers (i.e. less than 1), throw a warning for now, but
mark it for future hard failures.
Change-Id: I924162b8036e6a5f1e31b6ebb24b6a26ed63712d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256457
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Since most ref namespaces are shared among all worktrees, trying to
set the pseudo m/<branch> in the common git repo ends up clobbering
each other when using shared checkouts. For example, in CrOS:
<project path="src/third_party/kernel/v3.8"
name="chromiumos/third_party/kernel"
revision="refs/heads/chromeos-3.8" />
<project path="src/third_party/kernel/v3.10"
name="chromiumos/third_party/kernel"
revision="refs/heads/chromeos-3.10" />
Trying to set m/master in chromiumos/third_party/kernel.git/ will
keep clobbering the other.
Instead, when using git worktrees, lets set the m/ pseudo ref to
point into the refs/worktree/ namespace which is unique to each
git worktree. So we have in the common dir:
chromiumos/third_party/kernel.git/:
refs/remotes/m/master:
ref: refs/worktree/m/master
And then in each worktree we point refs/worktree/m/master to the
respective manifest revision expression. Now people can use the
m/master in each git worktree and have it resolve to the right
commit for that worktree.
Bug: https://crbug.com/gerrit/12404
Change-Id: I78814bdd5dd67bb13218c4c6ccd64f8a15dd0a52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256952
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Trying to use the config state when the git tree hasn't yet been
created hits bad side effects. Add a check to avoid probing the
config logic during the first run. It's not clear what's going
wrong at the lower layers, but this gets us back to the behavior
before we added worktree support, so lets settle the status quo.
Bug: https://crbug.com/gerrit/12387
Change-Id: I85b56797455f3c2e249d02c18496e060be05501d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256592
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We produce some simple "Get" messages that aren't super clear as to
what they're doing, especially for people not familiar with repo.
Rephrase these to explicitly state the thing we're doing so it's
clear why we're downloading a particular source.
Bug: https://crbug.com/gerrit/11293
Change-Id: I0749504f17c5385c6c65274a274e0ae25b117413
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256455
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Git likes to create .git files with read-only permissions which makes
it difficult to open+truncate+write in situ under Windows. Delete it
before we write the file content to workaround.
Change-Id: I3effd96525f0dfe0b90e298b6bf0b856ea26aa03
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256412
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Changing this to a file instead of using a symlink serves two purposes:
* We can insert some comments & doc links to help users learn what this
is for, discover relevant documentation, and to discourage them from
modifying things.
* Windows requires Administrator access to use symlinks. With this
last change, Windows users can get repo client checkouts with the new
--worktree option and not need symlinks anywhere at all. Which means
they no longer need to be an Administrator in order to `repo sync`.
Change-Id: I9bc46824fd8d4b0f446ba84bd764994ca1e597e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256313
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Windows requires Administrator access to create symlinks. We can
mitigate this a bit by falling back to hardlinks as those may be
created by any user on the system. Do this with the git hooks as
these are supposed to be internal only and people shouldn't be
modifying them. If they do, they'll have to delink first. This
seems worth it to allow repo usage without extra privileges.
Change-Id: I996ea9c9238f7bd7d27d1d9b1f2786593bf75ef7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256312
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We also need to check more things in the manifest/project handlers,
and use platform_utils in a few places to address Windows behavior.
Drop Python 2.7 from Windows testing as it definitely doesn't work
and we won't be fixing it.
Change-Id: I83d00ee9f1612312bb3f7147cb9535fc61268245
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256113
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We're using this for git worktrees because it handles the .git file
format, but it should work for all flows. Unify to simplify. This
also fixes the worktree logic which duplicated .git/config settings.
Change-Id: Ie3af2e206710859dccfc376b3593f415d6830738
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256034
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Switch the copyright holder to "The Android Open Source Project" to
match all the other source files in the tree, and move it to the top
of the file to match everything else we do.
Change-Id: Ie15d8e2bc004a626e45f715271deeaf3919dc44a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256235
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
- E129 visually indented line with same indent as next logical line
- E125 continuation line with same indent as next logical line
Fixed automatically by:
autopep8 --in-place --select E125,E129 subcmds/sync.py
Change-Id: Ia2f82f443e1e6a23ba22c6f9849c8485405aed0e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256092
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
This allows people to write ~/.repoconfig/config akin to ~/.gitconfig
and .repo/config akin to .git/config. This allows us to add settings
specific to repo without mixing up git, and to persist in general.
Change-Id: I1c6fbe31e63fb8ce26aa85335349c6ae5b1712c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255832
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Since deleting a source checkout involves a good bit of internal
knowledge of .repo/, move the DeleteProject helper out of the sync
code and into the Project class itself. This allows us to add git
worktree support to it so we can unlock/unlink project checkouts.
Change-Id: If9af8bd4a9c7e29743827d8166bc3db81547ca50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256072
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When using extensions, make sure we set the git repo format version
so git knows to check the extension compatibility. We can add a
helper to the Project API to simplify this and make it foolproof.
Change-Id: I9ab6c32d92fe2b8e5df6e2b080ca71556332e909
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256035
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
This provides initial support for using git worktrees internally
instead of our own ad-hoc symlink tree. It's been lightly tested
which is why it's not currently exposed via --help.
When people opt-in to worktrees in an existing repo client checkout,
no projects are migrated. Instead, only new projects will use the
worktree method. This allows for limited testing/opting in without
having to completely blow things away or get a second checkout.
Bug: https://crbug.com/gerrit/11486
Change-Id: Ic3ff891b30940a6ba497b406b2a387e0a8517ed8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254075
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Different Python & OS versions have different environ behavior wrt
accepted types & encoding. Since we're migrating to be Python 3 only,
lets change our code to assume strings always work as that's what the
newer Python 3 does. This will fail under Python 2 for some env vars,
mostly on Windows, but the effort of maintaining shim layers that can
handle these edge cases isn't worth it when we're dropping that code.
We leave the logic in the `repo` launcher for now as it is simple, and
we want it to be able to switch versions a bit longer than the rest of
the tree.
Here's the support table:
| *NIX | Windows |
Python 2 | ASCII string | str or bytes, not unicode |
Python 3 | str or bytes | str only |
Windows uses strings natively in its environment all the time. But it
doesn't allow unicode strings under Python 2, so we have to encode.
Python 2 on *NIX is funky in that it always lowers to ASCII, so we had
to manually encode to avoid errors regardless of unicode or str.
Python 3 on Windows & *NIX will accept strings. *NIX will also accept
bytes but Windows will not.
Bug: https://crbug.com/gerrit/12145
Change-Id: I3cf8f95a06902754ea1f08ad4b28503f7063531b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248972
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
For people used to running `repo xxx --help`, they might not realize
that there are detailed man pages behind `repo help xxx`. Add a note
to all --help commands to improve discoverability.
Change-Id: I84af58aa0514cc7ead185f6c2534a8f88e09a236
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255853
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The pager setting is tristate (where None means "auto"),
so make sure we still handle that setting.
Change-Id: I89fe352572dd15922c61e3bb65ac33f847d01ee0
Test: `repo help upload` triggers the pager
Test: `repo -p help upload` triggers the pager
Test: `repo --no-pager help upload` doesn't trigger the pager
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255852
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Named Temporary file defaults to mode 'w+b' which causes repo sync to
fail. By opening the tmpcookiefile in PersistentTransport.request as
writable, we are able to run sync successfully.
Bug: https://crbug.com/gerrit/12370
Test: Ran smartsync successfully
Change-Id: I01ddf915fc30eb3ff0e4d440a6f1aa261c63e88d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255692
Tested-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
If an older launcher script is used with newer repo source tree, we
might be issuing python version warnings. Plus, we want to be able
to roll Python version requirements independently of the launcher.
Add some version checking here too.
Change-Id: Ia35fc821f93c429296bdf5fd578276fef796b649
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255592
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This allows us to control sync output better by having three levels
of output: quiet (only errors), default (progress bars), verbose (all
the things). For now, we just put the chatty "already have persistent
ref" message behind the verbose level.
Bug: https://crbug.com/gerrit/11293
Change-Id: Ia61333fd8085719f3e99edb7b466cdb04031b67f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255414
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Includes the following commits (redacted to those that are relevant):
da300bd9bd8 - Do not create a change id if gerrit.createChangeId == false
731eb42b8ae - Do not strip out "-- >8 --" comment in commit-msg hook
627d07c2bfc - Handle messages with only comments in the commit-msg hook
68296f71804 - Simplify the hook script using git-interpret-trailers
Change-Id: I7a82836495427df3c5437ba88a9576b47629065f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255393
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
This gives us a bit of feedback by running our testsuite on Linux,
macOS, and Windows platforms. While Linux & macOS are passing,
Windows fails some of them. We can figure that out later. This
is better than what we have now which is manual one-offs.
Change-Id: I9d2d644be97ec76645db0bc15739e7679310a647
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255314
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We need to use the path separators provided by the python library,
and we need to set the git env vars so the name is always known.
Not all tests pass, but at least the basic frameworks work now.
Change-Id: Icea67098a8d7d58bbf918c78325681cf12a2e5f2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255313
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
- flake8 is a wrapper around pyflakes, so it's redundant to mention
both of them. Roll the explicit sections about coding errors and
coding style violations into a single section.
- After recent cleanups the project now has zero warnings or errors
from flake8. Reword the requirements so that it is now mandatory
to not introduce new warnings.
- Expand the section on suppression of warnings to differentiate
between suppressing inline individually and globally suppressing
for the whole project.
- Properly capitalize "Python Style Guide".
Change-Id: I4b333d013e985db252873441b16cb719ed5be5b5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255040
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
If programs emit non-UTF-8 output, we currently throw a fatal error.
We largely only care about the exit status of programs, and even the
output we do parse is a bit minimal. Lets make it into a warning and
mangle the invalid bytes into U+FFFD. This should complain enough to
annoy but not to break when it's not necessary.
Bug: https://crbug.com/gerrit/12337#c2
Change-Id: Idbc94f19ff4d84d2e47e01960dd17d5b492d4a8a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255272
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
All of the instances of this are related to Python 2 names that
don't exist in Python 3, and the warnings are raised when running
flake8 on Python 3.
All of these will go away once we completely remove support for
Python 2, so just suppress them inline. We don't globally suppress
the check so that we will still see legitimate errors if/when they
occur in new code.
Change-Id: Iccf955f50abfc9f83b371fc0af6cceb51037456f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255039
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Some versions of gpg on Windows mishandle native paths with homedir.
It manifests itself like:
gpg: keybox 'C:\Users\.../.repoconfig\gnupg/pubring.kbx' created
gpg: C:\Users\.../.repoconfig\gnupg/trustdb.gpg: trustdb created
gpg: key 16530D5E920F5C65: public key "Repo Maintainer <repo@android.kernel.org>" imported
gpg: can't connect to the agent: Invalid value passed to IPC
gpg: Total number processed: 1
gpg: imported: 1
fatal: registering repo maintainer keys failed
It seems gpg (at least version 2.2.17) needs paths to be specified
in cygwin form (e.g. "/c/Users/.../.repoconfig/gnupg") otherwise
it fails to talk to its own processes. We can work around this
with a minor trick: we cd to the right path and then invoke gpg
with --homedir . and let gpg itself resolve . to whatever form it
really wants.
This is a bit hacky, but we don't control gpg, and this allows us
to avoid having to muck with the environment. Since --homedir has
been around since at least gpg-1.4.x from 2004, backwards compat
shouldn't be an issue.
While we're here, touch up the output a bit: there's no need to
dump all the chatty gpg output if things don't fail, so always
swallow the output. If things do fail, our exception handler
takes care of dumping the full stdout & stderr.
Change-Id: I74ab98e1e61e95318fda6faf57c6a8699f775935
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255120
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We added support for `repo init -c` to main.py, but not to the
launcher, so the -c option only works after the first init has
run which kind of defeats its purpose. Rework the parser setup
so that we can tell it whether it's for "init" or "gitc-init"
and then add the -c option in the same way we do in main.py.
This has the benefit of getting the parser entirely out of the
module scope which makes it a lot easier to reason about, and
it means we can write some unittests.
Change-Id: Icbc2ec3aceb938d5a8f941d5fbce1548553dc5f7
Test: repo help init
Test: repo help gitc-init
Test: repo init -u https://android.googlesource.com/platform/manifest -c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255113
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We've been using a DSA/1024 key to sign our tags. Time to update to
something a bit newer. We'll include RSA & ECC keys, but only use
RSA keys initially for backwards compatibility and see how it goes
with our user base.
Change-Id: I683c97b6fbd860f220ed4ddc7b21f07db279a916
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255112
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We can't rely on subprocess.run yet as that requires Python 3.6,
but we can clean up the code we have with some ad-hoc replacement.
This unifies all the inconsistent subprocess.Popen usage we have.
Change-Id: I56af40a3df988ee47b299105d692ff419d07ad6b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254754
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
- W503 line break before binary operator
- W504 line break after binary operator
There doesn't seem to be a nice way of fixing all of these without
replacing W503 with W504 or vice-versa, or unwrapping them resulting
in excessively long lines. Let's just suppress them.
Change-Id: I7846d0124054f58e1cb480d4459cd9c86b737a50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254608
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
flake8 reports:
E722 do not use bare 'except'
Replace them with 'except Exception' per [1] which says:
Bare except will catch exceptions you almost certainly don't want
to catch, including KeyboardInterrupt (the user hitting Ctrl+C) and
Python-raised errors like SystemExit
If you don't have a specific exception you're expecting, at least
except Exception, which is the base type for all "Regular" exceptions.
[1] https://stackoverflow.com/a/54948581
Change-Id: Ic555ea9482645899f5b04040ddb6b24eadbf9062
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254606
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E301,E302,E303,E305,E306
Manually fix issues in project.py caused by misuse of block comments.
Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E121 continuation line under-indented for hanging indent
- E122 continuation line missing indentation or outdented
- E125 continuation line with same indent as next logical line
- E126 continuation line over-indented for hanging indent
- E127 continuation line over-indented for visual indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E121,E122,E125,E126,E127,E128,E129,E131
Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
The Google style guide for python [1] says the maximum line length
should be 80, but there are several lines in the code base that
exceed it:
git ls-files | grep py$ | xargs flake8 | grep E501 | wc -l
64
I don't think it's worth going through and re-wrapping all those,
so just increase the limit to 100 which seems to be a reasonable
compromise:
git ls-files | grep py$ | xargs flake8 | grep E501 | wc -l
6
Leave the re-rewrapping of those lines for a follow-up commit,
though.
[1] http://google.github.io/styleguide/pyguide.html#32-line-length
Change-Id: Ia37c34301163431fd1fb4fb6697a4a482d6be077
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254595
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
We have a few files that we optionally symlink from the work tree
.git/ to the .repo/projects/ path. If they don't exist when we
first initialize, then we skip creating symlinks. If the files
are created later on under the work tree .git/, repo gets upset.
This can happen with the packed-refs file: if we don't have any
packed refs initially, we don't symlink it. But if git tries to
pack refs later on and creates the file, the project gets wedged.
We could create an empty file initially and then symlink it, but
for some files, it's not clear we want to always do that (e.g.
the .git/shallow setting). Instead, lets make handling of these
paths more dynamic. If they show up later on in the work tree
.git/ only, we'll take care of relocating & symlinking. This
also makes repo a little more robust and autorecovers incase a
path goes missing in one of the dirs.
Ideally we wouldn't monkey around at all here, but considering
the only option we give to users currently is to blow things
away with --force-sync, this seems a bit better.
Bug: https://crbug.com/gerrit/12324
Change-Id: Ia6960f1896ac6d890c762d7d053684a1c6ab2c87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254632
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When upload hooks fail, people are forced to use --no-verify to upload
CLs anyways. When projects have flaky hooks, this trains people to
always use that option. This is obviously bad: hooks might get fixed,
or some of the hooks are always good & people should review.
Lets add an --ignore-hooks option. This still runs the hooks, but any
failures will be ignored and allow the user to upload anyways.
Bug: https://crbug.com/gerrit/12230
Change-Id: Ide2ac8a40a656bfcd6aae20c3ce8118e06bf909b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254452
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
flake8 reports:
F811 redefinition of unused 'test_src_block_dir' from line 259
which is caused by having two methods with the same name. Rename
them both to better desribe their purpose.
Change-Id: If7612a42001776d71bb1a6a80fc631d3d262e6ce
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254449
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
We were perhaps a bit too hasty to jump to git-2.10. Existing LTS
releases of Ubuntu are quite old still: Trusty has 1.9 while Xenial
has 2.5. While we plan on dropping support for those eventually as
we migrate to Python 3.6, we don't need to be so strict just yet on
the git versions.
We also want to disconnect the version the repo launcher requires
from the version the rest of the source tree requires. The repo
launcher doesn't need as many features, and being flexible there
allows us more freedom to upgrade & rollback as needed.
So we'll allow git-1.7 again, but start warning on any users older
than git-1.9. This aligns better with existing LTS releases, and
gives users a chance to start upgrading before we cut them off.
Change-Id: I140305dd8e42c9719c84e2aee0dc6a5c5b18da25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254573
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Going purely on upstream package release cycles doesn't tell the whole
story: a lot of people run LTS distros which will have older versions
of software we want to support.
Build out a table for us to quickly reference when making decisions as
to what versions of git/python we want to support, and when we can drop
them. This will also help to refer users to as why we made a specific
decision that might be affecting them.
Change-Id: I7aea24bbefd50e358aeacf11e8c15a346c8fb8a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254572
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Some automatic git operations will prune objects on us, and not just
the gc step. Normally we don't care, but with shared projects, we
will have multiple git checkouts with refs that the others cannot
see, but with a shared object dir. Any pruning of objects based on
refs in just one repo can easily break the others.
git-2.7.0 introduced a preciousObjects setting which tells git to
never prune objects for this exact scenario: there might be refs in
some location that git is unable to see.
Change-Id: I781de27c5bbe1d4c70f0187566141c9cce088bd8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254392
Reviewed-by: Nasser Grainawi <nasser@codeaurora.org>
Reviewed-by: David Riley <davidriley@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The code has an ad-hoc check in that it requires the launcher major
version to not be less than the source code version. We don't really
care about that requirement, and it doesn't fit with our other version
checks. Rework it so we explicitly declare the min launcher version
that is supported.
We'll start with requiring repo launcher 1.15 which was released back
in 2012. Hopefully no one has anything older than that, although it's
not clear we work with even newer versions than that :). But let's be
a little conservative with the first update to this logic.
Bug: https://crbug.com/gerrit/10418
Change-Id: I611d70c60324d313c76874e978b8499a491a5d00
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254278
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Execution of 'repo forall -p -c' doesn't work with Py3 and ends up
with an error:
Got an error, terminating the pool: TypeError: can only concatenate
str (not "bytes") to str
That's fixed by using the decode() method.
Change-Id: Ice01aaa1822dde8d957b5bf096021dd5a2b7dd51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253659
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Jiri Tyr <jiri.tyr@gmail.com>
Currently our default behavior is:
* Try to sync all repos
* If any errors seen, exit
* Try to garbage collect all repos
* If any errors seen, exit
* Try to update local project list
* If any errors seen, exit
* Try to checkout out all local repos
* If any errors seen, exit
Users find these incomplete syncs confusing, so lets try to complete
as much as possible by default and printing out summaries at the end.
Bug: https://crbug.com/gerrit/11293
Change-Id: Idd17cc9c3bbc574d8a0f08a30225dec7bfe414cb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238554
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We want to start warning about Python 2 usage, but we can't do it
simply because the shebang is /usr/bin/python which might be an old
version like python2.7.
We can't change the shebang because program name usage is spotty at
best: on some platforms (like macOS), it's not uncommon to not have
a `python3` wrapper, only a major.minor one like `python3.6`. Using
python3 wouldn't guarantee a new enough version of Python 3 anyways,
and we don't want to require Python 3.6 exactly, just that minimum.
So we check the current Python version. If it's older than the ver
of Python 3 we want, we search for a `python3.X` version to run. If
those don't work, we see if `python3` exists and is a new enough ver.
If it's not, we die if the current Python 3 is too old, and we start
issuing warnings if the current Python version is 2.7. This should
allow the user to take a bit more action by installing Python 3 on
their system without having to worry about changing /usr/bin/python.
Once we require Python 3 completely, we can simplify this logic a bit
by always bootstrapping up to Python 3 and failing with Python 2.
We have a few KI with Windows atm though, so keep it disabled there
until the fixes are merged.
Bug: https://crbug.com/gerrit/10418
Change-Id: I5e157defc788e31efb3e21e93f53fabdc7d75a3c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253136
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
The git-2.10 series was released in 2016. Since we're moving to
require Python 3.6 which was also released in 2016, bumping up the
git version seems reasonable. Also we don't really test any git
versions close to as old as 1.7.2 which was released in 2010.
Change-Id: Ib71b714de6cd0b7dd50d0b300b108a560ee27331
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253134
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The use case is any situation where your manifest does
not exist on server, but where you still want to do
full sync for the projects, without having your
workspace manifest switched to other branch or
forwarded to latest or similar.
This allows syncing to a historical manifest in git log,
that does not have a branch, as well as when integrating
something together that has not been pushed upstream yet.
Changes can also exist locally on a manifest that is
behind head, meaning not requiring rebase to latest.
Tested using:
$ cd .repo/manifests/
$ git checkout <any hash 1>
$ <do local modifications>
$ repo sync --no-manifest-update
$ git checkout <any hash 2>
$ repo sync --no-manifest-update
Change-Id: I0c9773aa8bc5876813a2e7d7fec697abcb2d9e94
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/246445
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This commit supports for the 'remote' attribute in
<extend-project>. This avoids the need to perform a <remove-project>
followed by a <project> in local manifests.
Change-Id: I9f9347913337ec9d159bc264d15ce97881ae5398
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253092
Tested-by: Kyunam Jo <kyunam.jo@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reject paths in <copyfile> & <linkfile> that try to use symlinks or
non-file or non-dirs.
We don't fully validate <linkfile> when src is a glob as it's a bit
complicated -- any component in the src could be the glob. We make
sure the destination is a directory, and that any paths in that dir
are created as symlinks. So while this can be used to read any path,
it can't be abused to write to any paths.
Bug: https://crbug.com/gerrit/11218
Change-Id: I68b6d789b5ca4e43f569e75e8b293b3e13d3224b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233074
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
This reverts commit 1e01a74445.
Not all platforms support select.poll() currently it seems.
At least macOS's Python 2 doesn't (while macOS Python 3 does).
Lets back this out for the existing release series and once we
start repo-2 which is Python 3-only, we can put this back in.
Change-Id: I205206b0fa4fe2d755f4fbc6ec683ad125f27cc2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253072
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
After Ib546f5ebbc8a23875fbd14bf166fbe95b7dd244e, repo info now displays
the current project revision in the 'Current revision' field.
While the output is more consistent, there are use cases for the
revision expression as shown in the manifest. This patch re-adds the
manifest revision as a new 'Manifest revision' field.
Change-Id: I50c1559dcb7ceb69af07352b956d78f85b8f592e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/240799
Tested-by: Diogo Ferreira <deovferreira@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
select() has a limit of FD_SETSIZE file descriptors. If you run repo
sync -j500 you'll pretty quickly hit this limit and get "file descriptor
out of range for select" errors. poll() has no such limit.
Change-Id: I21f350e472bda1db03dcbcc437645c23dbc7a901
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248852
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Theodore Dubois <tbodt@google.com>
Before a2cd6aeae8, "repo mirror with --current-branch" fetches git data
using command
git fetch --progress --update-head-ok cros --tags
No refspec is specified, thus it fetches default refspec, which is +refs/heads/*:refs/heads/*
After a2cd6aeae8, the fetch command became
git fetch --progress --update-head-ok cros --tags +refs/tags/*:refs/tags/*
It did not only add tags refspec, but also suppressed the fetching of default refspec.
In other words, repo mirrors doesn't work if current_branch_only=True.
This CL explicitly adds the default refspec to command line if none is
specified.
Bug: https://crbug.com/gerrit/11990
Change-Id: Iadcf7b9aa50f53c47132cfe6c53b3fb2076ebca2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/246632
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Kuang-che Wu <kcwu@chromium.org>
The upload module tries to turn the strings into bytes before passing
to EditString, but it combines bytes & strings causing an error. The
return value might be bytes or string, but the caller only expects a
string. Lets simplify this by sticking to strings everywhere and have
EditString take care of converting to/from bytes when reading/writing
the underlying files. This also avoids possible locale confusion when
reading the file by forcing UTF-8 everywhere.
Bug: https://crbug.com/gerrit/11929
Change-Id: I07b146170c5e8b5b0500a2c79e4213cd12140a96
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/245621
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Series of steps:
* Create a local "b1" branch with `repo start b1` that tracks a remote
branch (totally fine)
* Manually create a local "b2" branch with `git branch --track b1 b2`
that tracks the local "b1" (uh-oh...)
* Delete the local "b1" branch manually or via `repo prune` (....)
* Try to process the "b2" branch with `repo prune`
Since b2 tracks a branch that no longer exists, everything blows up
at this point as we try to probe the non-existent ref. Instead, we
should flag this as unknown and leave it up to the user to resolve.
This probably could come up if a local branch was tracking a remote
branch that was deleted from the server, and users ran something like
`repo sync --prune` which cleaned up the remote refs.
Bug: https://crbug.com/gerrit/11485
Change-Id: I6b6b6041943944b8efa6e2ad0b8b10f13a75a5c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236793
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Kirtika Ruchandani <kirtika@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the local branch changed state since its last upload, the data
cached in .git/config related to the last uploaded CL might not be
that relevant. If we're able to fast forward merge to the latest
tree state, then let's do that. This would be akin to checking
out a detached head before syncing where we already switch state.
If we aren't able to fast forward merge, then it's not a big deal
as we'll continue on to the existing branch checking logic.
This would be easy to reproduce by doing something like:
$ repo start foo .
$ git revert HEAD
$ repo upload --cbr .
$ git reset --hard HEAD^
<CL is merged>
$ repo sync .
<we can fast forward>
Change-Id: I7d62f3d1ba5314a349d85b4dbb0ec8352eca18bb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238552
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
The current sync output displays "Fetching project" and "Checking out
project" messages and progress bar updates independently leading to a
lot of spam. Lets merge these periodic outputs with the status bar to
get a little bit tighter output in the normal case. This doesn't solve
all our problems, but gets us closer.
Bug: https://crbug.com/gerrit/11293
Change-Id: Icd627830af4dd934a9355b7ace754b56dc96cfef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244934
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If you run `repo sync -l foo` without first `repo sync -n foo`,
repo sets up an invalid gitdir tree that gets wedged and requires
manual recovery. Add a sanity check to abort cleanly first.
Change-Id: Iad865ea860a3f1fd2f39ce683fe66bd4380745a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244732
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When repo sync fails because some git trees are not in clean state and
as such can not be rebased automatically, it is a pain to figure out
which trees are the culprits.
With this patch the list of offending trees is printed when repo sync
reports checkout errors.
TEST=ran 'repo sync' and observed the proper list of directories show
up after the final error message
Bug: https://crbug.com/gerrit/11293
Change-Id: Icdf1a03e9014ecb184f331f513cc9a2efc7d11ed
Signed-off-by: Vadim Bendebury <vbendeb@google.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244053
Reviewed-by: Mike Frysinger <vapier@google.com>
Hitting Ctrl-C in the middle of this func will leave the .git in a
bad state that requires manual recovery. The code tries to catch
all exceptions and recover by deleting the incomplete .git dir, but
it omits KeyboardInterrupt which Exception misses.
We could add that to the recovery path, but we can make this more
robust with a different approach: set up everything in .git.tmp/
and only move it to .git/ once we've fully initialized it.
Change-Id: I0f5b97f2e19fc39cffc3e5e23993a2da7220f4e3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244733
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The plan previously documented was <=1.13.x is Python 2 and >=1.14.x
is Python 3. Other projects that migrated Python versions and drop
support for older have tended to take a more drastic version jump to
make it clearer to users. So lets adjust the plan to say <=1.x will
support Python 2, and >=2.x will be Python 3-only.
This also allows us to harmonize the repo launcher version. It is
currently sitting at v1.26 and has been incremented independently of
the repo version for the life of the project. While we might know
these lower nuances, pretty much no one else does and it just leads
to confusion: do I know version 1.26 or version 1.13.7? Or do I
have both? What does that even mean?
Once we update the major version to 2.0.0, we can also adjust the
launcher script to 2.0.0, and then the launcher release process will
be tied to a new repo release in general.
Bug: https://crbug.com/gerrit/10418
Change-Id: Idb2257371a06e56d2923cf717345c028f49176a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/240372
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
`repo forall <proj>` will look up all <proj> in the manifest for all
manifest groups regardless of which are active. If <proj> is checked
out to different locations depending on the group, this ultimately
fails as we're unable to locate all of them.
Simple fix is to only include projects that match the manifest groups
that we already passed down & initialized to the active set, and that
we already use when getting the default project list.
Bug: https://crbug.com/gerrit/11677
Bug: https://crbug.com/1011226
Change-Id: I975f10f9a9e5a1cad7d87344123f8003732dab27
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239652
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raul Rangel <rrangel@chromium.org>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The "Current revision" field shows the revision as listed in the
manifest. I think most users expect this to show the revision
that the git tree is checked out to instead. Switch the output
to show that revision instead, and add a "Current branch" if it
matches a local branch.
Change-Id: Ib546f5ebbc8a23875fbd14bf166fbe95b7dd244e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239240
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
imp is deprecatedon py3. It's also not used with py3, so just move it
to the py2 import block
Test: run `repo` command and verify warning is no longer present
Test: verify `repo sync` and `repo upload` function as expected
Change-Id: I9d59403d7819c4a478c9f54cbef114f8a96486a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239713
Tested-by: Rashed Abdel-Tawab <rashedabdeltawab@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If the repo client checkout is in an incomplete sync state, the work
git repo might be in a bad way. Turn errors parsing HEAD into None
since callers of CurrentBranch already need to account for it.
Change-Id: Ia7682e29ef4182006b1fb5f5e57800f8ab67a9f4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239239
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If the user passes in bad projects like `repo info asdf`, we currently
silently swallow those and do nothing. Allow NoSuchProjectError to
bubble up to main which will handle & triage this correctly for us.
Change-Id: Ie04528e7b7a164293063a636813a73eaabdd5bc3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239238
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Depending on where/how repo is invoked, the active version might be
from a git tree, and it might be different from the .repo/repo.git/
version in the current repo client checkout. Report both if they're
different so it's clearer.
Lets also include the two different User-Agent's that we set up when
talking to networked services.
Bug: https://crbug.com/gerrit/11144
Change-Id: I2ebb6e3ac30e374a8406cab3e4438087246a8c57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239234
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We've been setting the User-Agent header when making connections
from repo itself, but not when running git (as the latter will set
up User-Agent itself). Our Gerrit/Git admins say it'll be helpful
if we pass through the repo version settings even when running git.
We currently set GIT_HTTP_USER_AGENT and not GIT_USER_AGENT as it's
unclear if the extended form works over all protocols. We can wait
for a user request.
Bug: https://crbug.com/gerrit/11144
Change-Id: I21d293f49534058dbc23225152451df26c5b7bef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239233
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Convert the RepoUserAgent function into a UserAgent class. This
makes it cleaner to hold internal state, and will make it easier
to add a separate git User-Agent, although we don't do it here.
We make the RepoSourceVersion independent of GitCommand so that
it can be called by the class (later).
Bug: https://crbug.com/gerrit/11144
Change-Id: Iab4e1f974b8733a36b243b2d03f5085a96effa19
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239232
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Lets switch the default rebase behavior to align with our new sync
behavior: we try to rebase all projects by default and exit/summarize
things at the very end if there were any errors. Or if people want
to exit immediately, they can use the new --fail-fast option.
Change-Id: I436ac563f972b45de6ce9ad74da1e4870e584902
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238553
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Cut out some more standalone code from Execute to make this func a
bit more manageable. The manifest project update is pretty simple
and standalone, but still takes up a good chunk of what's left.
Change-Id: Idc2442d9def495eccd0a49cda203c44aef16f129
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236614
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
When displaying progress bars, we use \r to reset the cursor to the
start of the line before showing the new update. This assumes the
new line will fully erase whatever was displayed there previously.
The "done" codepath tries to handle this by including a few extra
spaces at the end of the message to "white out" what was there.
Lets replace that hack with the standard ECMA escape sequence that
clears the current line completely. This is the CSI "erase in line"
sequence that the terminal will use to delete all content. The \r
is still needed to move the cursor to the start of the line. Using
this sequence should be OK since we're already assuming the terminal
is ECMA compliant with our use of coloring sequences. We also put
the \r after the CSI sequence on the off chance the terminal can't
process it and displays a few bytes of garbage.
The other improvement is to the syncbuffer API. When it dumps its
status information, it almost always comes after a progress bar
update which leads to confusing comingled output. Something like:
Fetching projects: 100% (2/2) error: src/platform2/: branch ...
Since the progress bar is "throw away", have the syncbuffer reset
the current output to the start of the line before showing whatever
messages it has queued.
Bug: https://crbug.com/gerrit/11293
Change-Id: I6544d073fe993d98ee7e91fca5e501ba5fecfe4c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236615
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The smart sync logic takes up about 45% of the overall Execute func
and is about 100 lines of code. The only effect it has on the rest
of the code is to set the manifest_name variable. Since this func
is already quite huge, split the smart sync logic out.
Change-Id: Id861849b0011ab47387d74e92c2ac15afcc938ba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234835
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This makes it way easier to recover from forgetting to run repo start
before committing: just run `repo start -b new-branch`, instead of
all that tedious mucking around with reflogs.
Change-Id: I56d49dce5d027e28fbba0507ac10cd763ccfc36d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232712
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
In commit d9e5cf0e ("sync: invert --force-broken with --fail-fast") the
force-broken option has been deprecated. Accidentally the option has
been changed from Boolean to Value. This breaks all users of repo with:
main.py: error: -f option requires an argument
This is easy to avoid by keeping the type.
Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Change-Id: Ia8b589cf41ac756d10c61e17ec8d76ba8f7031f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/235043
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
A common pattern in our subcommands is to verify the arguments &
options before executing things. For some subcommands, that check
stage is quite long which makes the execution function even bigger.
Lets split that logic out of the execute phase so it's easier to
manage these.
This is most noticeable in the sync subcommand whose Execute func
is quite large, and the option checking makes up ~15% of it.
The manifest command's Execute can be simplified significantly as
the optparse configuration always sets output_file to a string.
Change-Id: I7097847ff040e831345e63de6b467ee17609990e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234834
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
People seem to not expect the sync process to halt immediately if an
error is encountered. It's also basically guaranteed to leave their
tree in an incomplete state. Lets invert the default behavior so we
attempt to sync (both fetch & checkout) all projects. If an error is
hit, we still exit(1) and show it at the end.
If people want the sync to abort quickly, they can use the new option
--fail-fast.
Bug: https://crbug.com/gerrit/11293
Change-Id: I49dd6c4dc8fd5cce8aa905ee169ff3cbe230eb3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234812
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Callers don't actually see -1 (they'll usually see 255, but the exact
answer here is complicated). Just switch to 1 as that's the standard
value tools use to indicate an error.
Change-Id: Ib712db1924bc3e5f7920bafd7bb5fb61f3bda44f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233553
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The partial clone rework (commit 745be2ede1
"Add support for partial clone") changed the behavior when a single repo
hit a failure: it would always call sys.exit() immediately. This isn't
even necessary as we already pass down an error event object which the
workers set and the parent checks. Just delete the exit entirely.
Change-Id: Id72d8642aefa2bde24e1a438dbe102c3e3cabf48
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233552
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The GitCommand Wait helper takes care of decoding bytes to strings
for us. That means we don't have to decode stdout ourselves which
is what our local rev list, ls-remote, and generic get_attr helpers
were doing.
If we don't use Wait though to capture the output but instead go
directly to the subprocess stdout, we do have to handle decoding
ourselves. This is what the diff helpers were doing.
Bug: https://crbug.com/gerrit/10418
Change-Id: I057ca245af3ff18d6b4a074e3900887f06a5617d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233076
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
As we convert repo to support Python 3, the version of Python that we
use might not be the version that repo hooks users have written for.
Since repo upgrades are not immediate, and not easily under direct
control of end users (relative to the projects maintaining the hook
code), allow hook authors to declare the version of Python that they
want to use.
Now repo will read the shebang from the hook script and compare it
against the version of Python repo itself is running under. If they
differ, we'll try to execute a separate instance of Python and have
it load & execute the hook. If things are compatible, then we still
use the inprocess execution logic that we have today.
This allows repo hook users to upgrade on their own schedule (they
could even upgrade to Python 3 ahead of us) without having to worry
about their supported version being exactly in sync with repo's.
Bug: https://crbug.com/gerrit/10418
Change-Id: I97c7c96b64fb2ee465c39b90e9bdcc76394a146a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228432
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
A new option, --partial-clone is added to 'repo init' which tells repo
to utilize git's partial clone functionality, which reduces disk and
bandwidth usage when downloading by omitting blob downloads initially.
Different from restricting clone-depth, the user will have full access
to change history, etc., as the objects are downloaded on demand.
Change-Id: I60326744875eac16521a007bd7d5481112a98749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
We're relying on sys.stdout.write() to flush its buffer which isn't
guaranteed, and is not the case in Python 3. Change to use print()
everywhere to be standard, and utilize the end= keyword to get the
EOL semantics we need.
We can't use print's flush= keyword as that's only in Python 3.
Leave behind a TODO to clean it up when we can drop Python 2.
Bug: https://crbug.com/gerrit/10418
Change-Id: I562128c7f1e6d154f4a6ecdf33a70fa2811dc2af
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/230392
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This fix exception with python3 with stack-trace:
error: Cannot fetch platform_external_grpc-grpc-java.git (UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 640: invalid start byte)
[...]
File "[...]project.py", line 2255, in _IsValidBundle
if f.read(16) == '# v2 git bundle\n':
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
Even if we ask 16 characters, python buffered decoder will try to decode more in the buffer
The patch works for python2 and python3, and open the file in byte mode so that decoding is not attemped
Signed-off-by: Pierre Tardy <tardyp@gmail.com>
Change-Id: I837ae3c5cd724b34670fc2a84e853431f482b20d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/224642
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We've been truncating the git version info in the user agent to the
first three components. So given an example `git --version` like
"2.22.0.510.g264f2c817a-goog", we were cutting it down to "2.22.0".
For user-agent purposes, we usually want that full string, so use
the original full value instead.
Bug: https://crbug.com/gerrit/11144
Change-Id: I8ffe3186bdaac96164c34ac835a54bb3fc85527e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231056
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We were returning an e.g. tuple(1,2,3), but that strips off the full
version string which we might want in some places e.g. '1.2.3-rc3'.
Change the return value to a namedtuple so we can pass back up the
full version string. For code doing a compare with three elements
(all code today), things still work fine as the namedtuple will DTRT
in this scenario.
Bug: https://crbug.com/gerrit/11144
Change-Id: Ib897b5df308116ad1550b0cf18f49afeb662423e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231053
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
%prog represents the full subcommand ("repo" + subcommand name), not a
Windows-style environment variable for "repo". The current help output
shows
repo forall% forall ...
Correct the variable usage so it shows "repo forall ..." instead.
Change-Id: I1fea55572428cc922ddf24ace1168a3d8f82dad0
This takes a single argument (the error message), not multiple
arguments that get formatted implicitly.
Change-Id: Idfbc913ea9f93820edb7e955e9e4f57618c8cd1b
Currently we read the binary stream from the subprocess code directly
before waiting for it to finish, but there's no need to do so as we
aren't streaming the output to the user. This also means we pass up
binary data to the caller as we don't go through GitCommand's internal
logic which decodes the stream as utf-8.
Simplify the code by calling Wait first, then splitting the entire
captured output in one line.
Bug: https://crbug.com/gerrit/10418
Change-Id: I7a57904be8cb546a229980fb79c829fc3df31e7d
Python 3 returns bytes by default with urlopen. Adjust our code to
handle that scenario and decode as necessary.
Bug: https://crbug.com/gerrit/10418
Change-Id: Icf4cd80e7ef92d71a3eefbc6113f1ba11c32eebc
Neither of the fields here expect floats so make sure we use integer
division when calculating things.
Bug: https://crbug.com/gerrit/10418
Change-Id: Ibda068b16a7bba7ff3efba442c4bbff4415caa6e
This module uses print() so make sure we import the print function.
It doesn't really impact the current code due to the simple way it
is calling print, but we should be sane to avoid future issues.
Change-Id: I0b15344678c1dcb71207faa333c239b3fced1d62
We avoided this future import because Python 2.4 & 2.5 did not
support it. We've dropped support for Python 2.6 at this point,
and those versions are long dead. Since this workaround adds a
bit of complexity to the codebase, drop it. Considering we are
not running any actual tests against older versions, there's no
sense in trying to support them anymore.
Change-Id: Icda874861e8a8eb4fa07c624a9e7c5ee2a0da401
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.
Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
Use the `raise` statement directly.
Switch to using .items() instead of .iteritems(). Python 3 doesn't
have .iteritems() as .items() is a generator, and these are small
enough that the Python 2 overhead should be negligible.
We have to run .keys() through list() in a few places as Python 3
uses a generator and we sometimes want to iterate more than once.
That's why we don't change all .keys() or .items() calls -- most
are in places where generators are fine.
Bug: https://crbug.com/gerrit/10418
Change-Id: I469899d9b77ffd77ccabb831bc4b217407fefe6f
Standard utilities exit normally/zero when users explicitly request
--help, and they write to stdout. Exiting non-zero & using stderr
is meant for incorrect tool usage instead. We're already doing this
for `repo help <init|gitc-init>` calls, so lets fix `repo help` and
`repo --help|-h` to match.
Change-Id: Ia4f352b431c91eefef70dcafc11f00209ee69809
This isn't executable (+x), nor does it have a main func or code
that would run if it were. It's simply an imported module like
most others in here. Drop the shebang to avoid confusion.
Change-Id: I5e2881eb1de5e809a3fa9e8f49220ed797034fb1
The current implementation ignores the user-specified paths to
manifests. if the "repo diffmanifests" is invoked with absolute
file paths for one or both manifests, the command fails with message:
fatal: duplicate path ... in /tmp/manifest-old.xml
Also the current implementation fails to expand the absolute path to
manifest files if "repo diffmanifests" is invoked with relative
paths, i.e "repo diffmanifests manifest-old.xml manifest-new.xml".
fatal: manifest manifest-old.xml not found
This commit fixes the first issue by disabling the local manifest
discovery for diffmanifests command, and the second issue by
expanding paths to manifests within "diffmanifests" sub-command.
Test: repo manifest --revision-as-HEAD -o /tmp/manifest-old.xml
repo sync
repo manifest --revision-as-HEAD -o /tmp/manifest-new.xml
repo diffmanifests /tmp/manifest-old.xml /tmp/manifest-new.xml
Change-Id: Ia125d769bfbea75adb9aba81abbd8c636f2168d4
Signed-off-by: Vasyl Gello <vasek.gello@gmail.com>
During sync, repo runs `git read-tree --reset -u -v HEAD` which causes
git-lfs's smudge filter to run, which fails because git-lfs does not
work with bare repositories.
This was fixed in I091ff37998131e2e6bbc59aa37ee352fe12d7fcd to
automatically disable this smudge filter. However, later versions of
Git (2.11.0) introduced a new filter protocol [1], to avoid spawning
a new command for each filtered file. This was implemented in Git-LFS
1.5.0 [2].
This patch fixes the issue by setting the git lfs process filter, in
addition to the smudge filter. For any projects that have LFS objects,
`git lfs pull` must still be executed manually afterwards.
[1] edcc85814c
[2] https://github.com/git-lfs/git-lfs/pull/1617
Bug: https://crbug.com/gerrit/10911
Change-Id: I277fc68fdefc91514a2412b3887e3be9106cab48
If a user is asking for a shallow clone of the repos, they probably
expect a shallow clone of the manifest repo too. For very large
manifest repos, this can be a huge space and time savings. For one real-world
repo, a 'repo init --no-tags --current-branch' used 350MB of disk space and
took 7 minutes. Adding --depth 1 and this change reduced it to 10MB and 2.5
minutes.
Change-Id: I6fa662e174e623ede8861efc862ce26d65d4958d
When syncing a lot of projects in parallel, it is not otherwise
clear which one of them has failed to init work tree.
Change-Id: I8edfb4955023389a499e99cfa511bdc0d2850ba2
This allows projects to include submodules inside of
projects that use repo without repo incorrectly believing
the area is dirty just because a submodule has updates.
This is in line with git porcelain commands which generally
require a commandline flag to include submodules (git add,
git rebase).
Change-Id: Ide8a292162a42ab35145b5c4ca8ca0d020cdfe81
If "repo init" was run in a path containing "%", "repo info" would fail
printing the path with
File ".repo/repo/color.py", line 173, in f
return fmt % args
TypeError: not enough arguments for format string
as the "%" in the path name is interpreted as the start of a formatting
specifier. Avoid that by using the non-formatting printer for headtext
which does not require any formatting so there is no need to try to
expand "%" sequences.
Change-Id: Ie193b912191fe7cdabdce5c97bb100f0714f6e76
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
The newly introduced "Already have persistent ref" message prevents
repo from overwriting the last line when syncing quietly. Omit the
message when syncing quietly to clean up the output and to restore
the previous behaviour.
Change-Id: Idf42751c67f95924d6de50092ba54d4c6fe93096
The comment in _create_symlink is incorrect. The return value of
CreateSymbolicLink is as documented, it was just declared with
the wrong return type. The actual return type is BOOLEAN, not BOOL.
Fixing this allows us to simplify the code a bit.
Change-Id: I4d2190a50d45ba41dd9814bf7079a5784fc0a366
If a tag is rewritten on the server (which is bad), trying to recover
locally with `repo sync --force-sync` doesn't actually work. People
have to manually delete things themselves to fix syncing. While tags
should never be rewritten in practice, allow users to easily recover
from broken servers.
We updated some of these code paths already (see commit 6e53844f1e
"Allow clobbering of existing tags from remote."), but the incremental
update flow was missed.
Bug: b/120778183
Bug: chromium:932651
Test: delete local tag & recreate to diff commit, then check
`repo sync` & `repo sync --force-sync` behavior
Change-Id: I3648f7d2526732c06016b691a9a36c003157618d
Forcefully remove dirty projects if option '--force-remove-dirty' is given.
The '--force-remove-dirty' option can be used to remove previously used
projects with uncommitted changes. WARNING: This may cause data to be lost
since uncommitted changes may be removed with projects that no longer exist
in the manifest.
Change-Id: I844a6e943ded522fdc7b1b942c0a1269768054bc
Even if dots are used as separators for Git config keys, they are not
forbidden as part of submodule names. This fixes the issue of submodules
with a name like e.g. "long.js" to be skipped from checkout.
Change-Id: I77da07925ad207fa3d043067dfbbcb4a1ebdac4d
Running lots of sync processes in parallel can hit the failure:
Fetching projects: 23% (124/523)Exception in thread Thread-201:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/src/repo/subcmds/sync.py", line 278, in _FetchProjectList
success = self._FetchHelper(opt, project, *args, **kwargs)
File "/usr/local/src/repo/subcmds/sync.py", line 357, in _FetchHelper
start, finish, success)
File "/usr/local/src/repo/event_log.py", line 104, in AddSync
event = self.Add(project.relpath, task_name, start, finish, success)
File "/usr/local/src/repo/event_log.py", line 74, in Add
'id': (kind, next(self._next_id)),
ValueError: generator already executing
It looks like, while we lock the multiprocessing value correctly, the
generator that wraps the value isn't parallel safe. Since we don't
have a way of doing that (as it's part of the language), turn it into
a plain function call instead.
Bug: https://crbug.com/gerrit/10293
Change-Id: I0db03601986ca0370a1699bab32adb03e7b2910a
When --ne/--no-emails is added to 'repo upload' command line, gerrit
server will not generate notification emails.
project.py:Project.UploadForReview method is modified to accept a
string recognizable by gerrit to indicate different sets of destination
email addressees, but the upload command line allows only one option -
disable sending emails completely.
Default repo upload behavior is not being changed.
TEST=tried in the Chrome OS repo, observed that patches uploaded with
--ne or --no-emails indeed do not trigger any emails, while
patches uploaded without these command line options still trigger
email notifications.
Change-Id: I0301edec984907aedac277d883bd0e6d3099aedc
Pass options through the refspec for all transports, including ssh.
This means the behavior will be more consistent between the ssh and
https cases.
A downside is that this prevents passing special characters in
reviewer options. That already didn't work over https, so it seems
okay. It could be fixed by using push options instead.
Change-Id: Ia38d16e350cb8cb0de14463bfb3d9724e13bc4bf
* Add more file i/o wrappers in platform_utils to allow using
long paths (length > MAX_PATH) on Windows.
* Paths using the long path syntax ("\\?\" prefix) should never
escape the platform_utils API surface area, so that this
specific syntax is not visible to the rest of the repo code base.
* Forward many calls from os.xxx to platform_utils.xxx in various place
to ensure long paths support, specifically when repo decides to delete
obsolete directories.
* There are more places that need to be converted to support long paths,
this commit is an initial effort to unblock a few common use cases.
* Also, fix remove function to handle directory symlinks
Change-Id: If82ccc408e516e96ff7260be25f8fd2fe3f9571a
"repo init --reference" has two purposes: to decrease bandwidth used
at clone time, and to decrease disk usage afterward, by using the
reference repositories as an alternate store of objects even after
the clone. The downside is that it makes the borrowing repositories
dependent on the reference repositories, so it is easy to end up
with missing objects by mistake after a cleanup operation like "git
gc".
To prevent that, v2.3.0-rc0~30^2 (clone: --dissociate option to mark
that reference is only temporary, 2014-10-14), "git clone" gained a
--dissociate option that makes --reference reuse objects from the
reference repository at clone time but copy them over instead of
using the reference as an alternate. This is more straightforward to
use than plain --reference, at the cost of higher disk usage.
Introduce a --dissociate to "repo init" that brings the same benefits
to repo. The option is simply passed on to "git clone".
Change-Id: Ib50a549eb71e0a2b3e234aea57537923962a80d4
Since gitiles recommends using # headers over ---/=== underlines,
change the manifest-format.md over and all our help texts.
Change-Id: I96391d41fba769e9f26870d497cf7cf01c8d8ab3
pylint is not used since bb5b1a0. The pyflakes cleanup mentioned in that
commit has not been done, but given that this project is no longer being
actively developed I don't think it's worth spending time doing it.
Leaving the pylint suppressions causes confusion because it leads people
to think that we are still using pylint.
Change-Id: If7d9f280a0f408c780f15915ffdb80579ae21f69
Commit 27226e742d introduced a warning if
"repo" is not part of the bootstrapped REPO_URL. However, that check was
done too early, directly after the call to _Clone. As the _Clone function
does not actually clone but it only initializes and fetches, the check
needs to be moved to after the call to _Checkout.
To reproduce, call
repo init --no-clone-bundle --repo-branch=master -u https://android.googlesource.com/platform/manifest
which will currently always show the (bogus) warning message. With this
fix, the warning will only be shown if "repo" indeed does not exist.
While at it, also slightly improve the code by using os.path.join().
Change-Id: Ied89e24231addabab6075005065748df1ffa74c4
While on Linux stderr is unbuffered, it is buffered on Windows. Always
flush stderr on Windows to ensure any error messages appear in the right
order to ease diagnosing.
Change-Id: I37300e384ecd3a51a321a48818f0114d6f3357a0
Desktops and servers tend to have no power sensor, thus on_ac_power returns
255 ("unknown"). Thus, let's take any answer other than 1 ("battery") as
no contraindication to run gc.
If that tool returns "unknown", there's no point in querying other sources
as it already queried them, and is smarter than us (can handle multiple
adapters).
Reported by: Xin Li <delphij@google.com>
Signed-off-by: Adam Borowski <kilobyte@angband.pl>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
(cherry picked from git.git commit 781262c5e7ad4a7813c528803117ed0d2e8c5172)
Signed-off-by: Fredrik Roubert <roubert@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Change-Id: I51fe2eb1eb879492a61e8e09c86ee34d049036c1
It's convenient to set upstream for all projects in a manifest instead of
repeating the same value for each project.
Change-Id: I946b1de4efb01b351c332dfad108fa7d4f443cba
The hooks are run from the top of the manifest checkout, not from the
individual git repos. It's up to individual hooks to chdir as needed.
Change-Id: I53325e0c3dcaa9c250b02b223e78d238d2cbd36d
When someone does "repo download -c <project> <change>"
without specifying a patch number, by default patch 1 is
downloaded. An alternative is to look for the latest patch
and download the same when no explicit patch is given.
This commit does the same by identifying the latest patch
using "git ls-remote".
Change-Id: Ia5fa7364415f53a3d9436df4643e38f3c90ded58
This change adds support for the 'revision' attribute in
<extend-project>. This avoids the need to perform a <remove-project>
followed by a <project> in local manifests.
Change-Id: Id2834fcfc1ae0d74b3347bed3618f250bf696b1f
Allow the 'default' and 'project' element in the manifest
file to apply "--no-tags" option equivalent.
Change-Id: I7e0f8c17a0e25cca744d45df049076d203c52ff5
Signed-off-by: YOUNG HO CHA <ganadist@gmail.com>
Put the correctly-expanded relative paths in objects/info/alternates.
From gitrepository-layout(5), this path should be "relative to the
object database, not to the repository".
Change-Id: I7b2027ae23cf7d367b80f5a187603c4cbacdb2de
Issue: when subproject url is a relative in .gitmodules
repo tool cannot handle this and cause:
"fatal: '***' does not appear to be a git repository
fatal: Could not read from remote repository."
issue.
Signed-off-by: Shouheng Zhang <shouheng.zhang@intel.com>
Change-Id: I2a24c291ea0074ba13a740b32a11c0c25975e72b
If a user executes:
repo forall -c echo $REPO_PROJECT
then $REPO_NAME is expanded by the user's shell first, and passed
as $1 to the shell that executes echo. This will either result in
no output, or output of whatever REPO_NAME is set to in the user's
shell. Either way, this is an unexpected result.
The correct way to do it is:
repo forall -c 'echo $REPO_PROJECT'
such that $REPO_NAME is passed in to the shell literally, and then
expanded to the value set in the environment that was passed to
the shell.
Update the documentation to make this clearer.
Change-Id: I713caee914172ad8d8f0fafacd27026502436f0d
Make it explicit that the ssh wrapper we use for control master
support accepts OpenSSH-compatible command line arguments instead of
asking Git to guess.
The GIT_SSH_VARIANT setting was introduced in Git v2.13.0-rc0~3^2~2
(2017-02-01) as a more reliable detection method than relying on the
ssh command name. Fortunately the default variant was 'ssh' (i.e.,
OpenSSH-compatible) so this wasn't initially required.
Now Git wants to start using more OpenSSH features
(-o SendEnv=GIT_PROTOCOL), and in order to do so its ssh variant
detection will need to be tweaked. Set GIT_SSH_VARIANT explicitly
so this helper can continue to work regardless of how Git modifies
its autodetection.
Reported-by: William Yan <wyan@google.com>
Change-Id: I6bf2c53b4eb5303a429eae6cb68e0a5ccce89064
This change allows setting the EDITOR env. variable to point to a
program location that contains quotes and spaces.
For example:
> set EDITOR="C:\Program Files (x86)\Notepad++\notepad++.exe" -multiInst -nosession
> repo upload
Change-Id: Ic95b00f7443982b1956a2992d0220e50b1cf6bbb
os.remove raises an exception when deleting read-only files on
Windows. Replace all calls with calls to platform_utils.remove,
which deals with deals with that issue.
Change-Id: I4dc9e0c9a36b4238880520c69f5075eca40f3e66
Windows does not support pipe|fork, but we can simulate by creating
the pager as a child process, redirecting stdout/in/err appropriately
and then waiting for the child process to terminate after we are
done executing the repo command.
Change-Id: I5dd2bdeb4095e4d93bc678802e53c6d4eda0235b
Without this change, '.git\HEAD' files, for examples, are sometime
read incorrectly resulting in the current branch to be reset to
"master" when running a "repo init -b xxx" on an already initialized
repository.
Change-Id: I48c7ef85ff81626edf156914329a560e14252f2a
* changes:
Port os.rename calls to work on Windows
Workaround shutil.rmtree limitation on Windows
Add support for creating symbolic links on Windows
Make "git command" and "forall" work on Windows
When starting a branch, branch.merge is set to project revision unless
the revision is a SHA1. In that case, branch.merge is set to dest_branch
if defined or manifest default revision otherwise. This special handling
allows repo upload to work when the project revision is a SHA1.
Extend the special handling to also happen when the project revision
is a tag value or a change value so that repo upload will work in those
case as well.
Change-Id: Iff81ece40e770cd02535e80dcb023564d42dcf47
The output indicates that fetching happens even when it is skipped.
To avoid confusion, print a message when fetching is skipped for
an immutable ref so that the user knows when and why a fetch is skipped.
Change-Id: Id6e4812cebc5e57d379feb76a9d034af0b93043b
This option allow to bypass verification ssl certification while
establishing connection with Gerrit to upload review.
Change-Id: If2e15f5a273c18a700eb5093ca8a4d5a4cbf80cd
This reverts commit d88f53e2b9. I merged
it too hastily without paying enough attention to compatibility with
released Gerrit versions.
Change-Id: I4028d4737df1255f11e217da183a19a010597d5b
Considering that some users might expect changes created with
'-d' option are not public. Private changes may be a better
choice here than work-in-progress changes.
Change-Id: I46a8fb9ae38beb41cf96d6abe82bea6db2439669
This change adds options for git-repo tool to support private
changes and work-in-progress changes.
Change-Id: I343491f5949f06f1580d53f9cc0dee2dca09130f
$ git ls-files | grep py$ | xargs pyflakes
subcmds/stage.py:101: list comprehension redefines 'p' from line 63
subcmds/sync.py:784: list comprehension redefines 'p' from line 664
subcmds/upload.py:467: list comprehension redefines 'avail' from line 454
Change-Id: Ia65d1a72ed185ab3357e1a91ed4450c719e75a7c
The submodule argument to Sync_LocalHalf was missing in
MetaBranchSwitch, causing submodules not to get synced when the
-b/--manifest-branch argument to init is used.
Change-Id: Ie86d271abac2020725770be36ead83be3326e64b
Signed-off-by: Martin Kelly <mkelly@xevo.com>
With --force-broken it continue to fetch other projects but nothing
is added in directory because it abort some lines later.
Change-Id: I32c4a4619b3028893dc4f98e8d4e5bc5c09adb27
os.rename fails on Windows if the destination exists, so replace
os.rename to platform_utils.rename which handles the platform
differences.
Change-Id: I15a86f10f65eedee5b003b80f88a0c28a3e1aa48
By default, shutil.rmtree raises an exception when deleting readonly
files on Windows.
Replace all shutil.rmtree with platform_utils.rmtree, which adds an
error handler to make files read-write when they can't be deleted.
Change-Id: I9cfea9a7b3703fb16a82cf69331540c2c179ed53
Replace all calls to os.symlink with platform_utils.symlink.
The Windows implementation calls into the CreateSymbolicLinkW Win32
API, as os.symlink is not supported.
Separate the Win32 API definitions into a separate module
platform_utils_win32 for clarity.
Change-Id: I0714c598664c2df93383734e609d948692c17ec5
Python on Windows does not support non blocking file operations.
To workaround this issue, we instead use Threads and a Queue to
simulate non-blocking calls. This is happens only when running
with the native Windows version of Python, meaning Linux and Cygwin
are not affected by this change.
Change-Id: I4ce23827b096c5138f67a85c721f58a12279bb6f
See http://stackoverflow.com/questions/7004687/os-exec-on-windows:
execv on Windows does not behave as on Linux, i.e. a new process is
spawned and the parent process terminates right away, which makes the
shell prompt come back too soon.
Change-Id: I1f8d23208765988629f081e9b949c67cf71c08ae
Project names are stored as path using the '/' file separator, and
stored in a dictionary as keys.
Change-Id: Ide40dfe840958ac0d46caae5f77f1a49d71c9d90
The documentation of the XML file format contains DTD which contains
definition of all allowed elements and attributes. The "include" element
is defined but it's not referenced from the top-level "manifest"
element.
This patch is adding the "include" element into the list of elements of
the top-level "manifest" element.
Change-Id: I33beb8ef2846bbf42ffd42e6ae6888828566d604
This is required for setups, where Gerrit access using ssh is only available
for some networks.
For network without ssh access, repo will get ssh_info from Gerrit and
use ssh for communications - which will fail. To support this setup
we need to have an option to ignore the ssh_info provided by Gerrit and
use http(s).
Using git insteadOf as alternative results in the inability to add
reviewers using "repo upload --re=...", since the syntax of adding
reviewers differs for ssh and https. repo is assuming an ssh
connection and uses "git push --receive-pack=...", which will fail
since git silently uses https for push operation. repo must be aware
that https is used so it uses "git push remote ...:refs/for/...%r=..."
for upload.
Change-Id: Idd83baef0fb26ffcc9ac65e204b68d323ce177a1
As reported by pyflakes:
subcmds/abandon.py:84: undefined name 'p'
The name of the variable should be 'proj'.
Change-Id: Ic09eb92e8db6b510e99efce010bd0bb094d7cbfe
repo sync can sync submodules via the --fetch-submodules option.
However, if the manifest repo has submodules, those will not be synced.
Having submodules in the manifest repo -- while not commonly done -- can
be useful for inheriting a manifest from another project using <include>
and layering changes on top of it. In this way, you can avoid having to
deal with merge conflicts between your own manifests and the other
project's manifests (for example, if you're managing an Android fork).
Add a --submodule option to init that automatically syncs the submodules
in the manifest repo whenever the manifest repo changes.
Change-Id: I45d34f04517774c1462d7f233f482d1d81a332a8
Signed-off-by: Martin Kelly <mkelly@xevo.com>
Previously, this would always have exited with 1 on Windows, causing "git
gc --auto" to abort. Fix this by adding support for Windows.
Change-Id: Ie519b366a11b6b18b2d465e892e738de3f4bbc99
repo can be configured to download from any number of remote git repos.
However when one fails repo doesn't report which one. Example:
Fatal: remote error: Daily ls-remote rate limit exceeded for IP xx.xx.xx.xx
TEST=repo init -q -u https://chromium.googlesource.com/chromiumos/manifest.git
# Apply patch in ./.repo/repo/
# Simulate a git remote error:
sed -i -e 's#chromiumos/docs#chromiumos/XXdocs#' .repo/manifests/full.xml
repo sync --quiet --force-sync docs
# error message now shows the remote URL
Optional test tip: reduce the time.sleep(random(...)) in ./.repo/repo/project.py
Change-Id: I4509383b6a43a8e66064778e8ed612d8a735c8b6
If you don't know that the url to git-repo itself can be overridden via
REPO_URL, it's hard to debug cases where REPO_URL is accidentally set to
another repository, e.g. inside a Jenkins CI job. What makes is even
harder is that the ".repo/repo" directory gets silently removed in such
cases as verifications fails, which makes it impossible to look at the
cloned files to understand the problem.
To better protect against such an issue, warn if the cloned git-repo
repository does not contain a top-level "repo" file, and state that the
".repo/repo" directory will be removed in case of a clone failure.
Change-Id: I697b4999205a5967910c0237772ccaada01e74d4
Currently, if direct fetch of a sha1 is not supported by git server and
depth option is used, we fallback on syncing the upstream branch by
ignoring depth option.
This fallback doesn't work in next 2 cases:
(1) upstream attribute is not specified in manifest
(2) depth option is passed to repo init command
(not with clone-depth attribute in manifest)
This commit do the following:
- fixes (1) by updating condition used to apply fallback
first we retry with depth set to None, then by syncing all branches
- fixes (2) by passing depth as argument of _RemoteFetch() method
thus, its value is not set again to depth value passed to repo init
command when applying fallback
Change-Id: Ifd6fffafc49ba229df624b0d7b64c83d47619d17
repo already special-cases sso:// URLs to behave similarly to https://
and rpc:// elsewhere in repo, but it forgot to do so here.
Noticed when trying to use relative URLs in a manifest obtained using
an sso:// URL.
Change-Id: Ia11469a09bbd6e444dbc4f22c82f9bbe9f5fd083
A recent backward incompatible change created confusion and loss of
productivity and highlighted the very limited amount of information
provided when repo sync fails; merely recommending to --force-sync
and blow-up git repos without any hint as to why. The addition of
this basic _error(...) call would have provided a clue and will in
the future.
BUG=Issue 232
TEST=simulate a breakage similar to the ones reported at
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-dev/2-0oCy_CX5s
cd .repo/projects/src/third_party/libapps.git/
file info; rm info; ln -s wronglink info
cd -
repo sync src/third_party/libapps/
# error message now shows the failure
Change-Id: Idd2f177a096f1ad686caa8c67cb361d594ccaa57
The --quiet option reduces the output to just
a list of projects with modified workspaces (and
orphans if -o is specified)
A common use case is when performing a full-workspace
merge. The integrator will kick-off a merge via:
repo forall -c git merge <some tag>
And then produce a short list of conflicted projects via:
repo status -q
The integrator can then iteratively fix and clean up all conficted
components. The merge is complete when:
repo status -q
returns no output.
Change-Id: Ibbba8713eac35befd8287c95948874e23fd5c7e2
when you want to delete all local branches, you should be find
all branches' name, and type them behind 'repo abandon' command.
Usage:
repo abandon --all [<project>...]
Change-Id: I4d391f37fb9d89b8095488c585468eafc1a35f31
When repo syncs a manifest that utilizes multiple branches
in the same project, then the sync will use an extra
thread for each "duplicate". For example, if
the manifest includes the project "foo" and "bar"
twice, then "repo sync -jN" will fetch with N+2 threads.
This is caused by _FetchHelper() releasing the thread semaphore
object each time it's called, even though _FetchProjectList()
may call this function multiple times within the scope of a
single thread.
Fix by moving the thread semaphore release to
_FetchProjectList(), which is only called once per thread
instance.
Change-Id: I1da78b145e09524d40457db5ca5c37d315432bd8
When there's a symlink to a directory, os.walk still lists the symlink
in dirs, even if it isn't configured to follow symlinks. This will fail
the listdirs check if the symlink is broken (either before or during the
cleanup). So instead, check for directory symlinks and remove them using
os.remove.
Bug: Issue 231
Change-Id: I0ec45a26be566613a4a39bf694a3d9c6328481c2
When there are nested projects in a manifest, like on AOSP right now:
<project path="build" name="platform/build" />
<project path="build/blueprint" name="platform/build/blueprint" />
<project path="build/kati" name="platform/build/kati" />
<project path="build/soong" name="platform/build/soong" />
And the top "build" project is removed (or renamed to remove the
nesting), repo just wipes away everything under build/ and re-creates
the projects that are still there. But it only checks to see if the
build/ project is dirty, so if there are dirty files in a nested
project, they'll just be blown away, and a fresh worktree checked out.
Instead, behave similarly to how `git clean -dxf` behaves and preserve
any subdirectories that have git repositories in them. This isn't as
strict as git -- it does not check to see if the '.git' entry is a
readable gitdir, just whether an entry named '.git' exists.
If it encounters any errors removing files, we'll print them all out to
stderr and tell the user that we were unable to clean up the obsolete
project, that they should clean it up manually, then sync again.
Change-Id: I2f6a7dd205a8e0b7590ca5369e9b0ba21d5a6f77
Allow the 'remote' element in the manifest file to define an optional
'pushurl' attribute which is passed into the .git/config file.
Change-Id: If342d299d371374aedc4440645798888869c9714
Signed-off-by: Steve Rae <steve.rae@raedomain.com>
pylint reports a lot of warnings, but many of them are false positive,
and it's difficult to configure it. It also seems that for some reason
the included config file is not working well with the latest version.
Update the documentation to recommend using pyflakes and flake8 instead
of pylint. Remove the pylint config and add a basic flake8 config with
minimum settings:
- Maximum line length 80 columns
- Ignore warnings about indentation (repo uses 2 rather than expected 4)
- Ignore warnings about import placement
In this commit no code cleanup is done, and it's expected that most of
the files will throw up quite a few warnings, at least for flake8. These
can be cleaned up in follow-up commits.
The existing pylint suppression comments are left as-is. These will be
helpful when cleaning up pyflakes warnings later.
Change-Id: I2f7cb4340266ed07cc973ca6483b8f09d66a765b
The shared object stores confuse git and make it throw away objects which are
still in use. We'll avoid that problem by disabling automatic pruning on those
projects, but there's nothing preventing a user from changing the config back
or pruning a repository manually.
BUG=chromium:375945
TEST=Ran repo sync on fresh ChromeOS checkout, starting with a branch of repo
with this change. Verified that the kernel projects and no others were
identified as having shared object stores, and that repo successfully disabled
automatic pruning in their configs. Re-enabled pruning and ran repo sync just
on one of the kernel directories. Verified that pruning was re-disabled as a
result.
Change-Id: I728ed5b06f0087aeb5a23ba8f5410a7cd10af5b0
The requirement to explicitly specify the local project when starting
a new repo branch is somewhat counter intuitive.
This patch uses the current directory's git tree as the default
project.
Tested by running
'repo start <name>'
observed that the result is the same as if running
'repo start <name> .'
Change-Id: If106caa801b4cd5ba70dbe8354a227d59f100aa3
When nothing is pending, most of this code is already short-circuited.
Hoist the single check up to make this more obvious/slightly faster.
Change-Id: Iec3a7e08eacd23a7c5f964900d5776bf5252c804
if a gerrit server has ssh and https access enabled, but user access
(for some users) is limited to https, 'repo upload' command will fail
for them.
Gerrit returns a ssh configuration (gerrit/ssh_info), that does not work
for users limited to https.
With this patch repo will test, if the returned ssh configuration from
gerrit/ssh_info is working. if not, it will fall back to https for upload.
Change-Id: If98f472e994f350bf71f35610cd649b163f1ab33
this check can only detect errors that happen within 1 sec after launching
ssh. But this is typically enough to catch configuration issues like
'connection refused' or 'authentication failed'.
Change-Id: I00b6f62d4c2889b1faa6c820e49a198554c92795
Instead of
Do you want to allow this script to run (yes/yes-never-ask-again/NO)? (yes/always/NO)?
ask
Do you want to allow this script to run (yes/always/NO)?
Change-Id: I5f5a2d0e88086a8d85e54fb8623a62d74a20956a
Signed-off-by: Jonathan Nieder <jrn@google.com>
There have been a number of changes in the repo wrapper since the last
increment that was done in fee390ee:
- 9711a98 init: Add --no-clone-bundle option
- 631d0ec Support non-ASCII GNUPGHOME environment variable
- 4088eb4 repo: Cleaned up pylint/pep8 violations
- 5553628 repo: Add check of REPO_URL env variable
- 745b4ad Fix gitc-init behavior
- d3ddcdb Ignore clone.bundle on HTTP 501, i.e. Not Implemented
Change-Id: I3f763ef0ec2df2d726dff429021b48ad474148f1
The constant prompting when registered hooks change can be tedious and
has a large multiplication factor when the project is large (e.g. the
AOSP). It gets worse as people want to write more checks, hooks, docs,
and tests (or fix bugs), but every CL that goes in will trigger a new
prompt to approve.
Let's tweak our trust model when it comes to hooks. Since people start
off by calling `repo init` with a URL to a manifest, and that manifest
defines all the hooks, anchor trust in that. This requires that we get
the manifest over a trusted link (e.g. https or ssh) so that it can't
be MITM-ed. If the user chooses to use an untrusted link (e.g. git or
http), then we'll fallback to the existing hash based approval.
Bug: Issue 226
Change-Id: I77be9e4397383f264fcdaefb582e345ea4069a13
During sync, repo runs `git read-tree --reset -u -v HEAD` which causes
git-lfs's smudge filter to run. However this fails because git-lfs does
not work with bare repositories.
Add lfs.filter configuration to the project config as suggested in the
comments on the upstream git-lfs client issue [1]. This prevents the
smudge filter from running, and the sync completes successfully.
For any projects that have LFS objects, `git lfs pull` must be executed.
[1] https://github.com/github/git-lfs/issues/1422
Bug: Issue 224
Change-Id: I091ff37998131e2e6bbc59aa37ee352fe12d7fcd
Re-ordered to first create the symlink before checking the source
file and remove the destination if the source does not exists.
Change-Id: Iae923ba2ef0ba5a8dc1b8e42d8cc3f3708f773af
If upstream string is empty, current_branch_only variable will be assigned
to an empty string.
This is not what we expect here as this variable is a boolean.
Change-Id: Ibba935e25a74c2be1e50c88b4b403cf394ba365e
When sync-s="true" option is used, the checkout of a submodule will try
to use the revision attribute of the parent project.
If this revision is a named reference, the checkout will fail if there
is no reference with this name in the submodule.
The proposed solution is to use the git commit id as revisionExpr for
submodules.
Change-Id: Ie8390a11957fd6a9c61289c6861d13cb3fa11678
Adds the hook-scripts to .gitattributes due to the shell-scripts not
liking CRLF which they will get if a user sets 'autocrlf = true'
in their global gitconfig.
Further, since the python interpreter can handle either CRLF or LF,
python-scripts specific line-ending rules have been removed.
Change-Id: I2d6bfd491b2f626b9ca93c40a3a7f2cfba6c54f0
This ignores whitespaces errors, which we have quite a few of in argument
lists, for example:
************* Module git_config
C:209, 0: No space allowed around keyword argument assignment
def HasSection(self, section, subsection = ''):
^ (bad-whitespace)
C:320, 0: No space allowed around keyword argument assignment
capture_stdout = True,
^ (bad-whitespace)
C:321, 0: No space allowed around keyword argument assignment
capture_stderr = True)
^ (bad-whitespace)
C:427, 0: Exactly one space required after comma
'-o','ControlPath %s' % ssh_sock(),
^ (bad-whitespace)
C:436, 0: Exactly one space required after comma
check_command = command_base + ['-O','check']
^ (bad-whitespace)
C:464, 0: Exactly one space required after comma
% (host,port, str(e)), file=sys.stderr)
^ (bad-whitespace)
C:707, 0: No space allowed around keyword argument assignment
return self._config.GetString(key, all_keys = all_keys)
^ (bad-whitespace)
C:759, 0: No space allowed around keyword argument assignment
return self._config.GetString(key, all_keys = all_keys)
^ (bad-whitespace)
Change-Id: Ia8f154f6741ce609787551f65877d7584c457903
Signed-off-by: Stefan Beller <sbeller@google.com>
When the alias attribute is set for a remote, the RemoteSpec attached to
a Project only contains the alias name used by git, not the original
name used in the manifest. But that's not enough information to
reconstruct the manifest, so save off the original manifest name as
another RemoteSpec parameter, only used to write the manifest out.
Bug: Issue 181
Bug: Issue 219
Change-Id: Id7417dfd6ce5572e4e5fe14f22924fdf088ca4f3
Mention that the RPC endpoints are used when running repo
sync with the --smart-sync and --smart-tag options
Change-Id: I4b0b82e8b714fe923a5b325a6135f0128bf636ff
The repo script allows a manifest to specify a '.' as the path the
top-level directory, which co-locates the .git and .repo directories,
and places files from the git repository at the top-level:
<project name="proj_name" path="." />
<project name="sierra.other.git" path="other" />
Most commands work correctly with this setup. Some commands, however,
fail to find the project. For instance, 'repo sync' works, and 'repo sync .'
works in a sub-project ('other' in this case) but 'repo sync .' in the
top-level directory fails with the error:
error: project . not found
There are two reasons for this:
1. The self.worktree attribute of the Project object is not normalized,
so with a '.' for path its value would be '/my/project/root/.'. This is
fine when used as a path, since it's the same path as '/my/project/root',
but when used in a string comparison it fails. This commit applies
os.path.normpath() to that value before storing it.
2. The _GetProjectByPath method in command.py was not checking the path
against manifest.topdir, so even once it was normalized the project was
not found. This commit adds a check against manifest.topdir if the
loop drops out without finding a project.
Change-Id: Ic84d053f1bbb5a357cad566805d5a326ae8246d2
We weren't copying these lists, so the += was actually changing the
underlying lists.
When a new project was added to the manifest, we run _CheckDirReference
against the manifest project with share_refs=True, which added the
working_tree_* to the shareable_* lists. Then, when we load the new
manifest and create the new project, it uses the lists that already
contain the working_tree_* files, even though we passed
share_refs=False.
This happens reliably under the above conditions, but doesn't seem to
happen when syncing a fresh tree. So we've got a mixture of links that
may need to be cleaned up later. This patch will just stop it from
happening in the future.
Change-Id: Ib7935bfad78af1e494a75e55134ec829f13c2a41
Make it possible to exclude projects using regex/wildcard.
The syntax is similar to that of the -r option, e.g.:
repo forall -i ^platform/ ^device/ -c 'echo $REPO_PROJECT'
Change-Id: Id250de5665152228c044c79337d3ac15b5696484
A common design pattern is to use __file__ to find the location of the
active python module to assist in output or loading of related assets.
The current hook systems runs the pre-upload.py hook in a context w/out
that set leading to runtime errors:
$ repo upload --cbr .
ERROR: Traceback (most recent call last):
File ".../repo/project.py", line 481, in _ExecuteHook
self._script_fullpath, 'exec'), context)
File ".../repohooks/pre-upload.py", line 32, in <module>
path = os.path.dirname(os.path.realpath(__file__))
NameError: name '__file__' is not defined
Define this variable in this context so code can safely use it.
Change-Id: If6331312445fa61d9351b59f83abcc1c99ae6748
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a few violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the command.py script.
This script now has a pylint score of 10.0.
Change-Id: Ibb35fa9af0e0b9b40e02ae043682b3af23286748
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a number of violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the project.py script.
This script now has a pylint score of 10.0, and no violations reported
by pep8.
Change-Id: I1462fd84f5b6b4c0dc893052671373e7ffd838f1
It was only displaying 'Project list error: GitError()'
without any useful info about the project nor the error
Change-Id: Iad66cbaa03cad1053b5ae9ecc90d7772aa42ac13
This commit adds additional instructions on getting patches submitted,
based on my recent experience doing so.
Change-Id: I8e0d37d316214cc9a39383414773aad181f83f18
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a number of violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the repo script.
First I ran this to clean up the formatting:
autopep8 --max-line-length=80 --indent-size 2 repo
Following that the following violations remained:
% pylint --rcfile=.pylintrc repo
************* Module repo
W:220,21: Redefining name 'init_optparse' from outer scope (line 156)
(redefined-outer-name)
W:482, 2: No exception type(s) specified (bare-except)
C:704, 0: Old-style class defined. (old-style-class)
For line 220, the parameter to _GitcInitOptions was renamed so as not to
mask the init_optparse global.
For line 482, a pylint directive was added to disable the bare-execpt
violation for just that line.
For line 704, the _Options class was changed to subclass object.
Additionally, the comments at lines 107-113 were spaced out to line up
with the comment at line 112 that autopep8 moved.
This script now has a pylint score of 10.0
Change-Id: I779b66eb6b061a195d3c4372b99dec1b6d2a214f
We want to be able to run repo on a system that is not connected to
the Internet and cannot access https://gerrit.googlesource.com. We
can put a clone of that repos there, but would prefer to use the
stable version of the repo script instead of a locally modified
version.
This commit adds a check for the REPO_URL environment variable. If
that is set and not empty its value will be set in the REPO_URL
global in repo. Otherwise the standard path will be used.
Change-Id: I0616f5f81ef75f3463b73623b892cb5eed6bb7ba
As soon as we wrote the gitc manifest, the folder for that repo became
empty, causing the next GetProjects lookup to fail. Reorder the
GetProjects calls so that they all happen while we still have the
repository contents available.
If you were already in a subdir, for cases like 'repo start <branch> .',
this would still fail, since the working directory would disappear out
from under you. That's fine most of the time, since we shouldn't be
doing operations based on the local directory, but git has a realpath
function that tries to restore CWD by chdir'ing back to it. So if the
working directory no longer exists, chdir to the topdir before
continuing.
Change-Id: Ibdf6cd37ff6e5a5f8338347c3919175491f7166f
Some teams have a continuous build server that would mark certain
manifest green and safe to sync to. Then team members could repo
sync to that particular manifest file and make sure they always
sync to a green build. But if she/he has some local changes and
wants to rebase, currently it would be a manual process to find the
correct version to rebase onto. This patch helps with that use
case by automating the process to rebase onto the currently synced
manifest version.
Change-Id: I847c9eb6addf7f84fd3f5594fbf8c0bcc103f9a5
We don't really use HEAD much in the bare git repositories, but there
have been reports of errors in git-symbolic-ref:
symbolic-ref: fatal: Refusing to point HEAD outside of refs/
That happen when the bare git repo is in the detached head state. It's
possible that previous operations were killed while we were pruning
branches.
Use DetachHead instead of SetHead if we're restoring the repo into a
detached head state.
Change-Id: I9062e8957bc70367d3ded399685ac026fbb421fc
When repo sync is used with -f (--force-error) and a project fails to
sync, the sync will continue but then exit with an error status.
However if -n (--network-only) is also used, the exit code is 0, even
when a project failed.
Modify the logic to make sure the sync exits with the correct status.
Bug: Issue 214
Change-Id: I0b5d97a34642c5aa3743750ef14a42c9d5743c1d
See git commit 33cfccbbf35a -- some protocols allow arbitrary command
execution as part of the URL. Instead of blindly allowing those,
whitelist the allowed URL protocols unless the user has already done so.
Bug: Issue 210
Change-Id: I6bd8e721aa5e3dab53ef28cfdc8fde33eb74ef76
If a linkfile is a broken link (destination does not exist), and it
needs to be updated, we didn't notice that it needed to be removed
first. Use lexists instead of exists to check for this condition.
Change-Id: I1f6a1f0193d3fd2b9f7a647836044997f6ab32eb
By passing --prune to the sync command, the --prune option is
given to the `git fetch`, causing refs that no longer exist on
the remote to be removed.
Change-Id: I3cedacce14276d96ac2d5aabf2d07fd05e92bc02
Adds windows as one of the allowed platforms flags.
Fixes -p foo to append 'platform-foo', instead of each letter (list.extend
expects a list and thus appends each char in the string, rather than the
string itself).
Change-Id: I73a92127ac29a32fc31b335cc54a246302904140
With gitc-init, a gitc client may be specified using '-c'. If we're
not currently in that client, we need to change directories so that
we don't affect the local checkout, and to ensure that repo is
checked out in the new client.
This also makes '-c' optional if already in a gitc client, to match
the rest of the init options.
Change-Id: Ib514ad9fd101698060ae89bb035499800897e9bd
The .gitcookies file generated by googlesource.com does not have
the header:
# (Netscape) HTTP Cookie File
which causes python's MozillaCookieJar.load to fail with the
error:
"does not look like a Netscape format cookies file"
Prepend the expected header onto the generated cookie file.
We don't bother to check if the header already exists on the
file; repeating it does not cause any problem.
Bug: Issue 207
Change-Id: I7d39720a1d36a6aae00f70691156514ebc04e579
repo gitc-delete deletes a GITC client and all the locally
saved sources. Useful for removing unnecessary clients and
recovering disk space.
Change-Id: Idf23addcea52b8713d268c34a7b37da0c5e5cd26
These won't show up as common commands in the help text unless in a GITC
client, and will refuse to execute.
Change-Id: Iffe82adcc9d6ddde9cb4b204f83ff018042bdab0
The "copyfile" element is available since 2009 and
have been used in every Android manifest; the "linkfile"
element is available since 2014.
Now it's a good time to add both to the documentation
Change-Id: Ia987edf5f69a006235fbd3f33b744e9794a6d964
Signed-off-by: Ruslan Bilovol <ruslan.bilovol@gmail.com>
This way any changes made to the main manifest are reflected in the gitc
manifest. It's also necessary to use both manifests to sync since the
information required to update the gitc manifest is actually in the repo
manifest.
This also fixes a few issues that came up when testing. notdefault
groups weren't being saved to the gitc manifest in a method that matched
'sync'. The merge branch wasn't always being set to the correct value
either.
Change-Id: I435235cb5622a048ffad0059affd32ecf71f1f5b
This way any changes made to the main manifest are reflected in the gitc
manifest. It's also necessary to use both manifests to sync since the
information required to update the gitc manifest is actually in the repo
manifest.
This also fixes a few issues that came up when testing. notdefault
groups weren't being saved to the gitc manifest in a method that matched
'sync'. The merge branch wasn't always being set to the correct value
either.
Change-Id: I5dbc850dd73a9fbd10ab2470ae4c40e46ff894de
The source or destination attributes may have changed even if the source
didn't, so we need to make sure that these are up to date.
Change-Id: I266ef3598ddda7e8c23bc9c6a049905ddc586348
If the clone.bundle is out of date, repo may be installed with an old
version. It will upgrade with the next sync a day later, or when "repo
selfupdate" is run.
This behavior was added to normal project downloads, but was never added
to the repo launcher.
Change-Id: Ib04bef3a658c98fe1b6c53b3e8d0067165a5e3f7
Updates the repo launcher and gitc_utils to pull the manifest
directory location out of the gitc config file.
Change-Id: Id08381b8a7d61962093d5cddcb3ff6afbb13004b
The GITC filesystem does not understand relative URLs for remotes,
so now if a remote uses a relative URL, it will be be expanded to
be relative to the manifest URL.
Change-Id: Ie1210758560aeb1934da3f71496aaf19c2728214
Add repo start support for GITC checkouts. If the user is in
the GITC FS view, they can now run repo start to check out
the sources and create a new working branch.
When "repo start" is called on a GITC project, the revision
tag is set to an empty string and saved in a new tag:
old-revision. This tells the GITC filesystem to display the
local copy of the sources when being viewed. The local copy
is created by pulling the project sources and the new branch
is created based off the original project revision.
Updated main.py to setup each command's gitc_manifest when
appropriate.
Updated repo sync's logic to sync opened projects and
updating the GITC manifest file for the rest.
Change-Id: I7e4809d1c4fc43c69b26f2f1bebe45aab0cae628
If a hook file has been modified locally, it will not be replaced.
Improve the message to make this clearer.
Also change it from an error to a warning.
Change-Id: I62c635390f24d2868db17717c247861b0381c99f
Use the _error method instead of directly calling `print`.
Also add a new _warn convenience method.
Change-Id: Ia332c14ef8d9d1fe2df128dbf36b5521802ccdf1
Don't emit a message when the netrc file doesn't exist or couldn't
be opened.
Instead of trying to unpack the result of info.authenticators() and
catching the resulting TypeError when it's None, first store it to
a local and only unpack it if it has a value.
Also remove an unused import.
Change-Id: I5c404d91e48c261c1ab850c3e5f040c4f4c235cb
Use the same cookies and proxy that git traffic goes through for
persistent-http[s] to support authentication for smart-sync.
Change-Id: I20f4a281c259053a5a4fdbc48b1bca48e781c692
Passing the force_sync variable into the string formatting results in
the message:
"Retrying clone after deleting None"
or
"Retrying clone after deleting True".
Pass the name of the git directory instead.
Also, move the print inside the if-block so it's only displayed
when the retry is actually going to be attempted.
Change-Id: I76d9ecc176cecee4ad512d13e9d1f6bd36aacbbb
Add repo sync support for GITC checkouts. If the user is in the
GITC client directory they can still pull the sources as normal
if they pass in the --force-gitc argument. Otherwise the user
should call repo sync in the GITC view to update the user's
remote view. (This works because .repo in the GITC view will
link to .repo in the client config directory.)
Part of the support for this change is the refactoring of GITC
related code into gitc_utils.py.
Change-Id: I2636aaa50b450b6f091309db8dd0e8f4dbdad579
This argument wasn't being copied, which caused syncs from generated
manifests to pull down too much of the git history.
Change-Id: I269bab788d4557267c081628b3f8c6aec7744e81
Adds the new gitc-init command to set up a GITC client. Gitc-init
sets up the client directory and calls repo init within it. Once
the repo is initialized, then generates a GITC manifest file
by using git ls-remote on each project and retrieving the HEAD SHA
to use as the revision attribute.
Gitc-init inherits from and has all the options as repo init.
Change-Id: Icd7e47e90eab752a77de7c80ebc98cfe16bf6de3
For projects that have been cloned outside of the repo command (or
cloned a long time ago), commit abaa7f312f
introduced an error message to invite the user to use --force-sync.
However, due to the risk of data loss, it's useful to know which
project's git directory is being replaced before deciding whether or not
to provide --force-sync.
This change updates the exception's associated value to include the
project's relative path and explain to the user how they can resolve the
issue. A previous version of this commit used the project name. However,
for projects that have multiple work trees, the name can be ambiguous,
while the path clearly identifies which git directory will be replaced.
Change-Id: If717e66fda4d19accc0a8e889a91f4cd4ff14dff
The existing code here makes sure that switching clone-depth from on to
off actually causes the history to be fully restored. Unfortunately, it
does this by fetching the full history every time the fetch spec
changes. Switching between two clone-depth="1" branches will fetch far
more than the top commit.
Instead, when not using clone-depth, pass --depth=2147483647 to git
fetch so that it ensures that we have the entire history. That is
slightly less efficient, so limit it to only when there are shallow
objects in the project by checking for the existance of the 'shallow'
file.
Change-Id: Iee0cfc9c6992c208344b1d9123769992412db67b
Previously repo would only print the failing project path if
Sync_NetworkHalf returned false/empty, but if it threw an
exception the print() was never called.
Change-Id: I58c41de43930df5e34b21561c205e062a72e290f
This fixes these errors:
...
File ".repo/repo/project.py", line 2371, in _ReferenceGitDir
os.symlink(os.path.relpath(src, os.path.dirname(dst)), dst)
OSError: [Errno 17] File exists
Which was happening for checkouts that were created before v1.12.8, when
project-objects was created. Nothing had yet been forcing these
checkouts to use project-objects, until the recent verification changes.
In this OSError case, we already created the symlink, so src == dst, and
the directory did not exist. This caused us to run os.makedirs the
os.symlink on the same file.
dst really should be the file in gitdir, not the target of that symlink
if it exists. So just use realpath for the dotgit portion of the path.
Change-Id: Iff5396a2093de91029c42cf38aa57131fd22981c
Enable operating against groups of repositories. As it stands, it isn't
compatible with `-r/--regex`.
`repo forall -g groupname -c pwd` will run `pwd` for all projects in
groupname.
`repo forall -g thisgroup,-butnotthisone -c pwd` will run `pwd` for all
projects in `thisgroup` but not `butnotthisone`.
`repo list -g groupname -n` will list all the names of repos in
`groupname`.
Change-Id: Ia75c50ce52541d1c8cea2874b20a4db2e0e54960
In some cases, a user may wish to continue with a sync even though
it would require overwriting an existing git directory. This behavior
is not safe as a default because it could result in the loss of some
user data, but as an optional flag it allows the user more flexibility.
To support this, add a --force-sync flag to the sync command that will
attempt to overwrite the existing git dir if it is specified and the
existing git dir points to the wrong obj dir.
Change-Id: Ieddda8ad54e264a1eb4a9d54881dd6ebc8a03833
repo info will crash when using a manifest with no default element despite
default being an optional element. Output nothing for "Manifest Branch" if no
default element exists (or if no default revision exists).
Change-Id: I7ebffa2408863837ba980f0ab6e593134400aea9
If _InitGitDir fails, it leaves any progress it had made on the file
system. This can cause subsequent calls to repo sync to behave
differently. This is especially evident when _CheckDirReference() fails,
since it will not be invoked when sync is retried because both the
source and destination directories already exist.
To address this, have _InitGitDir() clean up any directories it has created
if it catches an exception. Also behave the same way for _InitWorkTree().
Change-Id: Ic16bb3feea649e115b59bd44be294e89e3692aeb
For some users it is not desirable to remove refs that don't exist
on the remote server when syncing a mirror repo.
This reverts commit b4d43b9f66.
Change-Id: Ie849b66682138ef88da6cd1a5fbb27e993197dd7
The fetch logic for the case where depth is set and revision is a
SHA1 has several failure modes that are not handled well by the
current logic.
1) 'git fetch <SHA1>' requires git version >= 1.8.3
2) 'git fetch <SHA1>' can be prevented by a configuration option on the server.
3) 'git fetch --depth=<N> <refspec>' can fail to contain a SHA1 specified by
the manifest.
Each of these cases cause infinite recursion when _RemoteFetch() tries to call
itself with current_branch_only=False because current_branch_only is set to
True when depth != None.
To try to prevent the infinite recursion, we set self.clone_depth to None
before the first retry of _RemoteFetch(). This will allow the Fetch to
eventually succeed in the case where clone-depth is specified in the manifest.
A user specified depth from the init command will still recurse infinitely.
In addition, never try to fetch a SHA1 directly if the git version being used
is not at least 1.8.3.
Change-Id: I802fc17878c0929cfd63fff611633c1d3b54ecd3
* changes:
forall: use smart sync override manifest if it exists
sync: Remove smart sync override manifest when not in smart sync mode
forall: Don't try to get lrev of projects in mirror workspace
sync: Improve error message when writing smart sync manifest fails
Previously, in running the `manifest` command, we wouldn't output the
upstream if the default upstream would include the pinned sha1.
However, now that fetching refs/heads/* doesn't guarantee that we will
have the sha1, we need to always output the specified upstream branch.
Change-Id: Ib8b409a8ecd439397b38ee9649c530407797f841
If a workspace is synced with the -s or -t option, the included projects
may be different to those in the original manifest. However, when using
the forall command, the list of the projects from the original manifest
is used.
If the smart sync manifest file exists, use it to override the original
manifest.
Change-Id: Iaefcbe148d2158ac046f158d98bbd8b5a5378ce7
When syncing with the -s or -t option, a smart_sync_override.xml file
is created. This file is left in the file system when syncing again
without the -s or -t option.
Remove the smart sync override manifest, if it exists, when not using
the -s or -t option.
Change-Id: I697a0f6405205ba5f84a4d470becf7cd23c07b4b
git rev-parse fails for projects that don't have an explicit revision
specified, and don't have a branch of the same name as the default
revision. This can be the case in a workspace synced with the smart
sync (-s) or smart tag (-t) option.
Change-Id: I19bfe9fe7396170379415d85f10f6440dc6ea08f
The error message only states that writing the manifest failed.
Include the exception message, so it's easier to track down the reason
that the write failed.
Change-Id: I06e942c48a19521ba45292199519dd0a8bdb1de7
After performing the actual cherry-pick operation, the code
in cherry_pick.py opens a pipe to 'git commit -F' to rewrite the commit
message, emits the fixed-up commit msg to the pipe, then waits
for 'git commit' to complete. The child 'git' process winds up
hanging while reading from the pipe, however, since the parent
process still has it open. To fix the hang, change the parent process
to close its end of the pipe after it has emitted the message.
Change-Id: I5929371e69a5b076f09009d00d40a2c72ac8ac33
output from a process is in bytes in python3. we need
to decode it.
in Python3, bytes don't have an encode attribute. use this
to identify it.
Change-Id: I152f2ec34614131027db680ead98b53f9b321ed5
This allows a project to use globs in the linkfile src attribute. When
a glob is used in the src the dest field must be a directory. Then
_LinkFile._Link(self) calls will create symbolic links in the dest
directory to all of the entries in the src as defined by the glob
specification.
Below all of the entries in master-configs/ will have symbolic links
in <root dir>/configs directory:
<project name="helloworld.git" path="apps/helloworld">
<linkfile src="master-configs/*" dest="configs"/>
</project>
Change-Id: Idfed8fa47c83d2ca6e2b8e867731b8e2f9e2eb47
The source (target) of the symlink is specified relative to a project
within a tree, and the destination is specified relative to the top
of the tree, so it should always be possible to create a relative symlink
to the target file. Relative symlinks will allow moving an entire tree
without breaking the symlink, and copying a tree (with -p) without leaving
a symlink to the old tree.
Change-Id: I16492a8b59a137d2abe43ca78e3b212e2c835599
Pressing ctrl-c during repo sync often hangs for 30 to 45 seconds
due to the time.sleep and retry in _RemoteFetch. If git exits with
a signal, for example -2 for SIGINT triggered by ctrl-c, skip the
sleep and retry.
Change-Id: I32da12c2dcc96d9cc0b12a066e824b12ebfb52a0
If the generator that produces per-project worker arguments raises an
exception it triggers python bug http://bugs.python.org/issue8296.
Rewrite the generator expression as a generator function, and catch
Exceptions and KeyboardInterrupts to end the iteration.
Also add a pool worker initializer to disable SIGINT to prevent
KeyboardInterrupts inside multiprocessing.Pool in the worker threads
causing the same problem.
Fixes easy-to-reproduce hangs when hitting ctrl-c during
repo forall -c echo
Change-Id: Ie4a65b3e1e07a64ed6bb6ff20f3912c4326718ca
There are a set of cases that can cause the git directory in
.repo/projects to point to a directory in .repo/project-objects that
is not the one specified in the manifest. This results in a tree that
is not sane, and so should cause a failure.
In order to reproduce the failure case:
1) Sync to any manifest
2) Change the 'name' of a project to a different repository. Leave the
'path' the same.
3) Resync the modified project. The project-objects directory will not
be created, and the projects directory will remain pointed at the old
project-objects.
Change-Id: Ie6711b1c773508850c5c9f748a27ff72d65e2bf2
In 2fb6466f79 an optimisation was
added to avoid fetching from remotes if the project is fixed to
a revision and the revision is already available locally.
This causes problems for users who expect all objects to be
fetched by default.
Change the logic so that the optimized behaviour is only enabled if
an option is explicitly given to repo sync.
Change-Id: I3b2794ddd8e0071b1787e166463cd8347ca9e24f
When syncing a mirror repo, add the --prune option to the fetch
command to force removal of stale refs from the mirror.
Change-Id: I4b43b2a5c86b9915627887c16f6569066f3ab978
Previously, we used a regex that would only remove a phony string from
a url if it existed, but we recently replaced that with a slice. This
change goes back to the previous behavior.
Change-Id: I8baf527be01c4b49d45b903b31a1cd6315563d5b
Appending the branch to the fetch spec causes sync of a mirror to
fail for projects that don't have an explicit revision specified,
and don't have a branch of the same name as the default revision.
For example, a manifest defining a default revision:
<default revision="master">
having a project without an explicit revision:
<project name="path/to/project">
and not having a branch named "master", will cause repo sync to
fail for that project with the error:
Couldn't find remote ref refs/heads/master
Modify the logic to not append the branch onto the fetch spec when
syncing to a mirror.
Change-Id: I5c4457bd125519abf27abe682dea62ad708978c9
Before, a list was generated, which is why there was a massive delay.
Using a generator will allow processes to start straight away
Change-Id: Ia325b0b340cc328c08c9bcc92a6709bbdaf6a664
buflist was being used, which isn't available in Python 3.
`Execute` was using StringIO to capture the output of `PrintWorkTreeStatus`,
only to redirect it straight to stdout.
Instead, just let `PrintWorkTreeStatus` do it's own thing directly to stdout.
for handling `_FindOrphans`, we swap StringIO for a list. Nothing was done
that needed a a file like object.
Change-Id: Ibdaae137904de66a5ffb590d84203ef0fe782d8b
When running repo branch, the git merge line (in many circumstances)
is set to the revision of the project specified in the manifest. If
this is a branch name that is not fully-qualified, we will end up with
something like "merge = master" instead of "merge = refs/heads/master".
This change examines the revision if we are going to use that and
changes branch short names to fully qualified branch names.
Change-Id: Ie1be94fb8d45df8eeac44a47f729a3819a05fa81
Instead of using regex, append the netloc and relative
scheme lists with the custom scheme.
The schemes will only be appended when needed, instead
of passing X amount of regex replaces.
see http://bugs.python.org/issue18828 for more details.
Change-Id: I10d26d5ddc32e7ed04c5a412bdd6e13ec59eb70f
Switch the GitCommand program to always capture the output for stdout
and stderr. And by default print the output while running.
The options capture_stdout and capture_stderr have effectively become
options to supress the printing of stdout and stderr.
Update the 'git fetch' to use '--progress' so that the progress messages
will be displayed. git checks if the output location isatty() and if it
is not a TTY it will by default not print the progress messages.
Change-Id: Ifdae138e008f80a59195f9f43c911a1a5210ec60
| repo.partialcloneexclude | `--partial-clone-exclude` | Comma-delimited list of project names (not paths) to exclude while using [partial git clones] |
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
