The number of jobs one wants to run against the network tends to
factor differently from the number of jobs one wants to run when
checking out local projects. The former is constrained by your
internet connection & server limits while the later is constrained
by your local computer's CPU & storage I/O. People with beefier
computers probably want to keep the network/server jobs bounded a
bit lower than the local/checkout jobs.
Change-Id: Ia27ab682c62c09d244a8a1427b1c65acf0116c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302804
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We've grown a lot of options in here and it's hard to make sense of
them. Add more groups to try and make it easier to pick things out.
Change-Id: I6b9dc0e83f96137f974baf82d3fb86992b857bd2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302803
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This can take a few seconds, if not a lot more, so add a progress bar
so users understand what's going on.
Change-Id: I5b4b54c1bbb9ec18728f979521310f7087afaa5c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The current logic has a downside in that it doesn't sync to the latest
signed version available if the latest commit itself is unsigned. This
can come up when using the "main" branch as it is sometimes signed, but
often not as it's holding the latest merged commits. When people use
the main branch, it's to get early testing on versions tagged but not
yet released, and we don't want them to get stuck indefinitely on that
old version of repo.
For example, this series of events:
* "stable" is at v2.12.
* "main" is tagged with v2.13.
* early testers use --repo-rev main to get v2.13.
* new commits are merged to "main".
* "main" is tagged with v2.14.
* new commits are merged to "main".
* devs who had synced in the past to test v2.13 are stuck on v2.13.
repo sees "main" is unsigned and so doesn't try to upgrade at all.
The only way to get unwedged is to re-run `repo init --repo-rev main`,
or to manually sync once with repo verification disabled, or for us to
leave "main" signed for a while and hope devs will sync in that window.
The new logic is that whenever changes are available, we switch to the
latest signed tag. We also replace some of the duplicated verification
code in the sync command with the newer wrapper logic. This handles a
couple of important scenarios inaddition to above:
* rollback (e.g. v2.13.8 -> v2.13.7)
* do not trash uncommitted changes (in case of ad-hoc testing)
* switch tag histories (e.g. v2.13.8 -> v2.13.8-cr1)
Change-Id: I5b45ba1dd26a7c582700ee3711f303dc7538579b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300122
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
These are manually kept in sync which is a pain. Have the init
subcmd reuse the wrapper code directly.
Change-Id: Ica73211422c64377bacc9bb3b1d1a8d9d5f7f4ca
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302762
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We still support Python 3.5, so make sure it keeps working.
Change-Id: I150158a656b26de6d733316a68a2cbb8b5b99716
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299625
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We need to pass back an int, not a CompletedProcess object. Switch to
check=False so we don't throw an exception on failure -- we're already
showing pytest's stderr, and will return the non-zero status.
Change-Id: Ib0d3862a09a3963f25025f39a8e34419cf2a54df
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299624
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We don't keep this up-to-date in general, but might as well keep
the major version in sync.
Change-Id: I20908005b3b393d384da0ef9342d7c9d094550cb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299622
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This avoids GIL limitations with using threads for parallel processing.
This reworks the fetch logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.
We have to tweak the chunking logic a little here because multiprocessing
favors batching over returning immediate results when using a larger value
for chunksize. When a single job can be quite slow, this tradeoff is not
good UX.
Bug: https://crbug.com/gerrit/12389
Change-Id: I0f0512d15ad7332d1eb28aff52c29d378acc9e1d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298642
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~40sec to ~16sec with the default -j8.
The output processing does not appear to be a significant bottle
neck -- it accounts for <1sec out of the ~16sec runtime. Thus we
leave it in the main thread to simplify the code.
Change-Id: I750b72c7711b0c5d26e65d480738fbaac3a69971
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297984
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~10sec to ~4sec with the default -j8.
This only does a simple conversion over to get an easy speedup. It
is currently written to collect all results before displaying them.
If we refactored this module more, we could have it display results
as they came in.
Change-Id: I5caf4ca51df0b7f078f0db104ae5232268482c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298643
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the project exists, but the ref the manifest wants doesn't exist,
don't throw an error (and abort the process in general). This can
come up with a partially synced tree: the manifest is up-to-date,
but not all the projects have yet been synced.
Bug: https://crbug.com/gerrit/14289
Change-Id: Iba97413c476544223ffe518198c900c2193a00ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301262
Reviewed-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v
Bug: [google internal] b/183232698
Bug: https://crbug.com/gerrit/13707
$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync --use-superproject
$ repo_dev sync
real 0m8.046s
user 0m2.866s
sys 0m2.457s
Second time repo sync took only 8 seconds and verified by printing that
urrent_branch_only is True in project.py's Sync_NetworkHalf function.
Change-Id: Ic48efb23ea427dfa36e12a5c49973d6ae776d818
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301182
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This makes it easy to use --H as a shortcut, and kind of matches the
use of storing HEAD as the revision.
Change-Id: I590bf488518f313e7a593853140316df98262d7e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301163
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v
Without --depth
$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject
real 6m48.086s
user 3m27.281s
sys 1m1.386s
With --depth=1
$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject
real 2m49.637s
user 2m51.458s
sys 0m39.108s
From dwillemsen@:
"For me it's the difference between 9m28s using the current code and
16s using --depth=1 while fetching the superproject."
Bug: [google internal] b/180451672
Bug: https://crbug.com/gerrit/13707
Change-Id: I1c3e4aef4414c4e9dd259fb6e4619da0421896b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300922
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Historically forall has been interactive since it ran in serial.
Recent rework in here dropped that to enable parallel processing.
Restore support for interactive commands when running -j1 or with
an explicit --interactive option.
Bug: https://crbug.com/gerrit/14256
Change-Id: I502007186f771914cfd7830846a4e1938b5e1f38
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300722
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Add a new "event": "command", which is emitted at when all command
arguments have been processed.
Additional fields:
"name": Name of the primary command (ex: repo, git)
"subcommands"': List of the sub-commands once command-line arguments
are processed
Examples:
Command: repo --version
Event: {"event": "command", <common fields>,
"name": "repo",
"subcommands": ["version"]
}
Bug: [google internal] b/178507266
Testing:
- Unit tests
- Verified repo git trace2 logs had expected data
Change-Id: I825bd0ecedee45135382461a4ba10f987f09aef3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300343
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The refactor to multiprocessing broke status reporting slightly when
checking out projects. Make sure we mark the step as failed if any
of the projects failed, not just when --fail-fast is set.
Change-Id: I0efb56ce83b068b2c334046df3fef23d797599c9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299882
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We've had a limited version of this in CrOS for a long time. There's
nothing CrOS specific about it, so lets move it to the repo project so
everyone can utilize it.
Change-Id: I04cd94610c1100f3afcd2baf8c8e7ab13e589490
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299202
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Re-allow checking out projects to the top of the repo client checkout.
We add checks to prevent checking out files under .repo/ as that path
is only managed by us, and projects cannot inject content or settings
into it.
Bug: https://crbug.com/gerrit/14156
Bug: https://crbug.com/gerrit/14200
Change-Id: Id6bf9e882f5be748442b2c35bbeaee3549410b25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299623
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the current project doesn't have any local branches, then there's
nothing to prune, so return right away. This avoids running a few
git commands when we aren't actually going to use the results, and
it avoids checking repository validity. Since we aren't going to do
anything in here, no need to check it.
Change-Id: Ie9d5c75a954e42807477299f3e5a63a92fac138b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299742
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Added the following methods to XmlManifest class.
+ GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower()
+ GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func.
+ Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr.
+ Used the above methods to get groups in command.py::GetProjects
and part of init.py.
TODO: clean up these funcs to take structured group data more instead
of passing strings around everywhere that need parsing.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL
and verified prebuilts/fullsdk-linux directory has all the folders.
Tested repo init and repo sync with --use-superproject and without
--use-superproject argument.
$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync -c -j32
Bug: [google internal] b/181804931
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The recent commit 84230009ee ("project:
make diff tools synchronous") broke repo diff if it includes % formats.
Add an explicit format string to fix.
Bug: https://crbug.com/gerrit/14208
Change-Id: Ie255a43c5b767488616b2b3dd15abc18f93bfab2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299402
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
There's a few rough edges here still, but no known corruption ones,
so open it up a bit for people to experiment with.
Bug: https://crbug.com/gerrit/11486
Change-Id: I81e0122ab6d3e032c546c8239dd4f03740676e80
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299242
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Superproject objects accept the optional argument “quiet”.
The following progress messages are displayed if quiet is false.
Displayed the following message whenever we find we have to make a new
folder (aka new remote), because if you started with repo init android
and later do googleplex-android that is when it will be slow.
"<location>: Performing initial setup for superproject; this might take
several minutes.".
After fetch completion, added the following notification:
"<location>: Initial setup for superproject completed."
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
Bug: [google internal] b/181178282
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Ib748c61b1e65aee6dff8b97a9753d14c470a827f
Reported-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299002
Reviewed-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Allow the user to specify relative or absolute or any other funky
path that they want when using `repo init` or `repo sync`. Our
goal is to restrict the paths in the remote manifest git repo we
cloned from the network, not protect the user from themselves.
Bug: https://crbug.com/gerrit/14156
Change-Id: I1ccfb2a6bd1dce2bd765e261bef0bbf0f8a9beb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298823
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
With the recent commit 0501b29e7a
("status: Use multiprocessing for `repo status -j<num>` instead of
threading"), the limitation with project serialization no longer
applies. It turns out that ad-hoc logic is expensive. In the CrOS
checkout (~1000 projects w/8 jobs by default), it adds about ~7sec
overhead to all invocations. With a fast nop run:
time repo forall -j8 -c true
This goes from ~11sec to ~4sec -- more than 50% speedup.
Change-Id: Ie6bcccd21eef20440692751b7ebd36c890d5bbcc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298724
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The ProjectArgs function can be inlined which simplifies it quite a
bit. We shouldn't need the custom exception handling here either.
This also makes the next commit easier to review.
Change-Id: If3be04f58c302c36a0f20b99de0f67e78beac141
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298723
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This is in preparation for simplifying the jobs support. The nested
function is referenced in the options object which can't be pickled,
so pull it out into a static method instead.
Change-Id: I01d3c4eaabcb8b8775ddf22312a6e142c84cb77d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298722
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If a manifest checksout a project multiple times, repo download isn't
able to accurately pick the right project. We were just picking the
first result which could be a bit random for the user. If we hit that
situation, check if the cwd is one of the projects, and if it isn't,
we emit an error and tell the user it's an ambiguous request.
Bug: https://crbug.com/gerrit/13070
Change-Id: Id1059b81330229126b48c7312569b37504808383
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298702
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The clone bundle logic assumes there is a one-to-one mapping between the
projects/ and project-objects/ trees. When using shared projects (where
we checkout different branches from the same project), this would lead us
to fetching the same clone bundle multiple times. Automatically skip the
clone bundle logic if the project-objects/ dir already exists.
Bug: https://crbug.com/gerrit/10993
Change-Id: I82c6fa1faf8605fd56c104fcea2a43dd4eecbce4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
On macOS, the Finder app likes to poop .DS_Store files in every path
that the user browses. If the user pokes around the .git/ tree, it
could generate a .DS_Store file in there too. When repo goes to read
all the local refs, it tries to decode this binary file as UTF-8 and
subsequently crashes.
Since paths that begin with . are not valid refs, ignore them like we
already do with paths that end in .lock. Also bump the check up to
ignore dirs that match since that follows the git rules: they apply
to any component in its path, not just the final path (name).
We don't implement the full valid ref algorithm that git employs as
it's a bit complicated, and we only really need to focus on what will
practically show up locally.
Bug: https://crbug.com/gerrit/14162
Change-Id: I6519f990e33cc58a72fcb00c0f983ad3285ace3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298662
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
When using Git under Windows, it seems that Git doesn't always parse
GIT_DIR correctly when it uses the Windows \ form, but does when it
uses / only.
For example, when using worktrees:
$ GIT_DIR='C:\Users\vapier\Desktop\repo\breakpad\tools\test\.git' git worktree list
fatal: not a git repository: ..\..\.repo\worktrees\linux-syscall-support.git\worktrees\test
$ GIT_DIR='C:/Users/vapier/Desktop/repo/breakpad/tools/test/.git' git worktree list
C:/Users/vapier/Desktop/repo/breakpad/.repo/worktrees/linux-syscall-support.git fd00dbbd0c06 (detached HEAD)
..\..\..\..\..\src\src\third_party\lss\.git fd00dbbd0c06 (detached HEAD)
..\..\..\..\..\tools\test\.git fd00dbbd0c06 (detached HEAD)
Change-Id: I666c03ae845ecb55d7f9800731ea6987d3e7f401
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298622
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
These attribute values are used to construct local filesystem paths,
so apply the existing filesystem checks to them.
Bug: https://crbug.com/gerrit/14156
Change-Id: Ibcceecd60fa74f0eb97cd9ed1a9792e139534ed4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298443
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This function is currently written with copyfile & linkfile in mind.
Generalize the logic & function arguments slightly so we can reuse
in more places that make sense.
This changes the validation logic slightly too in that we no longer
allow "." for the dest attribute with copyfile & linkfile, nor for
the src attribute with copyfile. We already rejected those later on
when checking against the active filesystem, but now we reject them
a little sooner when parsing.
The empty path check isn't a new requirement exactly -- repo used to
crash on it, so it was effectively blocked, but now we diagnosis it.
Bug: https://crbug.com/gerrit/14156
Change-Id: I0fdb42a3da60ed149ff1997c5dd4b85da70eec3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can speed things up. Across 1000 repos, it goes from
~9sec to ~5sec with the default -j8.
Change-Id: Ida6dd565db78ff7bac0ecb25d2805e8a1bf78048
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297982
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This avoids GIL limitations with using threads for parallel processing.
In a CrOS checkout with ~1000 repos, the nop case goes from ~6 sec down
to ~4 sec with -j8. Not a big deal, but shows that this actually works
to speed things up unlike the threading model.
This reworks the checkout logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.
Bug: https://crbug.com/gerrit/12389
Change-Id: I143e5e3f7158e83ea67e2d14e5552153a874248a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298063
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~30sec to ~3sec with the default -j8.
Change-Id: I0dc62d704c022dd02cac0bd67fe79224f4e34095
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297484
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
We're already keeping tracking of the start time, so might as
well use it to display overall execution time for steps.
Change-Id: Ib4cf8b2b0dfcdf7b776a84295d59cc569971bdf5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298482
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
+ superproject will be fetched into a directory with the name
“<remote name>-superproject.git” instead of the current
“superproject.git” folder.
+ Deleted _Clone method and added _Init method.
+ _Init method will do “git init --bare <remote>-superproject.git”.
It will create the folder and set up a bare repository in
<remote>-superproject.git folder.
+ _Fetch method, will pass <remote url>, <branch> arguments.
Moved the --filter argument from “git clone” to “git fetch”.
_Fetch method will execute the following command to fetch
superproject. Added --no-tags argument.
master: git fetch <remote url> --force --no-tags --filter blob:none
branch: git fetch <remote url> --force --no-tags --filter blob:none \
<branch>:<branch>
+ Performance improvements for aosp-master
++ repo init performance improved from 35 seconds to 17 seconds.
++ repo init --use-superproject is around 5 to 7 secsonds slower.
++ repo sync --use-superproject is around 3 to 4 minutes faster.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real 0m20.648s
user 0m8.046s
sys 0m3.271s
+ Without superproject
$ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main
real 0m13.078s
user 0m9.783s
sys 0m2.528s
$ time repo_dev sync -c -j32 --use-superproject
...
real 15m7.072s
user 110m7.216s
sys 20m17.559s
+ Without superproject
$ time repo sync -c -j32
...
real 19m25.644s
user 91m56.331s
sys 20m59.170s
Bug: [google internal] b/180492484
Bug: [google internal] b/179470886
Bug: [google internal] b/180124069
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
This is in preparation for adding jobs support. The nested function
is referenced in the options object which can't be pickled, so pull
it out into a static method instead.
Change-Id: I280ed2bf26390a0203925517a0d17c13053becaa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297983
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The default sync output should show a progress bar only for successful
commands, and the error output for any commands that fail. Implement
that policy here.
Bug: https://crbug.com/gerrit/11293
Change-Id: I85716032201b6e2b45df876b07dd79cb2c1447a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297905
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The idea for skipping some progress updates was to avoid spending
too much time on the progress bar itself. Unfortunately, for large
projects (100s if not 1000s) of repos, we get into the situation
with large/slow checkouts that we skip showing updates when a repo
finishes, but not enough repos finished to increase the percent.
Since the progress bar should be relatively fast compared to the
actual network & local dick operations, have it show an update
whenever the caller requests it. A test with ~1000 repos shows
that the progress bar in total adds <100ms.
Bug: https://crbug.com/gerrit/11293
Change-Id: I708a0c4bd923c59c7691a5b48ae33eb6fca4cd14
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297903
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
