mirror of
https://gerrit.googlesource.com/git-repo
synced 2024-12-21 07:16:21 +00:00
39ffd9977e
Background: - Manifest object is large (for projects like Android) in terms of serialization cost and size (more than 1mb). - Lots of Project objects usually share only a few manifest objects. Before this CL, Project objects were passed to workers via function parameters. Function parameters are pickled separately (in chunk). In other words, manifests are serialized again and again. The major serialization overhead of repo sync was O(manifest_size * projects / chunksize) This CL uses following tricks to reduce serialization overhead. - All projects are pickled in one invocation. Because Project objects share manifests, pickle library remembers which objects are already seen and avoid the serialization cost. - Pass the Project objects to workers at worker intialization time. And pass project index as function parameters instead. The number of workers is much smaller than the number of projects. - Worker init state are shared on Linux (fork based). So it requires zero serialization for Project objects. On Linux (fork based), the serialization overhead is O(projects) --- one int per project On Windows (spawn based), the serialization overhead is O(manifest_size * min(workers, projects)) Moreover, use chunksize=1 to avoid the chance that some workers are idle while other workers still have more than one job in their chunk queue. Using 2.7k projects as the baseline, originally "repo sync" no-op sync takes 31s for fetch and 25s for checkout on my Linux workstation. With this CL, it takes 12s for fetch and 1s for checkout. Bug: b/371638995 Change-Id: Ifa22072ea54eacb4a5c525c050d84de371e87caa Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439921 Tested-by: Kuang-che Wu <kcwu@google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Kuang-che Wu <kcwu@google.com> |
||
|---|---|---|
| .github/workflows | ||
| docs | ||
| hooks | ||
| man | ||
| release | ||
| subcmds | ||
| tests | ||
| .flake8 | ||
| .gitattributes | ||
| .gitignore | ||
| .gitreview | ||
| .isort.cfg | ||
| .mailmap | ||
| .project | ||
| .pydevproject | ||
| color.py | ||
| command.py | ||
| completion.bash | ||
| constraints.txt | ||
| editor.py | ||
| error.py | ||
| event_log.py | ||
| fetch.py | ||
| git_command.py | ||
| git_config.py | ||
| git_refs.py | ||
| git_ssh | ||
| git_superproject.py | ||
| git_trace2_event_log_base.py | ||
| git_trace2_event_log.py | ||
| hooks.py | ||
| LICENSE | ||
| main.py | ||
| manifest_xml.py | ||
| MANIFEST.in | ||
| pager.py | ||
| platform_utils_win32.py | ||
| platform_utils.py | ||
| progress.py | ||
| project.py | ||
| pyproject.toml | ||
| README.md | ||
| repo | ||
| repo_logging.py | ||
| repo_trace.py | ||
| requirements.json | ||
| run_tests | ||
| run_tests.vpython3 | ||
| setup.py | ||
| ssh.py | ||
| SUBMITTING_PATCHES.md | ||
| tox.ini | ||
| wrapper.py | ||
repo
Repo is a tool built on top of Git. Repo helps manage many Git repositories, does the uploads to revision control systems, and automates parts of the development workflow. Repo is not meant to replace Git, only to make it easier to work with Git. The repo command is an executable Python script that you can put anywhere in your path.
- Homepage: https://gerrit.googlesource.com/git-repo/
- Mailing list: repo-discuss on Google Groups
- Bug reports: https://issues.gerritcodereview.com/issues?q=is:open%20componentid:1370071
- Source: https://gerrit.googlesource.com/git-repo/
- Overview: https://source.android.com/source/developing.html
- Docs: https://source.android.com/source/using-repo.html
- repo Manifest Format
- repo Hooks
- Submitting patches
- Running Repo in Microsoft Windows
- GitHub mirror: https://github.com/GerritCodeReview/git-repo
- Postsubmit tests: https://github.com/GerritCodeReview/git-repo/actions
Contact
Please use the repo-discuss mailing list or issue tracker for questions.
You can file a new bug report under the "repo" component.
Please do not e-mail individual developers for support. They do not have the bandwidth for it, and often times questions have already been asked on repo-discuss or bugs posted to the issue tracker. So please search those sites first.
Install
Many distros include repo, so you might be able to install from there.
# Debian/Ubuntu.
$ sudo apt-get install repo
# Gentoo.
$ sudo emerge dev-vcs/repo
You can install it manually as well as it's a single script.
$ mkdir -p ~/.bin
$ PATH="${HOME}/.bin:${PATH}"
$ curl https://storage.googleapis.com/git-repo-downloads/repo > ~/.bin/repo
$ chmod a+rx ~/.bin/repo