Kuang-che Wu 39ffd9977e sync: reduce multiprocessing serialization overhead ... Background: - Manifest object is large (for projects like Android) in terms of serialization cost and size (more than 1mb). - Lots of Project objects usually share only a few manifest objects. Before this CL, Project objects were passed to workers via function parameters. Function parameters are pickled separately (in chunk). In other words, manifests are serialized again and again. The major serialization overhead of repo sync was O(manifest_size * projects / chunksize) This CL uses following tricks to reduce serialization overhead. - All projects are pickled in one invocation. Because Project objects share manifests, pickle library remembers which objects are already seen and avoid the serialization cost. - Pass the Project objects to workers at worker intialization time. And pass project index as function parameters instead. The number of workers is much smaller than the number of projects. - Worker init state are shared on Linux (fork based). So it requires zero serialization for Project objects. On Linux (fork based), the serialization overhead is O(projects) --- one int per project On Windows (spawn based), the serialization overhead is O(manifest_size * min(workers, projects)) Moreover, use chunksize=1 to avoid the chance that some workers are idle while other workers still have more than one job in their chunk queue. Using 2.7k projects as the baseline, originally "repo sync" no-op sync takes 31s for fetch and 25s for checkout on my Linux workstation. With this CL, it takes 12s for fetch and 1s for checkout. Bug: b/371638995 Change-Id: Ifa22072ea54eacb4a5c525c050d84de371e87caa Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439921 Tested-by: Kuang-che Wu <kcwu@google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Kuang-che Wu <kcwu@google.com>		2024-10-23 02:58:45 +00:00
..
__init__.py	cleanup: Update codebase to expect Python 3.6	2023-10-31 16:03:54 +00:00
abandon.py	cleanup: Update codebase to expect Python 3.6	2023-10-31 16:03:54 +00:00
branches.py	cleanup: Update codebase to expect Python 3.6	2023-10-31 16:03:54 +00:00
checkout.py	cleanup: Update codebase to expect Python 3.6	2023-10-31 16:03:54 +00:00
cherry_pick.py	Update logger.warn to logger.warning	2023-10-13 19:34:26 +00:00
diff.py	isort: format codebase	2023-08-22 18:32:22 +00:00
diffmanifests.py	cleanup: Update codebase to expect Python 3.6	2023-10-31 16:03:54 +00:00
download.py	subcmds: Use repo logger	2023-09-14 17:13:37 +00:00
forall.py	subcmds: Use repo logger	2023-09-14 17:13:37 +00:00
grep.py	subcmds: Use repo logger	2023-09-14 17:13:37 +00:00
help.py	cleanup: Update codebase to expect Python 3.6	2023-10-31 16:03:54 +00:00
info.py	info: Handle undefined mergeBranch	2023-11-20 19:22:53 +00:00
init.py	init: add --manifest-upstream-branch	2024-09-26 00:52:28 +00:00
list.py	cleanup: Update codebase to expect Python 3.6	2023-10-31 16:03:54 +00:00
manifest.py	Update logger.warn to logger.warning	2023-10-13 19:34:26 +00:00
overview.py	Format codebase with black and check formatting in CQ	2023-03-22 17:46:28 +00:00
prune.py	cleanup: Update codebase to expect Python 3.6	2023-10-31 16:03:54 +00:00
rebase.py	Update logger.warn to logger.warning	2023-10-13 19:34:26 +00:00
selfupdate.py	subcmds: Use repo logger	2023-09-14 17:13:37 +00:00
smartsync.py	Format codebase with black and check formatting in CQ	2023-03-22 17:46:28 +00:00
stage.py	subcmds: Use repo logger	2023-09-14 17:13:37 +00:00
start.py	cleanup: Update codebase to expect Python 3.6	2023-10-31 16:03:54 +00:00
status.py	isort: format codebase	2023-08-22 18:32:22 +00:00
sync.py	sync: reduce multiprocessing serialization overhead	2024-10-23 02:58:45 +00:00
upload.py	upload: add a --topic option for setting topic explicitly	2024-07-01 17:54:19 +00:00
version.py	cleanup: Update codebase to expect Python 3.6	2023-10-31 16:03:54 +00:00