git-repo/git_superproject.py

291 lines
9.9 KiB
Python
Raw Normal View History

# Copyright (C) 2021 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
"""Provide functionality to get all projects and their commit ids from Superproject.
For more information on superproject, check out:
https://en.wikibooks.org/wiki/Git/Submodules_and_Superprojects
Examples:
superproject = Superproject()
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
project_commit_ids = superproject.UpdateProjectsRevisionId(projects)
"""
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
import hashlib
import os
import sys
from git_command import GitCommand
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
from git_refs import R_HEADS
from wrapper import Wrapper
_SUPERPROJECT_GIT_NAME = 'superproject.git'
_SUPERPROJECT_MANIFEST_NAME = 'superproject_override.xml'
class Superproject(object):
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
"""Get commit ids from superproject.
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
Initializes a local copy of a superproject for the manifest. This allows
lookup of commit ids for all projects. It contains _project_commit_ids which
is a dictionary with project/commit id entries.
"""
def __init__(self, manifest, repodir, superproject_dir='exp-superproject',
quiet=False):
"""Initializes superproject.
Args:
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
manifest: A Manifest object that is to be written to a file.
repodir: Path to the .repo/ dir for holding all internal checkout state.
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
It must be in the top directory of the repo client checkout.
superproject_dir: Relative path under |repodir| to checkout superproject.
quiet: If True then only print the progress messages.
"""
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
self._project_commit_ids = None
self._manifest = manifest
self._quiet = quiet
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
self._branch = self._GetBranch()
self._repodir = os.path.abspath(repodir)
self._superproject_dir = superproject_dir
self._superproject_path = os.path.join(self._repodir, superproject_dir)
self._manifest_path = os.path.join(self._superproject_path,
_SUPERPROJECT_MANIFEST_NAME)
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
git_name = ''
if self._manifest.superproject:
remote_name = self._manifest.superproject['remote'].name
git_name = hashlib.md5(remote_name.encode('utf8')).hexdigest() + '-'
self._work_git_name = git_name + _SUPERPROJECT_GIT_NAME
self._work_git = os.path.join(self._superproject_path, self._work_git_name)
@property
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
def project_commit_ids(self):
"""Returns a dictionary of projects and their commit ids."""
return self._project_commit_ids
def _GetBranch(self):
"""Returns the branch name for getting the approved manifest."""
p = self._manifest.manifestProject
b = p.GetBranch(p.CurrentBranch)
if not b:
return None
branch = b.merge
if branch and branch.startswith(R_HEADS):
branch = branch[len(R_HEADS):]
return branch
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
def _Init(self):
"""Sets up a local Git repository to get a copy of a superproject.
Returns:
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
True if initialization is successful, or False.
"""
if not os.path.exists(self._superproject_path):
os.mkdir(self._superproject_path)
if not self._quiet and not os.path.exists(self._work_git):
print('%s: Performing initial setup for superproject; this might take '
'several minutes.' % self._work_git)
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
cmd = ['init', '--bare', self._work_git_name]
p = GitCommand(None,
cmd,
cwd=self._superproject_path,
capture_stdout=True,
capture_stderr=True)
retval = p.Wait()
if retval:
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
print('repo: error: git init call failed with return code: %r, stderr: %r' %
(retval, p.stderr), file=sys.stderr)
return False
return True
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
def _Fetch(self, url):
"""Fetches a local copy of a superproject for the manifest based on url.
Args:
url: superproject's url.
Returns:
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
True if fetch is successful, or False.
"""
if not os.path.exists(self._work_git):
print('git fetch missing drectory: %s' % self._work_git,
file=sys.stderr)
return False
cmd = ['fetch', url, '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none']
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
if self._branch:
cmd += [self._branch + ':' + self._branch]
p = GitCommand(None,
cmd,
cwd=self._work_git,
capture_stdout=True,
capture_stderr=True)
retval = p.Wait()
if retval:
print('repo: error: git fetch call failed with return code: %r, stderr: %r' %
(retval, p.stderr), file=sys.stderr)
return False
return True
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
def _LsTree(self):
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
"""Gets the commit ids for all projects.
Works only in git repositories.
Returns:
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
data: data returned from 'git ls-tree ...' instead of None.
"""
if not os.path.exists(self._work_git):
print('git ls-tree missing drectory: %s' % self._work_git,
file=sys.stderr)
return None
data = None
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
branch = 'HEAD' if not self._branch else self._branch
cmd = ['ls-tree', '-z', '-r', branch]
p = GitCommand(None,
cmd,
cwd=self._work_git,
capture_stdout=True,
capture_stderr=True)
retval = p.Wait()
if retval == 0:
data = p.stdout
else:
print('repo: error: git ls-tree call failed with return code: %r, stderr: %r' % (
retval, p.stderr), file=sys.stderr)
return data
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
def Sync(self):
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
"""Gets a local copy of a superproject for the manifest.
Returns:
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
True if sync of superproject is successful, or False.
"""
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
print('WARNING: --use-superproject is experimental and not '
'for general use', file=sys.stderr)
if not self._manifest.superproject:
print('error: superproject tag is not defined in manifest',
file=sys.stderr)
return False
url = self._manifest.superproject['remote'].url
if not url:
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
print('error: superproject URL is not defined in manifest',
file=sys.stderr)
return False
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
if not self._Init():
return False
if not self._Fetch(url):
return False
if not self._quiet:
print('%s: Initial setup for superproject completed.' % self._work_git)
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
return True
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
def _GetAllProjectsCommitIds(self):
"""Get commit ids for all projects from superproject and save them in _project_commit_ids.
Returns:
A dictionary with the projects/commit ids on success, otherwise None.
"""
if not self.Sync():
return None
data = self._LsTree()
if not data:
sync: superproject - support for switching hosts and switching branches. + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-23 00:54:56 +00:00
print('error: git ls-tree failed to return data for superproject',
file=sys.stderr)
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
return None
# Parse lines like the following to select lines starting with '160000' and
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
# build a dictionary with project path (last element) and its commit id (3rd element).
#
# 160000 commit 2c2724cb36cd5a9cec6c852c681efc3b7c6b86ea\tart\x00
# 120000 blob acc2cbdf438f9d2141f0ae424cec1d8fc4b5d97f\tbootstrap.bash\x00
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
commit_ids = {}
for line in data.split('\x00'):
ls_data = line.split(None, 3)
if not ls_data:
break
if ls_data[0] == '160000':
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
commit_ids[ls_data[3]] = ls_data[2]
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
self._project_commit_ids = commit_ids
return commit_ids
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
def _WriteManfiestFile(self):
"""Writes manifest to a file.
Returns:
manifest_path: Path name of the file into which manifest is written instead of None.
"""
if not os.path.exists(self._superproject_path):
print('error: missing superproject directory %s' %
self._superproject_path,
file=sys.stderr)
return None
manifest_str = self._manifest.ToXml(groups=self._manifest.GetGroupsStr()).toxml()
manifest_path = self._manifest_path
try:
with open(manifest_path, 'w', encoding='utf-8') as fp:
fp.write(manifest_str)
except IOError as e:
print('error: cannot write manifest to %s:\n%s'
% (manifest_path, e),
file=sys.stderr)
return None
return manifest_path
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
def UpdateProjectsRevisionId(self, projects):
"""Update revisionId of every project in projects with the commit id.
Args:
projects: List of projects whose revisionId needs to be updated.
Returns:
manifest_path: Path name of the overriding manfiest file instead of None.
"""
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
commit_ids = self._GetAllProjectsCommitIds()
if not commit_ids:
print('error: Cannot get project commit ids from manifest', file=sys.stderr)
return None
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
projects_missing_commit_ids = []
superproject_remote_name = self._manifest.superproject['remote'].name
for project in projects:
path = project.relpath
if not path:
continue
# Some manifests that pull projects from the "chromium" GoB
# (remote="chromium"), and have a private manifest that pulls projects
# from both the chromium GoB and "chrome-internal" GoB (remote="chrome").
# For such projects, one of the remotes will be different from
# superproject's remote. Until superproject, supports multiple remotes,
# don't update the commit ids of remotes that don't match superproject's
# remote.
if project.remote.name != superproject_remote_name:
continue
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
commit_id = commit_ids.get(path)
if commit_id:
project.SetRevisionId(commit_id)
else:
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
projects_missing_commit_ids.append(path)
if projects_missing_commit_ids:
print('error: please file a bug using %s to report missing commit_ids for: %s' %
(Wrapper().BUG_URL, projects_missing_commit_ids), file=sys.stderr)
return None
init: added --use-superproject option to clone superproject. Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-09 08:26:31 +00:00
manifest_path = self._WriteManfiestFile()
return manifest_path