To be addressed in another change:
- a partial `repo sync` (with a list of projects/paths to sync)
requires `--this-tree-only`.
Change-Id: I6c7400bf001540e9d7694fa70934f8f204cb5f57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322657
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This removes the need for git_superproject to include manifest_xml, and
puts the logic for local_manifest detection in one place.
Change-Id: I4d33ded0542ceea4606a1ea24304f678de20c59e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/330499
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
This reverts commit d53cb9549a. As long as
repo's reference docs treat this feature as a work in progress and don't
cover it well enough to allow all repo maintainers to easily support it,
it is inconsistent to report to users that it is no longer in beta.
Thanks for vapier@google.com for noticing.
https://crbug.com/gerrit/15527 tracks the required documentation changes
before we'd be ready to roll forward again.
Change-Id: Ic9bd951cfb3c1abf6e1bfa30dfe4afa1c9b7bec6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327337
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jonathan Nieder <jrn@google.com>
The current code would check for enrollment status when the user did not
explicitly specify --[no-]use-superproject and do not have a remembered
value in their repo client. However, because superproject only makes
sense for manifests that have one specified, we should skip the
enrollment logic in that case.
Address this by checking manifest.superproject prior to proceeding. This
would avoid showing the greeting message of superproject enrollment
which can be confusing for developers.
Tested:
For manifest without superproject:
- repo sync --use-superproject will still show message for
superproject;
- repo sync will not show message regardless of enrollment state
For manifest with superproject:
- repo sync will show message and perform enrollment if not
previously enrolled
Bug: https://crbug.com/gerrit/15039
Change-Id: Ic2be9f9d037f0e7cf3446da474a5a0d0e4bd88da
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319255
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested:
$ ./run_tests
Verified that a manifest that specified superproject revision would use
the specified revision, and superproject will use the default revision.
Note that this is a slight behavior change from earlier repo versions,
which would always use the branch name of the manifest itself. However,
the new behavior would be more consisitent with regular "project"
element and would allow superproject be used even if it is not enabled
for the particular manifest branch, so we have decided to make the
change as it would provide more flexibility and better matches what
other elements would do.
Bug: [google internal] b/187868160
Change-Id: I35255ee347aff6e65179f7879d52931f168b477e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317643
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Our internal experiments was a success so far and we are enrolling 100%
users now. Instead of asking every two weeks, simply consider a lack of
unexpired choice as accepting the system default.
With this change the user would still be able to override the system
default with --no-use-superproject, or to permanently set the choice in
user's profile with git config --global repo.superprojectchoice.
Bug: [google internal] b/190688390
Change-Id: Idc77a9cbf88a169d90304169e91f0d722dc4ac8b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317975
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Saved superproject's remote URL in _remote_url data and used it
in the _Fecth function.
Tested:
$ ./run_tests
$ flake8 git_superproject.py
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
$ repo_dev sync
Verified the all log messages have the following format.
repo superproject branch: <branch> url: <url> warning: <message>
Bug: [google internal] b/200072098
Change-Id: Iac6af7c99225479fd50bc6909396b22e0ce5f76b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318177
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Currently the code would give a message that would appear like the user
have enrolled the experiment regardless of the actual choice. For users
who choose to not enroll in the experiment, we should give them
instructions to override (enable) superproject once instead of how to
disable it, which is what the code already behave.
Bug: [google internal] b/199167992
Change-Id: Iba3314cb510aedf024375a26baa8bc1d5e2846cf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317382
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
1) If the manifest has superproject tag (git_master, etc), then
display error/warning messages (as it is doing today)
2) If the manifest doesn't have superproject tag (nest, chromeos
manifests), then don't display any error/warning messages about
superrproject (behave as though user has specified
--no-use-superproject).
3) Print error/warning messages if --use-superproject passed as
argument to repo sync.
4) No change in behavior for the repo init command.
git_superproject.py:
+ Fixed typo in _WriteManifestFile method name
+ Superproject accepts print_message as an argument and it defaults
to True. All messages that are printed to stderr are controlled by
this flag. If it is True, then messages get printed.
+ Added PrintMessages function which return true if either
--use-superproject is specified on the command line or if the
manifest has a superproject tag.
sync.py:
+ Displays the warning message if PrintMessgages are enabled and
passes that as argument to superproject object.
+ Added 'hassuperprojecttag' trace2 log entry for analysis. We can
find users/branches that are using superproject, but the manifest is
missing the superproject tag.
Tested:
$ ./run_tests
+ Verified printing of messages with and without superproject tag, with
with --use-superproject option.
+ aosp-master
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
$ repo_dev sync
+ A manifest without superproject tag.
$ repo_dev init -m $(pwd)/manifest_7482982.xml
$ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml
Bug: [google internal] b/196411099
Change-Id: I92166dcad15a4129fab82edcf869e7c8db3efd4b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314982
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Skip updating the superproject when -l is present and use the existing
superproject, if available (this would make sync -l work as it's
intended to do), and fall back to sync without superproject when not
(this would catch the case when superproject is enabled by automatic
rollout).
Tested:
$ repo sync -j 20 -n
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/android/src/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Fetching: 100% (1032/1032), done in 41.184s
...
$ repo_dev sync -j 20 -l
prebuilts/asuite/: discarding 1 commits
prebuilts/runtime/: discarding 1 commits
...
repo sync has finished successfully.
+ With superproject-override.xml and test it.
$ ls -l .repo/exp-superproject/
total 176
drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
-rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 superproject_override.xml
rtenneti@rtenneti:~/work/android/src/aosp$ repo_dev sync -j 20 -l
...
repo sync has finished successfully.
+ Rename the file superproject-override.xml and test it.
$ ls -l .repo/exp-superproject/
total 176
drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
-rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 temp.xml
$ repo_dev sync -j 20 -l
Checking out: 1% (12/1031) platform/external/rust/crates/fallible-streaming-iteexternal/linux-kselftest/: discarding 1 commits
prebuilts/remoteexecution-client/: discarding 1 commits
Checking out: 51% (536/1031) platform/prebuilts/gcc/darwin-x86/aarch64/....
....
Checking out: 100% (1031/1031), done in 5.478s
repo sync has finished successfully.
Bug: [google internal] b/184368268
Change-Id: I3aba5872e4f7c299977b92c2a39847ef28698c5a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312962
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
superproject is going to be default for some users. This change
doesn't fail for repo init or repo sync if source couldn't be synced
because of errors in superproject and superproject=true in the config
file. The commands will fail if --use-superproject is specified on
the command line explicitly.
The error messages are logged with trace2 event logs and will be
monitored.
+ sync - When there are errors with superproject and git_superproject
says it is fatal failure, sync will exit only when --use-superproject
option is specified on the command line.
+ init - command doesn't fail *if there are any superproject errors),
but it will print a warning message and logs message via trace2 event
logs. For fatal errors, init will exit only when --use-superproject
option is specified on the command line.
+ All git commands log the command that is being executed so trace2
event logs will know the manifest, remote url and the branch name.
There is no functional change other than fatal errors are honored with
--use-supeproject option with init/sync commands.
Tested the code with the following commands.
$ ./run_tests -v
Test 1 - sync'ing without errors
--------------------------------
Added the following lines to '~/.repoconfig/config
[repo]
superproject = true
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in /usr/local/google/home/rtenneti/work/drive2/android/test
$ repo_dev sync
remote: Total 4 (delta 1), reused 4 (delta 1)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/drive2/android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
...
Test 2 - init and sync fail when --use-superproject option is passed
--------------------------------------------------------------------
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 --use-superproject
remote: Total 57 (delta 16), reused 56 (delta 16)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning
rtenneti@rtenneti2:~/work/drive2/android/test$ repo_dev sync --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: Cannot get project commit ids from manifest
warning: Update of revisionId from superproject has failed, repo sync will not use superproject to fetch the source. Please resync with the --no-use-superproject option to avoid this repo warning.
Test 3 - git fetch command fails and git command is printed
-----------------------------------------------------------
With config change
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2
...
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/test/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Performing initial setup for superproject; this might take several minutes.
repo: error: git fetch call failed,command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in ....
Test 4 - no superproject tag
-----------------------------
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b pie-dev
...
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: .../android/pie_dev/.repo/manifest.xml
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in ...
$ repo_dev sync
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: /usr/local/google/home/rtenneti/work/drive2/android/pie_dev/.repo/manifest.xml
warning: Cannot get project commit ids from manifest
warning: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option to avoid the repo warning.
Bug: [google internal] b/192614798
Bug: [google internal] b/Bug: [google internal] b/192614798
Change-Id: I9a97a0e7d9e609fad151bd7dd9cfc523eaa887cd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311502
Reviewed-by: Amith Dsouza <amithds@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Repo will remember a choice and an expiration time of the choice, per
user, about whether to use superproject by default. When not specified
from command line and the choice is not expired, repo would use the
user default value.
When a user default value is not present and when the system wide
enable default is provided in git's system configuration, repo would
ask the user for a confirmation which will be valid for two weeks.
git_config.py: Add support for system config. When reading system
config, we would use --system to avoid hardcoding a path as the
value may be different on some other distributions.
git_superproject.py: Add a new subroutine, _UseSuperproject(), which
returns whether superproject should be used and whether it
is from a user configuration.
The value is determined in the following order:
1. If the user specifies either --use-superproject or
--no-use-superproject, then that choice is being used.
2. If neither is specified, we would then check the saved value
(upon repo init) and use that choice when there was a choice.
3. We then check if there is a saved and unexpired value for
user's choice in their ~/.gitconfig, and use the unexpired
choice, if available.
4. Finally, if all the above didn't give us a decision, and if
the git system configuration is providing a rollout hint, present
a prompt to user for their decision and save it in ~/.gitconfig.
subcmds/sync.py: Make use of the new UseSuperproject() provided by
git_superproject.py.
While there also silent stderr from git describe when determining the
version of repo.
Bug: [google internal] b/190688390
Change-Id: Iad3ee03026342ee500e5d65e2f0fa600d7637613
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309762
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
Pinned manifests and release manifest have revisionId set for all
projects. For such projects don't update the commit ids.
Tested the code with the following commands.
$ ./run_tests -v
$ repo_dev sync --use-superproject -j8
$ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml
Bug: [google internal] b/191995372
Change-Id: I4681135b1d15f4a63527b6f0356d76ec842485d6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310582
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Don't exit if there are missing commit ids in superproject.
This change implements the following suggestion from delphij@:
"we should note the event (so we know that --use-superproject but there
were some errors, e.g. manifest didn't specify commit id for some
reason, or if there is no superproject but --use-superproject is
used), print out a message telling the use that this is not support,
but continue as if --no-use-superproject was specified?"
Changes:
superproject:
+ Added git_trace2_event_log as an argument to the constructor.
+ Sync method returns SyncResult a NamedTuple of
++ success - True if sync of superproject is successful, or False.
++ fatal - True if caller should exit, Or False.
+ UpdateProjectsRevisionId returns UpdateProjectsResult a NamedTuple of
++ manifest_path - path name of the overriding manifest file instead
of None
++ fatal - True if caller should exit, Or False
+ _GetAllProjectsCommitIds returns CommitIdsResult a NamedTuple of
++ commit_ids - a dictionary with the projects/commit ids on success,
otherwise None
++ fatal - True if caller should exit, Or False
+ Added _SkipUpdatingProjectRevisionId a helper function to see if a
project's revision id needs to be updated or not. This function is
used to exclude projects from local manifest file.
+ Added the following error events into git_trace2_event_log
++ If superproject is missing in a manifest
++ If there are missing commit ids for projects.
command.py:
+ Deleted unused import - platform
+ Added git_trace2_event_log as a member so all subcmds can log error
events.
main.py:
+ Initialized git_trace2_event_log as a member of command object.
init.py:
+ Deleted unused import - optparse
init.py:
+ Called sys.exit only if Sync returns exit=True
sync.py:
+ Called sys.exit only if Superproject's UpdateProjectsRevisionId returns
exit=True
+ Reloaded the manifest only if manifest path is returned by
UpdateProjectsRevisionId. If not, fall back to the old way of doing
repo sync.
test_git_superproject:
+ Added code to verify error events are being logged.
+ Added a test for no superproject tag
+ Added test for UpdateProjectsRevisionId not updating the revision id
with the commit ids.
Tested the code with the following commands.
+ Positive test case with aosp-master.
$ repo_dev init -u persistent-https://android.git.corp.google.com/platform/manifest -b master --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in .../android/aosp
$ repo_dev sync -j40 --use-superproject
remote: Total 12 (delta 4), reused 12 (delta 4)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
...
repo sync has finished successfully.
+ Negative test case without superproject tag.
$ repo_dev sync -j40 --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: .../android/aosp/.repo/manifest.xml
error: Cannot get project commit ids from manifest
error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
...
Checking out: 100% (1022/1022), done in 3.589s
repo sync has finished successfully.
+ Test for missing commit_id for a project.
$ repo_dev sync -j40 --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
error: please file a bug using go/repo-bug to report missing commit_ids for: ['build/blueprint']
error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
...
Checking out: 100% (1022/1022), done in 3.364s
repo sync has finished successfully.
$ ./run_tests -v
...
...== 164 passed in 2.87s ==...
Bug: [google internal] b/189371541
Change-Id: I5ea49f87e8fa41be590fc0c914573e16c8cdfcfa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309162
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
With repo sync --use-superproject, don't update the commit ids of every project
that comes from local manifest.
Tested the code with the following commands.
$ ./run_tests -v
+ Test with local.xml
1. repo init --use-superproject -u persistent-https://googleplex-android.git.corp.google.com/a/platform/manifest
2. cd .repo
cp -r /google/src/head/depot/google3/wireless/android/build_tools/translations/pipeline/local_manifests local_manifests
cd ..
local$ time repo_dev sync --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../local/.repo/exp-superproject/feb2c2847da5e274f3d530d5ab438af8-superproject.git: Initial setup for superproject completed.
...
Bug: [google internal] b/189360443
Bug: [google internal] b/189139268
Bug: https://crbug.com/gerrit/14499
Change-Id: Ideaf268c294e9b500b2b9726ffbd733dd8d63004
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308822
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
superproject supports multiple remotes. Get all commit ids
from superproject for all projects that are in the manifest.
$ ./run_tests -v
Bug: [google internal] b/186395810
Change-Id: I6edce3918853a7a3a65aec5528e6a43a544eff53
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308862
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
+ In XmlManifest._Unload set 'bugurl' to Wrapper().BUG_URL.
+ contactinfo returns a namedtuple.
+ bug_url can be accessed as self._manifest.contactinfo.bugurl.
Tested the code with the following commands.
$ ./run_tests -v
Added contactinfo tag to default.xml and verified that bugurl is used.
Bug: [google internal] b/186220520.
Change-Id: Iaafd6465e072b2e47a0a0b548bf6cb608a0b0a04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/306342
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Make it a bit easier for people to locate bug reporting info.
Change-Id: If9c8939c84ebd52eb96b353c1797afa25868bb85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303943
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested the code with the following commands.
$ ./run_tests -v
Without --depth
$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject
real 6m48.086s
user 3m27.281s
sys 1m1.386s
With --depth=1
$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject
real 2m49.637s
user 2m51.458s
sys 0m39.108s
From dwillemsen@:
"For me it's the difference between 9m28s using the current code and
16s using --depth=1 while fetching the superproject."
Bug: [google internal] b/180451672
Bug: https://crbug.com/gerrit/13707
Change-Id: I1c3e4aef4414c4e9dd259fb6e4619da0421896b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300922
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Added the following methods to XmlManifest class.
+ GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower()
+ GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func.
+ Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr.
+ Used the above methods to get groups in command.py::GetProjects
and part of init.py.
TODO: clean up these funcs to take structured group data more instead
of passing strings around everywhere that need parsing.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL
and verified prebuilts/fullsdk-linux directory has all the folders.
Tested repo init and repo sync with --use-superproject and without
--use-superproject argument.
$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync -c -j32
Bug: [google internal] b/181804931
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Superproject objects accept the optional argument “quiet”.
The following progress messages are displayed if quiet is false.
Displayed the following message whenever we find we have to make a new
folder (aka new remote), because if you started with repo init android
and later do googleplex-android that is when it will be slow.
"<location>: Performing initial setup for superproject; this might take
several minutes.".
After fetch completion, added the following notification:
"<location>: Initial setup for superproject completed."
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
Bug: [google internal] b/181178282
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
+ superproject will be fetched into a directory with the name
“<remote name>-superproject.git” instead of the current
“superproject.git” folder.
+ Deleted _Clone method and added _Init method.
+ _Init method will do “git init --bare <remote>-superproject.git”.
It will create the folder and set up a bare repository in
<remote>-superproject.git folder.
+ _Fetch method, will pass <remote url>, <branch> arguments.
Moved the --filter argument from “git clone” to “git fetch”.
_Fetch method will execute the following command to fetch
superproject. Added --no-tags argument.
master: git fetch <remote url> --force --no-tags --filter blob:none
branch: git fetch <remote url> --force --no-tags --filter blob:none \
<branch>:<branch>
+ Performance improvements for aosp-master
++ repo init performance improved from 35 seconds to 17 seconds.
++ repo init --use-superproject is around 5 to 7 secsonds slower.
++ repo sync --use-superproject is around 3 to 4 minutes faster.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real 0m20.648s
user 0m8.046s
sys 0m3.271s
+ Without superproject
$ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main
real 0m13.078s
user 0m9.783s
sys 0m2.528s
$ time repo_dev sync -c -j32 --use-superproject
...
real 15m7.072s
user 110m7.216s
sys 20m17.559s
+ Without superproject
$ time repo sync -c -j32
...
real 19m25.644s
user 91m56.331s
sys 20m59.170s
Bug: [google internal] b/180492484
Bug: [google internal] b/179470886
Bug: [google internal] b/180124069
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Added --no-use-superproject to repo and init.py to disable use of
manifest superprojects.
Replaced the term "sha" with "commit id".
Added _GetBranch method to Superproject object.
Moved shared code between init and sync into SyncSuperproject function.
This function either does git clone or git fetch. If git fetch fails
it does git clone.
Changed Superproject constructor to accept manifest, repodir and branch
to avoid passing them to multiple functions as argument.
Changed functions that were raising exceptions to return either True
or False.
Saved the --use-superproject option in config as repo.superproject.
Updated internal-fs-layout.md document.
Updated the tests to work with the new API changes in Superproject.
Performance for the first time sync has improved from 20 minutes to
around 15 minutes.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo init took around 20 seconds longer because of cloning of superproject.
$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real 0m35.919s
user 0m21.947s
sys 0m8.977s
First run
$ time repo sync --use-superproject
...
real 16m41.982s
user 100m6.916s
sys 19m18.753s
No difference in repo sync time after the first run.
Bug: [google internal] b/179090734
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Changed "git pull" to "git fetch" as we are using --bare option. Used the
following command to fetch:
git fetch origin +refs/heads/*:refs/heads/* --prune
Pass --branch argument to Superproject's UpdateProjectsRevisionId function.
Returned False/None when directories don't exist instead of raise
GitError exception from _Fetch and _LsTree functions. The caller of Fetch
does Clone if Fetch fails.
Tested the code with the following commands.
$ ./run_tests -v
Tested the init and sync code by copying all the repo changes into my Android
AOSP checkout and running repo sync with --use-superproject option.
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222
Reviewed-by: Mike Frysinger <vapier@google.com>
After updating all project’s revsionIds with the SHAs from superproject,
write the updated manifest into superproject_override.xml file. Reload
that file for future Reloads. This file is created in exp-superproject
directory.
Moved most of the code that is superproject specific into
git_superproject.py and wrote test code.
If git pull fails, did a git clone of the superproject.
We saw performance gains for consecutive repo sync's. The time to sync
went down from around 120 secs to 40 secs when repo sync is executed
consecutively.
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
First run
$ time repo sync --use-superproject
...
real 21m3.745s
user 97m59.380s
sys 19m11.286s
After two consecutive sync runs
$ time repo sync -c -j8 --use-superproject
real 0m39.626s
user 0m29.937s
sys 0m38.155s
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
+ This is without --depth option. This is done for reachability.
Server doesn't know what you know about in the history so it always
sends you the whole thing Which is very slow.
If we have the full history it can send you incremental update history
which is very small and fast.
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
.../WORKING_DIRECTORY$ repo sync --use-superproject
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I239de6d8f1c2ed6b4c69e7a78b8aa95338fa838c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/295362
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
.../WORKING_DIRECTORY$ repo sync --use-superproject
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I7e4b1e51ca1d18b836a5fa8d139a0765262ba500
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
.../WORKING_DIRECTORY$ repo sync --use-superproject
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Ieea31445ca89ba1d217e779ec7a7a2ebe81ac518
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.
Tested the code with the following commands.
$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,
.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []
Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
