gc.autoDetach is enabled by default which makes 'git gc --auto' return
immediately and run in background. This can lead to a pile up of
operations all using large amounts of memory at the same time. To avoid
this set gc.autoDetach to false so that the garbage collect task waits
for instances to finish before spawning more.
Bug: https://crbug.com/gerrit/15113
Test: repo sync -j # and check the number of 'git gc' processes
Change-Id: Ic0815156ba3db03972968f33f6f9f51e4928f23b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319835
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Some of the file removal calls are subject to race conditions (if
something else deletes the file), so extend our remove API to have
an option to ignore ENOENT errors. Then update a bunch of random
call sites to use this new functionality.
Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195
Reviewed-by: Sean McAllister <smcallis@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Added --standalone_manifest to repo tool. If set, the
manifest is downloaded directly from the appropriate source
(currently, we only support GS) and used instead of creating
a manifest git checkout. The manifests.git repo is still created to
keep track of various config but is marked as being for a standalone
manifest so that the repo tool doesn't try to run networked git
commands in it.
BUG=b:192664812
TEST=existing tests (no coverage), manual runs
Change-Id: I84378cbc7f8e515eabeccdde9665efc8cd2a9d21
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312942
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
git_trace2_event_log.py:
+ Added LogDataConfigEvents method to log 'data' events.
Sync's current_sync_state and previous_sync_state are logged
as 'data' events in the current log.
It logs are key/value in the |config| argument. Each key is
prefixed with |prefix| argument.
The following are sample events that are logged during repo sync.
{"event":"data",
"sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
"thread":"MainThread",
"time":"2021-09-14T18:16:19.935846Z",
"key":"previous_sync_state/repo.syncstate.main.synctime",
"value":"2021-09-14T17:27:11.573717Z"}
{"event":"data",
"sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
"thread":"MainThread",
"time":"2021-09-14T18:16:19.955546Z",
"key":"current_sync_state/repo.syncstate.main.synctime",
"value":"2021-09-14T18:16:19.935979Z"}
tests/test_git_trace2_event_log.py:
+ Added unit tests
sync.py:
+ Changed logging calls to LogDataConfigEvents.
Tested:
$ ./run_tests
Tested it by running the following command multiple times.
$ repo_dev sync -j 20
repo sync has finished successfully
Verified config data is looged in trace2 event logs.
Bug: [google internal] b/199758376
Change-Id: I75fd830e90c1811ec28510538c99a2632b104e85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317823
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
1) If the manifest has superproject tag (git_master, etc), then
display error/warning messages (as it is doing today)
2) If the manifest doesn't have superproject tag (nest, chromeos
manifests), then don't display any error/warning messages about
superrproject (behave as though user has specified
--no-use-superproject).
3) Print error/warning messages if --use-superproject passed as
argument to repo sync.
4) No change in behavior for the repo init command.
git_superproject.py:
+ Fixed typo in _WriteManifestFile method name
+ Superproject accepts print_message as an argument and it defaults
to True. All messages that are printed to stderr are controlled by
this flag. If it is True, then messages get printed.
+ Added PrintMessages function which return true if either
--use-superproject is specified on the command line or if the
manifest has a superproject tag.
sync.py:
+ Displays the warning message if PrintMessgages are enabled and
passes that as argument to superproject object.
+ Added 'hassuperprojecttag' trace2 log entry for analysis. We can
find users/branches that are using superproject, but the manifest is
missing the superproject tag.
Tested:
$ ./run_tests
+ Verified printing of messages with and without superproject tag, with
with --use-superproject option.
+ aosp-master
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
$ repo_dev sync
+ A manifest without superproject tag.
$ repo_dev init -m $(pwd)/manifest_7482982.xml
$ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml
Bug: [google internal] b/196411099
Change-Id: I92166dcad15a4129fab82edcf869e7c8db3efd4b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314982
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
git_config.py:
+ Added SyncAnalysisState class, which saves the following data
into the config object.
++ sys.argv, options, superproject's logging data.
++ repo.*, branch.* and remote.* parameters from config object.
++ current time as synctime.
++ Version number of the object.
+ All the keys for the above data are prepended with 'repo.syncstate.'
+ Added GetSyncAnalysisStateData and UpdateSyncAnalysisState methods
to GitConfig object to save/get the above data.
git_trace2_event_log.py:
+ Added LogConfigEvents method with code from DefParamRepoEvents
to log events.
sync.py:
+ superproject_logging_data is a dictionary that collects all the
superproject data that is to be logged as trace2 event.
+ Sync at the end logs the previously saved syncstate.* parameters
as previous_sync_state. Then it calls config's UpdateSyncAnalysisState
to save and log all the current options, superproject logged data.
docs/internal-fs-layout.md:
+ Added doc string explaining [repo.syncstate ...] sections of
.repo/manifests.git/config file.
test_git_config.py:
+ Added unit test for the new methods of GitConfig object.
Tested:
$ ./run_tests
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
Tested it by running the following command multiple times.
$ repo_dev sync -j 20
repo sync has finished successfully
Verified config file has [syncstate ...] data saved.
Bug: [google internal] b/188573450
Change-Id: I1f914ce50f3382111b72940ca56de7c41b53d460
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313123
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Skip updating the superproject when -l is present and use the existing
superproject, if available (this would make sync -l work as it's
intended to do), and fall back to sync without superproject when not
(this would catch the case when superproject is enabled by automatic
rollout).
Tested:
$ repo sync -j 20 -n
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/android/src/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Fetching: 100% (1032/1032), done in 41.184s
...
$ repo_dev sync -j 20 -l
prebuilts/asuite/: discarding 1 commits
prebuilts/runtime/: discarding 1 commits
...
repo sync has finished successfully.
+ With superproject-override.xml and test it.
$ ls -l .repo/exp-superproject/
total 176
drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
-rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 superproject_override.xml
rtenneti@rtenneti:~/work/android/src/aosp$ repo_dev sync -j 20 -l
...
repo sync has finished successfully.
+ Rename the file superproject-override.xml and test it.
$ ls -l .repo/exp-superproject/
total 176
drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
-rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 temp.xml
$ repo_dev sync -j 20 -l
Checking out: 1% (12/1031) platform/external/rust/crates/fallible-streaming-iteexternal/linux-kselftest/: discarding 1 commits
prebuilts/remoteexecution-client/: discarding 1 commits
Checking out: 51% (536/1031) platform/prebuilts/gcc/darwin-x86/aarch64/....
....
Checking out: 100% (1031/1031), done in 5.478s
repo sync has finished successfully.
Bug: [google internal] b/184368268
Change-Id: I3aba5872e4f7c299977b92c2a39847ef28698c5a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312962
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Currently we have the behavior:
* `repo`: Equivalent to `repo help` -- only shows common subcommands
(with short description), and then exits 0.
* `repo --help`: Shows repo's core options, lists all commands (no
specific info), and then exits 0.
The first case is not behaving well:
* If you run `repo` without a specific subcommand, that's an error,
so we should be exiting 1 instead.
* Showing only subcommands and no actual option summary makes it seem
like repo itself doesn't take any options. This confuses users.
Let's rework things a bit. Now we have the behavior:
* `repo`: Shows repo's core options, lists all commands (no specific
info), and then exits 1.
* `repo --help`: Shows repo's core options, shows common subcommands
(with short description), and then exits 0.
* `repo --help-all`: Shows repo's core options, shows all subcommands
(with short description), and then exits 0.
Basically we swap the behavior of `repo` and `repo --help`, and fix
the exit status when the subcommand is missing.
The addition of --help-all is mostly for the man pages. We were
relying on `repo help --all` to generate the repo(1) man page, but
that too omitted the core repo options. Now the man page includes
all the core repo options and provides a summary of all commands.
Change-Id: I1f99b99d5b8af2591f96a078d0647a3d76d6b0fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312908
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The current list output only shows project paths relative to the
root of the repo client checkout. It can be helpful to also get
a listing of paths based on other paths (e.g. the current working
directory), so add an option to repo list to support that. We'll
leverage this in bash completion to support completing projects by
their local paths and not just remote names.
Bug: https://crbug.com/gerrit/14797
Change-Id: Ia2b35d18c890217768448118b003874a1016efd4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312904
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Currently users need to look up the baseline manifest by loading the
specific manifest file. This exposes them to the internals of how the
manifest is stored which may potentially be fragile (eg: It was
switched from a symlink pointing at the file in the report to an
actual file with an 'include' tag).
Instead of doing this, we can provide an option to the 'repo manifest'
command which will emit the baseline manifest and decouple users from
the internal manifest details.
Change-Id: I12ee9160feaa591484ae71f404bc529be500ae4e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311202
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This is similar to smart sync, allowing sync to benefit from the patched
manifest.
Bug: [google internal] b/190688390
Change-Id: I158a80afceca606dcd81ec76b2caede369f7ed03
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Xin Li <delphij@google.com>
superproject is going to be default for some users. This change
doesn't fail for repo init or repo sync if source couldn't be synced
because of errors in superproject and superproject=true in the config
file. The commands will fail if --use-superproject is specified on
the command line explicitly.
The error messages are logged with trace2 event logs and will be
monitored.
+ sync - When there are errors with superproject and git_superproject
says it is fatal failure, sync will exit only when --use-superproject
option is specified on the command line.
+ init - command doesn't fail *if there are any superproject errors),
but it will print a warning message and logs message via trace2 event
logs. For fatal errors, init will exit only when --use-superproject
option is specified on the command line.
+ All git commands log the command that is being executed so trace2
event logs will know the manifest, remote url and the branch name.
There is no functional change other than fatal errors are honored with
--use-supeproject option with init/sync commands.
Tested the code with the following commands.
$ ./run_tests -v
Test 1 - sync'ing without errors
--------------------------------
Added the following lines to '~/.repoconfig/config
[repo]
superproject = true
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in /usr/local/google/home/rtenneti/work/drive2/android/test
$ repo_dev sync
remote: Total 4 (delta 1), reused 4 (delta 1)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/drive2/android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
...
Test 2 - init and sync fail when --use-superproject option is passed
--------------------------------------------------------------------
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 --use-superproject
remote: Total 57 (delta 16), reused 56 (delta 16)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning
rtenneti@rtenneti2:~/work/drive2/android/test$ repo_dev sync --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: Cannot get project commit ids from manifest
warning: Update of revisionId from superproject has failed, repo sync will not use superproject to fetch the source. Please resync with the --no-use-superproject option to avoid this repo warning.
Test 3 - git fetch command fails and git command is printed
-----------------------------------------------------------
With config change
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2
...
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/test/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Performing initial setup for superproject; this might take several minutes.
repo: error: git fetch call failed,command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in ....
Test 4 - no superproject tag
-----------------------------
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b pie-dev
...
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: .../android/pie_dev/.repo/manifest.xml
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in ...
$ repo_dev sync
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: /usr/local/google/home/rtenneti/work/drive2/android/pie_dev/.repo/manifest.xml
warning: Cannot get project commit ids from manifest
warning: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option to avoid the repo warning.
Bug: [google internal] b/192614798
Bug: [google internal] b/Bug: [google internal] b/192614798
Change-Id: I9a97a0e7d9e609fad151bd7dd9cfc523eaa887cd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311502
Reviewed-by: Amith Dsouza <amithds@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
When loading of superproject failed, we were resetting the manifest to
None, and later code would reload the manifest to see if there are
submodules, which would load the non-local manifest, causing sync with
superproject to fail.
Address this by setting the manifest_name to opt.manifest_name instead.
Bug: [google internal] b/189139268
Change-Id: I3616512e1c4b73e7eca0d83fd1fc474b825adbbf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311102
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
Repo will remember a choice and an expiration time of the choice, per
user, about whether to use superproject by default. When not specified
from command line and the choice is not expired, repo would use the
user default value.
When a user default value is not present and when the system wide
enable default is provided in git's system configuration, repo would
ask the user for a confirmation which will be valid for two weeks.
git_config.py: Add support for system config. When reading system
config, we would use --system to avoid hardcoding a path as the
value may be different on some other distributions.
git_superproject.py: Add a new subroutine, _UseSuperproject(), which
returns whether superproject should be used and whether it
is from a user configuration.
The value is determined in the following order:
1. If the user specifies either --use-superproject or
--no-use-superproject, then that choice is being used.
2. If neither is specified, we would then check the saved value
(upon repo init) and use that choice when there was a choice.
3. We then check if there is a saved and unexpired value for
user's choice in their ~/.gitconfig, and use the unexpired
choice, if available.
4. Finally, if all the above didn't give us a decision, and if
the git system configuration is providing a rollout hint, present
a prompt to user for their decision and save it in ~/.gitconfig.
subcmds/sync.py: Make use of the new UseSuperproject() provided by
git_superproject.py.
While there also silent stderr from git describe when determining the
version of repo.
Bug: [google internal] b/190688390
Change-Id: Iad3ee03026342ee500e5d65e2f0fa600d7637613
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309762
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
Don't exit if there are missing commit ids in superproject.
This change implements the following suggestion from delphij@:
"we should note the event (so we know that --use-superproject but there
were some errors, e.g. manifest didn't specify commit id for some
reason, or if there is no superproject but --use-superproject is
used), print out a message telling the use that this is not support,
but continue as if --no-use-superproject was specified?"
Changes:
superproject:
+ Added git_trace2_event_log as an argument to the constructor.
+ Sync method returns SyncResult a NamedTuple of
++ success - True if sync of superproject is successful, or False.
++ fatal - True if caller should exit, Or False.
+ UpdateProjectsRevisionId returns UpdateProjectsResult a NamedTuple of
++ manifest_path - path name of the overriding manifest file instead
of None
++ fatal - True if caller should exit, Or False
+ _GetAllProjectsCommitIds returns CommitIdsResult a NamedTuple of
++ commit_ids - a dictionary with the projects/commit ids on success,
otherwise None
++ fatal - True if caller should exit, Or False
+ Added _SkipUpdatingProjectRevisionId a helper function to see if a
project's revision id needs to be updated or not. This function is
used to exclude projects from local manifest file.
+ Added the following error events into git_trace2_event_log
++ If superproject is missing in a manifest
++ If there are missing commit ids for projects.
command.py:
+ Deleted unused import - platform
+ Added git_trace2_event_log as a member so all subcmds can log error
events.
main.py:
+ Initialized git_trace2_event_log as a member of command object.
init.py:
+ Deleted unused import - optparse
init.py:
+ Called sys.exit only if Sync returns exit=True
sync.py:
+ Called sys.exit only if Superproject's UpdateProjectsRevisionId returns
exit=True
+ Reloaded the manifest only if manifest path is returned by
UpdateProjectsRevisionId. If not, fall back to the old way of doing
repo sync.
test_git_superproject:
+ Added code to verify error events are being logged.
+ Added a test for no superproject tag
+ Added test for UpdateProjectsRevisionId not updating the revision id
with the commit ids.
Tested the code with the following commands.
+ Positive test case with aosp-master.
$ repo_dev init -u persistent-https://android.git.corp.google.com/platform/manifest -b master --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in .../android/aosp
$ repo_dev sync -j40 --use-superproject
remote: Total 12 (delta 4), reused 12 (delta 4)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
...
repo sync has finished successfully.
+ Negative test case without superproject tag.
$ repo_dev sync -j40 --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: .../android/aosp/.repo/manifest.xml
error: Cannot get project commit ids from manifest
error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
...
Checking out: 100% (1022/1022), done in 3.589s
repo sync has finished successfully.
+ Test for missing commit_id for a project.
$ repo_dev sync -j40 --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
error: please file a bug using go/repo-bug to report missing commit_ids for: ['build/blueprint']
error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
...
Checking out: 100% (1022/1022), done in 3.364s
repo sync has finished successfully.
$ ./run_tests -v
...
...== 164 passed in 2.87s ==...
Bug: [google internal] b/189371541
Change-Id: I5ea49f87e8fa41be590fc0c914573e16c8cdfcfa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309162
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Instead of setting properties on the instantiated command, pass them
via the constructor like normal objects.
Change-Id: I8787499bd2be68565875ffe243c3cf2024b36ae7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309324
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Switch it to uppercase to make it clear it's a constant, and add
documentation so its usage is clear.
Change-Id: I6d281a66a90b5908b3131585c9945e88cfe815ea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309322
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit b4429439 (sync: refactor main fetch loop) broke syncing of
submodules with sync-s="true". The first `repo sync` would just fetch
the superrepo, while the second `repo sync` would fetch the submodules.
This was due to the new _FetchMain() function not passing back the
all_projects variable that it had modified.
Change-Id: Ie8187cde7bb894e4e9a6b76c2aed83873d9f69a4
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/307065
Reviewed-by: Mike Frysinger <vapier@google.com>
We changed sync to use multiprocessing for parallel work. This broke
the ssh proxy code as it's all based on threads. Rewrite the logic to
be multiprocessing safe.
Now instead of the module acting as a stateful object, callers have to
instantiate a new ProxyManager class that holds all the state, an pass
that down to any users.
Bug: https://crbug.com/gerrit/12389
Change-Id: I4b1af116f7306b91e825d3c56fb4274c9b033562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305486
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
The only time we really need ssh proxies is when we want to run many
connections and reuse them. That only happens when running sync.
Every other command makes at most two connections, and even then it's
only one or none. So the effort of setting up & tearing down ssh
proxies isn't worth it most of the time.
The big reason we want to move this logic to sync is that it's now
using multiprocessing for parallel work. The current ssh proxy code
is all based on threads, which means none of the logic is working
correctly. The current ssh design makes it hard to fix when all of
the state lives in the global/module scope.
So the first step to fixing this is top move the setup & teardown to
the one place that really needs it: sync. No other commands will use
proxies anymore, just direct connections.
Bug: https://crbug.com/gerrit/12389
Change-Id: Ibd351acdec39a87562b3013637c5df4ea34e03c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305485
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v
$ repo_dev sync -c -j8 --no-use-superproject
Fetching: 100% (1041/1041), done in 1m22.743s
$ repo_dev sync -c -j8 --use-superproject
WARNING: --use-superproject is experimental and not for general use
..
Bug: [google internal] b/187459275
Change-Id: I3f4269df38cd24a21723e8b2be5a1f013e7b5a91
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305682
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If superproject was not available for a branch, then the next
repo sync would also fail because --use-superproject is
remembered across repo init. In such cases, hoping the hint to
to use --no-use-superproject will help.
Tested the code with the following commands and by forcing
a failure.
$ ./run_tests -v
Bug: [google internal] b/187459275
Change-Id: Ie250812b7ba83afc230b5b1d154ba11f245f8b8a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305622
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
This is a large chunk of code that is largely isolated. Move it into
a class method to make it easier to manage & reason about, and in a
follow up CL, easier to scope.
Bug: https://crbug.com/gerrit/12389
Change-Id: I0c69d95a9e03478d347b761580b2343bffa012d5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305484
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Commit b2fa30a2b8 ("sync: switch network
fetch to multiprocessing") accidentally changed the variable passed to
the 2nd fetch call from |missing| to |to_fetch| due to a copy & paste
of the earlier changed logic. Undo that to fix git submodule fetching.
Bug: https://crbug.com/gerrit/14489
Change-Id: I627954f80fd2e80d9d5809b530aa6b0ef9260abb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305262
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit fb527e3f52 ("sync: create dedicated
manifest project update func") refactored code from the main body into a
dedicated method. The manifest_name was passed as an argument, but never
used it, and instead reaches back out to the command line options. This
ignores the logic in the main loop where manifest_name might have changed
(like when using smart sync).
Change-Id: I4b84638fbb10c2b6f8f4b555e1475b0669c2daf4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305148
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We've switched most of this command over to multiprocessing and off
of _threading, so do the Event object too. The APIs are the same
between the modules, so we shouldn't need to update anything else.
Change-Id: I52d31f1c6ef2bcbe7bbc1dd1add79a8d5d08784a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305147
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We're inconsistent with help text as to whether it uses title case and
whether it ends in a period. Add a test to enforce a standard, and use
the style that Python optparse & argparse use themselves (e.g. with the
--help option): always lowercase, and never trailing period.
Change-Id: Ic1defae23daeac0ac9116aaf487427f50b34050d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305144
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
While this provides a way to undo earlier command line options (e.g.
`repo sync --tags --no-tags`) which can be helpful for scripting &
automation, this more importantly allows the user to override the
manifest settings for syncing tags from a project.
Bug: https://crbug.com/gerrit/12401
Change-Id: Id4c36cd82e6ca7cb073b5d63a09f6c7ccdebba83
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304904
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
For most commands, this is more about providing a way to undo earlier
command line options (e.g. `repo info -c --no-current-branch`) which
can be helpful for scripting & automation. But for the sync command,
this is helpful to undo the setting that exists in the manifest itself.
With this in place, tweak the sync current_branch_only logic to only
apply the manifest settings when the user hasn't specified a command
line option.
Bug: https://crbug.com/gerrit/12401
Change-Id: I21e2384624680cc740d1b5d1e49c50589d2fe6a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304903
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We're inconsistent with the short option for this flag:
* gitc-init: <none as -c is already used>
* info: -b
* init: -c
* overview: -b
* sync: -c
* upload: --cbr
Since info & overview are not as heavily used as the others, switch
them from -b to -c. We leave -b in as a hidden alias for now.
Similarly, switch upload from --cbr to just -c. A lot of people
use --cbr, so we leave this as a hidden alias for now too.
Ideally gitc-init wouldn't use -c, but that ship has sailed, and
we're more likely to deprecate gitc entirely at this point.
This provides a consistent set of options across subcommands.
Bug: https://crbug.com/gerrit/12401
Change-Id: Iec249729223866fe1ea0ebabed12ca851cc38b35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304902
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Search for project branches to upload in parallel. This can cut the
lookup time in half for large projects. We still run the actual hooks
in serial once we have the list of projects to process, but we would
need to rethink things quite a bit before we could handle running them
in parallel too.
Change-Id: I8da0cbc5010566aa860e1a158f3dc07f0709dcff
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304842
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When running upload across multiple projects, include the project in
any error messages that come up. This lets users figure out where
the problem might be.
Change-Id: I09470c9a1b512baf910d6d97b747816d1a6f3a87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304783
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Currently, copyfiles and linkfiles which marked by
"<copyfile/>" and "<linkfile/>" in manifest will
be created by first exec 'repo sync'.
But if some "<copyfile/>" or "<linkfile/>" are removed
in manifest, then 'repo sync', these removed item
dest can not be removed in the sourcecode workspace.
This patch is intent to fix this issue, by save a
'copy-link-files.json' in .repo and then compared with
new dest path when next sync. If any "<copyfile/>" or
"<linkfile/>" were removed, the dest path will be
removed in sourcecode at the same time.
Bug: https://crbug.com/gerrit/11008
Change-Id: I6b7b41e94df0f9e6e52801ec755951a4c572d05d
Signed-off-by: jiajia tang <tangjiajia@xiaomi.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304202
Reviewed-by: Mike Frysinger <vapier@google.com>
Make it a bit easier for people to locate bug reporting info.
Change-Id: If9c8939c84ebd52eb96b353c1797afa25868bb85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303943
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Some refactors during review dropped this import when it was reworked,
but it's still needed when using the --quiet setting.
Change-Id: I6d9302ef5a056e52415ea63f35bad592b9dfa75d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303942
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
In _FetchOne & _CheckOne, only print error.GitError exception,
but other exceptions are still thrown
Fixes the GitError exceptions from /usr/lib/python3.8/multiprocessing/pool.py
exiting the repo sync.
Tested the code with the following commands and verified repo sync
continues after fetch error because of an invalid SHA1.
$ ./run_tests -v
$ python3 ~/work/repo/git-repo/repo sync -m manifest_P21623846.xml -j32
...
error.GitError: Cannot fetch platform/vendor/google_devices/redbull/proprietary update-ref: fatal: d5a99e518f09d6abb0c0dfa899594e1ea6232459^0: not a valid SHA1
....
An error like the following when jobs=1
error.GitError: Cannot checkout platform/vendor/qcom/sdm845/proprietary/qcrilOemHook: Cannot initialize work tree for platform/vendor/qcom/sdm845/proprietary/qcrilOemHook
Bug: https://crbug.com/gerrit/14392
Change-Id: I8922ad6c07c733125419f5698b0f7e32d70c7905
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303544
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Now that we have a bunch of subcommands doing parallel execution, a
common pattern arises that we can factor out for most of them. We
leave forall alone as it's a bit too complicated atm to cut over.
Change-Id: I3617a4f7c66142bcd1ab030cb4cca698a65010ac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301942
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Since Python has deprecated the formatter module, switch to the textwrap
module instead for reflowing text. We weren't really using any other
feature anyways.
Verified by diffing the output before & after the change and making sure
it was the same.
Then made a few tweaks to tighten up the output.
Change-Id: I0be1bc2a6661a311b1a4693c80d0f8366320ba55
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303282
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Currently, list only shows projects that exist in the checkout, and
doesn't offer any way to list all projects in the manifest (based on
the current settings, or on the options passed to list). This seems
to be the opposite of what (at least some) users expect, so let's
add an option to show all of them regardless of checkout state.
Change-Id: I94bbdc5bd0ff2a411704fa215e7fc2b60fa3360e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301263
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We want progress bars in the default output mode, but not when the
user specifies --quiet. Add a setting to the Progress bar class so
it takes care of not displaying anything itself rather than having
to update every subcommand to conditionally setup & call the object.
Change-Id: I1134993bffc5437bc22e26be11a512125f10597f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303225
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Add new CommonOptions entry points to move the existing --jobs to,
and relocate all --verbose/--quiet options to that. This provides
both a consistent interface for users as well as for code.
Change-Id: Ifaf83b88872421f4749b073c472b4a67ca6c0437
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303224
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
partial-clone-exclude option excludes projects during
partial clone. This is a comma-delimited project names
(from manifest.xml). This option is persisted and it
is used by the sync command.
A project that has been unparital'ed will remain unpartial if
that project's name is specified in the --partial-clone-exclude
option. The project name should match exactly.
Added
$ ./run_tests -v
Bug: [google internal] b/175712967
"I can't "unpartial" my androidx-main checkout"
$ rm -rf androidx-main/
$ mkdir androidx-main/
$ cd androidx-main/
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml
$ repo_dev sync -c -j8
+ Verify a project is partial
$ cd frameworks/support/
$ git config -l | grep 'partial'
+ Unpartial a project.
$ /google/bin/releases/android/git_repack/git_unpartial
+ Verify project is unpartial
$ git config -l | grep 'partial'
$ cd ../..
+ Exclude the project from being unparial'ed after init and sync.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml
+ Verify project is unpartial
$ cd frameworks/support/
$ git config -l | grep 'partial'
$ cd ../..
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep 'partial'
$ cd ../..
+ Remove the project from exclude list and verify that project is partially cloned.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep 'partial'
Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The number of jobs one wants to run against the network tends to
factor differently from the number of jobs one wants to run when
checking out local projects. The former is constrained by your
internet connection & server limits while the later is constrained
by your local computer's CPU & storage I/O. People with beefier
computers probably want to keep the network/server jobs bounded a
bit lower than the local/checkout jobs.
Change-Id: Ia27ab682c62c09d244a8a1427b1c65acf0116c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302804
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This can take a few seconds, if not a lot more, so add a progress bar
so users understand what's going on.
Change-Id: I5b4b54c1bbb9ec18728f979521310f7087afaa5c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The current logic has a downside in that it doesn't sync to the latest
signed version available if the latest commit itself is unsigned. This
can come up when using the "main" branch as it is sometimes signed, but
often not as it's holding the latest merged commits. When people use
the main branch, it's to get early testing on versions tagged but not
yet released, and we don't want them to get stuck indefinitely on that
old version of repo.
For example, this series of events:
* "stable" is at v2.12.
* "main" is tagged with v2.13.
* early testers use --repo-rev main to get v2.13.
* new commits are merged to "main".
* "main" is tagged with v2.14.
* new commits are merged to "main".
* devs who had synced in the past to test v2.13 are stuck on v2.13.
repo sees "main" is unsigned and so doesn't try to upgrade at all.
The only way to get unwedged is to re-run `repo init --repo-rev main`,
or to manually sync once with repo verification disabled, or for us to
leave "main" signed for a while and hope devs will sync in that window.
The new logic is that whenever changes are available, we switch to the
latest signed tag. We also replace some of the duplicated verification
code in the sync command with the newer wrapper logic. This handles a
couple of important scenarios inaddition to above:
* rollback (e.g. v2.13.8 -> v2.13.7)
* do not trash uncommitted changes (in case of ad-hoc testing)
* switch tag histories (e.g. v2.13.8 -> v2.13.8-cr1)
Change-Id: I5b45ba1dd26a7c582700ee3711f303dc7538579b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300122
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
These are manually kept in sync which is a pain. Have the init
subcmd reuse the wrapper code directly.
Change-Id: Ica73211422c64377bacc9bb3b1d1a8d9d5f7f4ca
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302762
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
