git_config.py:
+ Added SyncAnalysisState class, which saves the following data
into the config object.
++ sys.argv, options, superproject's logging data.
++ repo.*, branch.* and remote.* parameters from config object.
++ current time as synctime.
++ Version number of the object.
+ All the keys for the above data are prepended with 'repo.syncstate.'
+ Added GetSyncAnalysisStateData and UpdateSyncAnalysisState methods
to GitConfig object to save/get the above data.
git_trace2_event_log.py:
+ Added LogConfigEvents method with code from DefParamRepoEvents
to log events.
sync.py:
+ superproject_logging_data is a dictionary that collects all the
superproject data that is to be logged as trace2 event.
+ Sync at the end logs the previously saved syncstate.* parameters
as previous_sync_state. Then it calls config's UpdateSyncAnalysisState
to save and log all the current options, superproject logged data.
docs/internal-fs-layout.md:
+ Added doc string explaining [repo.syncstate ...] sections of
.repo/manifests.git/config file.
test_git_config.py:
+ Added unit test for the new methods of GitConfig object.
Tested:
$ ./run_tests
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
Tested it by running the following command multiple times.
$ repo_dev sync -j 20
repo sync has finished successfully
Verified config file has [syncstate ...] data saved.
Bug: [google internal] b/188573450
Change-Id: I1f914ce50f3382111b72940ca56de7c41b53d460
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313123
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Skip updating the superproject when -l is present and use the existing
superproject, if available (this would make sync -l work as it's
intended to do), and fall back to sync without superproject when not
(this would catch the case when superproject is enabled by automatic
rollout).
Tested:
$ repo sync -j 20 -n
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/android/src/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Fetching: 100% (1032/1032), done in 41.184s
...
$ repo_dev sync -j 20 -l
prebuilts/asuite/: discarding 1 commits
prebuilts/runtime/: discarding 1 commits
...
repo sync has finished successfully.
+ With superproject-override.xml and test it.
$ ls -l .repo/exp-superproject/
total 176
drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
-rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 superproject_override.xml
rtenneti@rtenneti:~/work/android/src/aosp$ repo_dev sync -j 20 -l
...
repo sync has finished successfully.
+ Rename the file superproject-override.xml and test it.
$ ls -l .repo/exp-superproject/
total 176
drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
-rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 temp.xml
$ repo_dev sync -j 20 -l
Checking out: 1% (12/1031) platform/external/rust/crates/fallible-streaming-iteexternal/linux-kselftest/: discarding 1 commits
prebuilts/remoteexecution-client/: discarding 1 commits
Checking out: 51% (536/1031) platform/prebuilts/gcc/darwin-x86/aarch64/....
....
Checking out: 100% (1031/1031), done in 5.478s
repo sync has finished successfully.
Bug: [google internal] b/184368268
Change-Id: I3aba5872e4f7c299977b92c2a39847ef28698c5a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312962
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Currently we have the behavior:
* `repo`: Equivalent to `repo help` -- only shows common subcommands
(with short description), and then exits 0.
* `repo --help`: Shows repo's core options, lists all commands (no
specific info), and then exits 0.
The first case is not behaving well:
* If you run `repo` without a specific subcommand, that's an error,
so we should be exiting 1 instead.
* Showing only subcommands and no actual option summary makes it seem
like repo itself doesn't take any options. This confuses users.
Let's rework things a bit. Now we have the behavior:
* `repo`: Shows repo's core options, lists all commands (no specific
info), and then exits 1.
* `repo --help`: Shows repo's core options, shows common subcommands
(with short description), and then exits 0.
* `repo --help-all`: Shows repo's core options, shows all subcommands
(with short description), and then exits 0.
Basically we swap the behavior of `repo` and `repo --help`, and fix
the exit status when the subcommand is missing.
The addition of --help-all is mostly for the man pages. We were
relying on `repo help --all` to generate the repo(1) man page, but
that too omitted the core repo options. Now the man page includes
all the core repo options and provides a summary of all commands.
Change-Id: I1f99b99d5b8af2591f96a078d0647a3d76d6b0fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312908
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The current list output only shows project paths relative to the
root of the repo client checkout. It can be helpful to also get
a listing of paths based on other paths (e.g. the current working
directory), so add an option to repo list to support that. We'll
leverage this in bash completion to support completing projects by
their local paths and not just remote names.
Bug: https://crbug.com/gerrit/14797
Change-Id: Ia2b35d18c890217768448118b003874a1016efd4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312904
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Currently users need to look up the baseline manifest by loading the
specific manifest file. This exposes them to the internals of how the
manifest is stored which may potentially be fragile (eg: It was
switched from a symlink pointing at the file in the report to an
actual file with an 'include' tag).
Instead of doing this, we can provide an option to the 'repo manifest'
command which will emit the baseline manifest and decouple users from
the internal manifest details.
Change-Id: I12ee9160feaa591484ae71f404bc529be500ae4e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311202
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This is similar to smart sync, allowing sync to benefit from the patched
manifest.
Bug: [google internal] b/190688390
Change-Id: I158a80afceca606dcd81ec76b2caede369f7ed03
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Xin Li <delphij@google.com>
superproject is going to be default for some users. This change
doesn't fail for repo init or repo sync if source couldn't be synced
because of errors in superproject and superproject=true in the config
file. The commands will fail if --use-superproject is specified on
the command line explicitly.
The error messages are logged with trace2 event logs and will be
monitored.
+ sync - When there are errors with superproject and git_superproject
says it is fatal failure, sync will exit only when --use-superproject
option is specified on the command line.
+ init - command doesn't fail *if there are any superproject errors),
but it will print a warning message and logs message via trace2 event
logs. For fatal errors, init will exit only when --use-superproject
option is specified on the command line.
+ All git commands log the command that is being executed so trace2
event logs will know the manifest, remote url and the branch name.
There is no functional change other than fatal errors are honored with
--use-supeproject option with init/sync commands.
Tested the code with the following commands.
$ ./run_tests -v
Test 1 - sync'ing without errors
--------------------------------
Added the following lines to '~/.repoconfig/config
[repo]
superproject = true
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in /usr/local/google/home/rtenneti/work/drive2/android/test
$ repo_dev sync
remote: Total 4 (delta 1), reused 4 (delta 1)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/drive2/android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
...
Test 2 - init and sync fail when --use-superproject option is passed
--------------------------------------------------------------------
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 --use-superproject
remote: Total 57 (delta 16), reused 56 (delta 16)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning
rtenneti@rtenneti2:~/work/drive2/android/test$ repo_dev sync --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: Cannot get project commit ids from manifest
warning: Update of revisionId from superproject has failed, repo sync will not use superproject to fetch the source. Please resync with the --no-use-superproject option to avoid this repo warning.
Test 3 - git fetch command fails and git command is printed
-----------------------------------------------------------
With config change
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2
...
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/test/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Performing initial setup for superproject; this might take several minutes.
repo: error: git fetch call failed,command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in ....
Test 4 - no superproject tag
-----------------------------
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b pie-dev
...
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: .../android/pie_dev/.repo/manifest.xml
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in ...
$ repo_dev sync
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: /usr/local/google/home/rtenneti/work/drive2/android/pie_dev/.repo/manifest.xml
warning: Cannot get project commit ids from manifest
warning: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option to avoid the repo warning.
Bug: [google internal] b/192614798
Bug: [google internal] b/Bug: [google internal] b/192614798
Change-Id: I9a97a0e7d9e609fad151bd7dd9cfc523eaa887cd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311502
Reviewed-by: Amith Dsouza <amithds@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
When loading of superproject failed, we were resetting the manifest to
None, and later code would reload the manifest to see if there are
submodules, which would load the non-local manifest, causing sync with
superproject to fail.
Address this by setting the manifest_name to opt.manifest_name instead.
Bug: [google internal] b/189139268
Change-Id: I3616512e1c4b73e7eca0d83fd1fc474b825adbbf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311102
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
Repo will remember a choice and an expiration time of the choice, per
user, about whether to use superproject by default. When not specified
from command line and the choice is not expired, repo would use the
user default value.
When a user default value is not present and when the system wide
enable default is provided in git's system configuration, repo would
ask the user for a confirmation which will be valid for two weeks.
git_config.py: Add support for system config. When reading system
config, we would use --system to avoid hardcoding a path as the
value may be different on some other distributions.
git_superproject.py: Add a new subroutine, _UseSuperproject(), which
returns whether superproject should be used and whether it
is from a user configuration.
The value is determined in the following order:
1. If the user specifies either --use-superproject or
--no-use-superproject, then that choice is being used.
2. If neither is specified, we would then check the saved value
(upon repo init) and use that choice when there was a choice.
3. We then check if there is a saved and unexpired value for
user's choice in their ~/.gitconfig, and use the unexpired
choice, if available.
4. Finally, if all the above didn't give us a decision, and if
the git system configuration is providing a rollout hint, present
a prompt to user for their decision and save it in ~/.gitconfig.
subcmds/sync.py: Make use of the new UseSuperproject() provided by
git_superproject.py.
While there also silent stderr from git describe when determining the
version of repo.
Bug: [google internal] b/190688390
Change-Id: Iad3ee03026342ee500e5d65e2f0fa600d7637613
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309762
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
Don't exit if there are missing commit ids in superproject.
This change implements the following suggestion from delphij@:
"we should note the event (so we know that --use-superproject but there
were some errors, e.g. manifest didn't specify commit id for some
reason, or if there is no superproject but --use-superproject is
used), print out a message telling the use that this is not support,
but continue as if --no-use-superproject was specified?"
Changes:
superproject:
+ Added git_trace2_event_log as an argument to the constructor.
+ Sync method returns SyncResult a NamedTuple of
++ success - True if sync of superproject is successful, or False.
++ fatal - True if caller should exit, Or False.
+ UpdateProjectsRevisionId returns UpdateProjectsResult a NamedTuple of
++ manifest_path - path name of the overriding manifest file instead
of None
++ fatal - True if caller should exit, Or False
+ _GetAllProjectsCommitIds returns CommitIdsResult a NamedTuple of
++ commit_ids - a dictionary with the projects/commit ids on success,
otherwise None
++ fatal - True if caller should exit, Or False
+ Added _SkipUpdatingProjectRevisionId a helper function to see if a
project's revision id needs to be updated or not. This function is
used to exclude projects from local manifest file.
+ Added the following error events into git_trace2_event_log
++ If superproject is missing in a manifest
++ If there are missing commit ids for projects.
command.py:
+ Deleted unused import - platform
+ Added git_trace2_event_log as a member so all subcmds can log error
events.
main.py:
+ Initialized git_trace2_event_log as a member of command object.
init.py:
+ Deleted unused import - optparse
init.py:
+ Called sys.exit only if Sync returns exit=True
sync.py:
+ Called sys.exit only if Superproject's UpdateProjectsRevisionId returns
exit=True
+ Reloaded the manifest only if manifest path is returned by
UpdateProjectsRevisionId. If not, fall back to the old way of doing
repo sync.
test_git_superproject:
+ Added code to verify error events are being logged.
+ Added a test for no superproject tag
+ Added test for UpdateProjectsRevisionId not updating the revision id
with the commit ids.
Tested the code with the following commands.
+ Positive test case with aosp-master.
$ repo_dev init -u persistent-https://android.git.corp.google.com/platform/manifest -b master --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in .../android/aosp
$ repo_dev sync -j40 --use-superproject
remote: Total 12 (delta 4), reused 12 (delta 4)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
...
repo sync has finished successfully.
+ Negative test case without superproject tag.
$ repo_dev sync -j40 --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: .../android/aosp/.repo/manifest.xml
error: Cannot get project commit ids from manifest
error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
...
Checking out: 100% (1022/1022), done in 3.589s
repo sync has finished successfully.
+ Test for missing commit_id for a project.
$ repo_dev sync -j40 --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
error: please file a bug using go/repo-bug to report missing commit_ids for: ['build/blueprint']
error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
...
Checking out: 100% (1022/1022), done in 3.364s
repo sync has finished successfully.
$ ./run_tests -v
...
...== 164 passed in 2.87s ==...
Bug: [google internal] b/189371541
Change-Id: I5ea49f87e8fa41be590fc0c914573e16c8cdfcfa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309162
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Instead of setting properties on the instantiated command, pass them
via the constructor like normal objects.
Change-Id: I8787499bd2be68565875ffe243c3cf2024b36ae7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309324
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Switch it to uppercase to make it clear it's a constant, and add
documentation so its usage is clear.
Change-Id: I6d281a66a90b5908b3131585c9945e88cfe815ea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309322
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit b4429439 (sync: refactor main fetch loop) broke syncing of
submodules with sync-s="true". The first `repo sync` would just fetch
the superrepo, while the second `repo sync` would fetch the submodules.
This was due to the new _FetchMain() function not passing back the
all_projects variable that it had modified.
Change-Id: Ie8187cde7bb894e4e9a6b76c2aed83873d9f69a4
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/307065
Reviewed-by: Mike Frysinger <vapier@google.com>
We changed sync to use multiprocessing for parallel work. This broke
the ssh proxy code as it's all based on threads. Rewrite the logic to
be multiprocessing safe.
Now instead of the module acting as a stateful object, callers have to
instantiate a new ProxyManager class that holds all the state, an pass
that down to any users.
Bug: https://crbug.com/gerrit/12389
Change-Id: I4b1af116f7306b91e825d3c56fb4274c9b033562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305486
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
The only time we really need ssh proxies is when we want to run many
connections and reuse them. That only happens when running sync.
Every other command makes at most two connections, and even then it's
only one or none. So the effort of setting up & tearing down ssh
proxies isn't worth it most of the time.
The big reason we want to move this logic to sync is that it's now
using multiprocessing for parallel work. The current ssh proxy code
is all based on threads, which means none of the logic is working
correctly. The current ssh design makes it hard to fix when all of
the state lives in the global/module scope.
So the first step to fixing this is top move the setup & teardown to
the one place that really needs it: sync. No other commands will use
proxies anymore, just direct connections.
Bug: https://crbug.com/gerrit/12389
Change-Id: Ibd351acdec39a87562b3013637c5df4ea34e03c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305485
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v
$ repo_dev sync -c -j8 --no-use-superproject
Fetching: 100% (1041/1041), done in 1m22.743s
$ repo_dev sync -c -j8 --use-superproject
WARNING: --use-superproject is experimental and not for general use
..
Bug: [google internal] b/187459275
Change-Id: I3f4269df38cd24a21723e8b2be5a1f013e7b5a91
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305682
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If superproject was not available for a branch, then the next
repo sync would also fail because --use-superproject is
remembered across repo init. In such cases, hoping the hint to
to use --no-use-superproject will help.
Tested the code with the following commands and by forcing
a failure.
$ ./run_tests -v
Bug: [google internal] b/187459275
Change-Id: Ie250812b7ba83afc230b5b1d154ba11f245f8b8a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305622
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
This is a large chunk of code that is largely isolated. Move it into
a class method to make it easier to manage & reason about, and in a
follow up CL, easier to scope.
Bug: https://crbug.com/gerrit/12389
Change-Id: I0c69d95a9e03478d347b761580b2343bffa012d5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305484
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Commit b2fa30a2b8 ("sync: switch network
fetch to multiprocessing") accidentally changed the variable passed to
the 2nd fetch call from |missing| to |to_fetch| due to a copy & paste
of the earlier changed logic. Undo that to fix git submodule fetching.
Bug: https://crbug.com/gerrit/14489
Change-Id: I627954f80fd2e80d9d5809b530aa6b0ef9260abb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305262
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit fb527e3f52 ("sync: create dedicated
manifest project update func") refactored code from the main body into a
dedicated method. The manifest_name was passed as an argument, but never
used it, and instead reaches back out to the command line options. This
ignores the logic in the main loop where manifest_name might have changed
(like when using smart sync).
Change-Id: I4b84638fbb10c2b6f8f4b555e1475b0669c2daf4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305148
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We've switched most of this command over to multiprocessing and off
of _threading, so do the Event object too. The APIs are the same
between the modules, so we shouldn't need to update anything else.
Change-Id: I52d31f1c6ef2bcbe7bbc1dd1add79a8d5d08784a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305147
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We're inconsistent with help text as to whether it uses title case and
whether it ends in a period. Add a test to enforce a standard, and use
the style that Python optparse & argparse use themselves (e.g. with the
--help option): always lowercase, and never trailing period.
Change-Id: Ic1defae23daeac0ac9116aaf487427f50b34050d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305144
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
While this provides a way to undo earlier command line options (e.g.
`repo sync --tags --no-tags`) which can be helpful for scripting &
automation, this more importantly allows the user to override the
manifest settings for syncing tags from a project.
Bug: https://crbug.com/gerrit/12401
Change-Id: Id4c36cd82e6ca7cb073b5d63a09f6c7ccdebba83
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304904
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
For most commands, this is more about providing a way to undo earlier
command line options (e.g. `repo info -c --no-current-branch`) which
can be helpful for scripting & automation. But for the sync command,
this is helpful to undo the setting that exists in the manifest itself.
With this in place, tweak the sync current_branch_only logic to only
apply the manifest settings when the user hasn't specified a command
line option.
Bug: https://crbug.com/gerrit/12401
Change-Id: I21e2384624680cc740d1b5d1e49c50589d2fe6a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304903
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We're inconsistent with the short option for this flag:
* gitc-init: <none as -c is already used>
* info: -b
* init: -c
* overview: -b
* sync: -c
* upload: --cbr
Since info & overview are not as heavily used as the others, switch
them from -b to -c. We leave -b in as a hidden alias for now.
Similarly, switch upload from --cbr to just -c. A lot of people
use --cbr, so we leave this as a hidden alias for now too.
Ideally gitc-init wouldn't use -c, but that ship has sailed, and
we're more likely to deprecate gitc entirely at this point.
This provides a consistent set of options across subcommands.
Bug: https://crbug.com/gerrit/12401
Change-Id: Iec249729223866fe1ea0ebabed12ca851cc38b35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304902
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Search for project branches to upload in parallel. This can cut the
lookup time in half for large projects. We still run the actual hooks
in serial once we have the list of projects to process, but we would
need to rethink things quite a bit before we could handle running them
in parallel too.
Change-Id: I8da0cbc5010566aa860e1a158f3dc07f0709dcff
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304842
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When running upload across multiple projects, include the project in
any error messages that come up. This lets users figure out where
the problem might be.
Change-Id: I09470c9a1b512baf910d6d97b747816d1a6f3a87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304783
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Currently, copyfiles and linkfiles which marked by
"<copyfile/>" and "<linkfile/>" in manifest will
be created by first exec 'repo sync'.
But if some "<copyfile/>" or "<linkfile/>" are removed
in manifest, then 'repo sync', these removed item
dest can not be removed in the sourcecode workspace.
This patch is intent to fix this issue, by save a
'copy-link-files.json' in .repo and then compared with
new dest path when next sync. If any "<copyfile/>" or
"<linkfile/>" were removed, the dest path will be
removed in sourcecode at the same time.
Bug: https://crbug.com/gerrit/11008
Change-Id: I6b7b41e94df0f9e6e52801ec755951a4c572d05d
Signed-off-by: jiajia tang <tangjiajia@xiaomi.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304202
Reviewed-by: Mike Frysinger <vapier@google.com>
Make it a bit easier for people to locate bug reporting info.
Change-Id: If9c8939c84ebd52eb96b353c1797afa25868bb85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303943
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Some refactors during review dropped this import when it was reworked,
but it's still needed when using the --quiet setting.
Change-Id: I6d9302ef5a056e52415ea63f35bad592b9dfa75d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303942
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
In _FetchOne & _CheckOne, only print error.GitError exception,
but other exceptions are still thrown
Fixes the GitError exceptions from /usr/lib/python3.8/multiprocessing/pool.py
exiting the repo sync.
Tested the code with the following commands and verified repo sync
continues after fetch error because of an invalid SHA1.
$ ./run_tests -v
$ python3 ~/work/repo/git-repo/repo sync -m manifest_P21623846.xml -j32
...
error.GitError: Cannot fetch platform/vendor/google_devices/redbull/proprietary update-ref: fatal: d5a99e518f09d6abb0c0dfa899594e1ea6232459^0: not a valid SHA1
....
An error like the following when jobs=1
error.GitError: Cannot checkout platform/vendor/qcom/sdm845/proprietary/qcrilOemHook: Cannot initialize work tree for platform/vendor/qcom/sdm845/proprietary/qcrilOemHook
Bug: https://crbug.com/gerrit/14392
Change-Id: I8922ad6c07c733125419f5698b0f7e32d70c7905
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303544
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Now that we have a bunch of subcommands doing parallel execution, a
common pattern arises that we can factor out for most of them. We
leave forall alone as it's a bit too complicated atm to cut over.
Change-Id: I3617a4f7c66142bcd1ab030cb4cca698a65010ac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301942
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Since Python has deprecated the formatter module, switch to the textwrap
module instead for reflowing text. We weren't really using any other
feature anyways.
Verified by diffing the output before & after the change and making sure
it was the same.
Then made a few tweaks to tighten up the output.
Change-Id: I0be1bc2a6661a311b1a4693c80d0f8366320ba55
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303282
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Currently, list only shows projects that exist in the checkout, and
doesn't offer any way to list all projects in the manifest (based on
the current settings, or on the options passed to list). This seems
to be the opposite of what (at least some) users expect, so let's
add an option to show all of them regardless of checkout state.
Change-Id: I94bbdc5bd0ff2a411704fa215e7fc2b60fa3360e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301263
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We want progress bars in the default output mode, but not when the
user specifies --quiet. Add a setting to the Progress bar class so
it takes care of not displaying anything itself rather than having
to update every subcommand to conditionally setup & call the object.
Change-Id: I1134993bffc5437bc22e26be11a512125f10597f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303225
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Add new CommonOptions entry points to move the existing --jobs to,
and relocate all --verbose/--quiet options to that. This provides
both a consistent interface for users as well as for code.
Change-Id: Ifaf83b88872421f4749b073c472b4a67ca6c0437
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303224
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
partial-clone-exclude option excludes projects during
partial clone. This is a comma-delimited project names
(from manifest.xml). This option is persisted and it
is used by the sync command.
A project that has been unparital'ed will remain unpartial if
that project's name is specified in the --partial-clone-exclude
option. The project name should match exactly.
Added
$ ./run_tests -v
Bug: [google internal] b/175712967
"I can't "unpartial" my androidx-main checkout"
$ rm -rf androidx-main/
$ mkdir androidx-main/
$ cd androidx-main/
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml
$ repo_dev sync -c -j8
+ Verify a project is partial
$ cd frameworks/support/
$ git config -l | grep 'partial'
+ Unpartial a project.
$ /google/bin/releases/android/git_repack/git_unpartial
+ Verify project is unpartial
$ git config -l | grep 'partial'
$ cd ../..
+ Exclude the project from being unparial'ed after init and sync.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml
+ Verify project is unpartial
$ cd frameworks/support/
$ git config -l | grep 'partial'
$ cd ../..
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep 'partial'
$ cd ../..
+ Remove the project from exclude list and verify that project is partially cloned.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep 'partial'
Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The number of jobs one wants to run against the network tends to
factor differently from the number of jobs one wants to run when
checking out local projects. The former is constrained by your
internet connection & server limits while the later is constrained
by your local computer's CPU & storage I/O. People with beefier
computers probably want to keep the network/server jobs bounded a
bit lower than the local/checkout jobs.
Change-Id: Ia27ab682c62c09d244a8a1427b1c65acf0116c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302804
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This can take a few seconds, if not a lot more, so add a progress bar
so users understand what's going on.
Change-Id: I5b4b54c1bbb9ec18728f979521310f7087afaa5c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The current logic has a downside in that it doesn't sync to the latest
signed version available if the latest commit itself is unsigned. This
can come up when using the "main" branch as it is sometimes signed, but
often not as it's holding the latest merged commits. When people use
the main branch, it's to get early testing on versions tagged but not
yet released, and we don't want them to get stuck indefinitely on that
old version of repo.
For example, this series of events:
* "stable" is at v2.12.
* "main" is tagged with v2.13.
* early testers use --repo-rev main to get v2.13.
* new commits are merged to "main".
* "main" is tagged with v2.14.
* new commits are merged to "main".
* devs who had synced in the past to test v2.13 are stuck on v2.13.
repo sees "main" is unsigned and so doesn't try to upgrade at all.
The only way to get unwedged is to re-run `repo init --repo-rev main`,
or to manually sync once with repo verification disabled, or for us to
leave "main" signed for a while and hope devs will sync in that window.
The new logic is that whenever changes are available, we switch to the
latest signed tag. We also replace some of the duplicated verification
code in the sync command with the newer wrapper logic. This handles a
couple of important scenarios inaddition to above:
* rollback (e.g. v2.13.8 -> v2.13.7)
* do not trash uncommitted changes (in case of ad-hoc testing)
* switch tag histories (e.g. v2.13.8 -> v2.13.8-cr1)
Change-Id: I5b45ba1dd26a7c582700ee3711f303dc7538579b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300122
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
These are manually kept in sync which is a pain. Have the init
subcmd reuse the wrapper code directly.
Change-Id: Ica73211422c64377bacc9bb3b1d1a8d9d5f7f4ca
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302762
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This avoids GIL limitations with using threads for parallel processing.
This reworks the fetch logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.
We have to tweak the chunking logic a little here because multiprocessing
favors batching over returning immediate results when using a larger value
for chunksize. When a single job can be quite slow, this tradeoff is not
good UX.
Bug: https://crbug.com/gerrit/12389
Change-Id: I0f0512d15ad7332d1eb28aff52c29d378acc9e1d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298642
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~40sec to ~16sec with the default -j8.
The output processing does not appear to be a significant bottle
neck -- it accounts for <1sec out of the ~16sec runtime. Thus we
leave it in the main thread to simplify the code.
Change-Id: I750b72c7711b0c5d26e65d480738fbaac3a69971
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297984
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~10sec to ~4sec with the default -j8.
This only does a simple conversion over to get an easy speedup. It
is currently written to collect all results before displaying them.
If we refactored this module more, we could have it display results
as they came in.
Change-Id: I5caf4ca51df0b7f078f0db104ae5232268482c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298643
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the project exists, but the ref the manifest wants doesn't exist,
don't throw an error (and abort the process in general). This can
come up with a partially synced tree: the manifest is up-to-date,
but not all the projects have yet been synced.
Bug: https://crbug.com/gerrit/14289
Change-Id: Iba97413c476544223ffe518198c900c2193a00ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301262
Reviewed-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v
Bug: [google internal] b/183232698
Bug: https://crbug.com/gerrit/13707
$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync --use-superproject
$ repo_dev sync
real 0m8.046s
user 0m2.866s
sys 0m2.457s
Second time repo sync took only 8 seconds and verified by printing that
urrent_branch_only is True in project.py's Sync_NetworkHalf function.
Change-Id: Ic48efb23ea427dfa36e12a5c49973d6ae776d818
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301182
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This makes it easy to use --H as a shortcut, and kind of matches the
use of storing HEAD as the revision.
Change-Id: I590bf488518f313e7a593853140316df98262d7e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301163
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Historically forall has been interactive since it ran in serial.
Recent rework in here dropped that to enable parallel processing.
Restore support for interactive commands when running -j1 or with
an explicit --interactive option.
Bug: https://crbug.com/gerrit/14256
Change-Id: I502007186f771914cfd7830846a4e1938b5e1f38
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300722
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The refactor to multiprocessing broke status reporting slightly when
checking out projects. Make sure we mark the step as failed if any
of the projects failed, not just when --fail-fast is set.
Change-Id: I0efb56ce83b068b2c334046df3fef23d797599c9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299882
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Added the following methods to XmlManifest class.
+ GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower()
+ GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func.
+ Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr.
+ Used the above methods to get groups in command.py::GetProjects
and part of init.py.
TODO: clean up these funcs to take structured group data more instead
of passing strings around everywhere that need parsing.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL
and verified prebuilts/fullsdk-linux directory has all the folders.
Tested repo init and repo sync with --use-superproject and without
--use-superproject argument.
$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync -c -j32
Bug: [google internal] b/181804931
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
There's a few rough edges here still, but no known corruption ones,
so open it up a bit for people to experiment with.
Bug: https://crbug.com/gerrit/11486
Change-Id: I81e0122ab6d3e032c546c8239dd4f03740676e80
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299242
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Superproject objects accept the optional argument “quiet”.
The following progress messages are displayed if quiet is false.
Displayed the following message whenever we find we have to make a new
folder (aka new remote), because if you started with repo init android
and later do googleplex-android that is when it will be slow.
"<location>: Performing initial setup for superproject; this might take
several minutes.".
After fetch completion, added the following notification:
"<location>: Initial setup for superproject completed."
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
Bug: [google internal] b/181178282
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
With the recent commit 0501b29e7a
("status: Use multiprocessing for `repo status -j<num>` instead of
threading"), the limitation with project serialization no longer
applies. It turns out that ad-hoc logic is expensive. In the CrOS
checkout (~1000 projects w/8 jobs by default), it adds about ~7sec
overhead to all invocations. With a fast nop run:
time repo forall -j8 -c true
This goes from ~11sec to ~4sec -- more than 50% speedup.
Change-Id: Ie6bcccd21eef20440692751b7ebd36c890d5bbcc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298724
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The ProjectArgs function can be inlined which simplifies it quite a
bit. We shouldn't need the custom exception handling here either.
This also makes the next commit easier to review.
Change-Id: If3be04f58c302c36a0f20b99de0f67e78beac141
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298723
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This is in preparation for simplifying the jobs support. The nested
function is referenced in the options object which can't be pickled,
so pull it out into a static method instead.
Change-Id: I01d3c4eaabcb8b8775ddf22312a6e142c84cb77d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298722
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If a manifest checksout a project multiple times, repo download isn't
able to accurately pick the right project. We were just picking the
first result which could be a bit random for the user. If we hit that
situation, check if the cwd is one of the projects, and if it isn't,
we emit an error and tell the user it's an ambiguous request.
Bug: https://crbug.com/gerrit/13070
Change-Id: Id1059b81330229126b48c7312569b37504808383
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298702
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can speed things up. Across 1000 repos, it goes from
~9sec to ~5sec with the default -j8.
Change-Id: Ida6dd565db78ff7bac0ecb25d2805e8a1bf78048
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297982
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This avoids GIL limitations with using threads for parallel processing.
In a CrOS checkout with ~1000 repos, the nop case goes from ~6 sec down
to ~4 sec with -j8. Not a big deal, but shows that this actually works
to speed things up unlike the threading model.
This reworks the checkout logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.
Bug: https://crbug.com/gerrit/12389
Change-Id: I143e5e3f7158e83ea67e2d14e5552153a874248a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298063
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~30sec to ~3sec with the default -j8.
Change-Id: I0dc62d704c022dd02cac0bd67fe79224f4e34095
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297484
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
This is in preparation for adding jobs support. The nested function
is referenced in the options object which can't be pickled, so pull
it out into a static method instead.
Change-Id: I280ed2bf26390a0203925517a0d17c13053becaa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297983
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The default sync output should show a progress bar only for successful
commands, and the error output for any commands that fail. Implement
that policy here.
Bug: https://crbug.com/gerrit/11293
Change-Id: I85716032201b6e2b45df876b07dd79cb2c1447a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297905
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The idea for skipping some progress updates was to avoid spending
too much time on the progress bar itself. Unfortunately, for large
projects (100s if not 1000s) of repos, we get into the situation
with large/slow checkouts that we skip showing updates when a repo
finishes, but not enough repos finished to increase the percent.
Since the progress bar should be relatively fast compared to the
actual network & local dick operations, have it show an update
whenever the caller requests it. A test with ~1000 repos shows
that the progress bar in total adds <100ms.
Bug: https://crbug.com/gerrit/11293
Change-Id: I708a0c4bd923c59c7691a5b48ae33eb6fca4cd14
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297903
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When people switch to non-default branches, they sometimes want to
switch back to the default, but don't know the exact name for that
branch. Add a -b HEAD shortcut for that.
Change-Id: I090230da25f9f5a169608115d483f660f555624f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297843
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If --use-superproject is passed as argument to "repo init", then
--use-superproject need not be specified during "repo sync".
Tested the code with the following commands.
$ time repo_dev sync -c -j32
...
WARNING: --use-superproject is experimental and not for general use
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: Ibb33f3038a2515f74a6c4f7cb785d354b26ee680
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298102
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Python 3 renamed this method from isSet to is_set.
Change-Id: I8f9bb0b302d55873bed3cb20f2d994fa2d082157
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297742
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run diff in parallel.
Change-Id: I61e973d9c2cde039d5eebe8d0fe8bb63171ef447
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297483
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
This fixes intermingling of parallel jobs and simplifies the code
by switching to subprocess.run. This also provides stable output
in the order of projects by returning the output as a string that
the main loop outputs.
This drops support for interactive commands, but it's unclear if
anyone was relying on that, and the default behavior (-j2) made
that unreliable. If it turns out someone still wants this, we can
look at readding it.
Change-Id: I7555b4e7a15aad336667292614f730fb7a90bd26
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297482
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use a pool contextmanager to take care of the messy details like
properly cleaning it up when aborting.
Change-Id: I264ebb591c2e67c9a975b6dcc0f14b29cc66a874
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297243
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The status command runs a bunch of jobs in parallel, and each one
is responsible for writing to stdout directly. When running many
noisy jobs in parallel, output can get intermingled. Pass down a
StringIO buffer for writing to so we can return the entire output
as a string so the main job can handle displaying it. This fixes
interleaved output as well as making the output stable: we always
display results in the same project order now. By switching from
map to imap, this ends up not really adding any overhead.
Bug: https://crbug.com/gerrit/12231
Change-Id: Ic18b07c8074c046ff36e306eb8d392fb34fb6eca
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297242
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Extend the Command class to support adding the --jobs option to the
parser if the command declares it supports running in parallel. Also
pull the default value used for the number of local jobs into the
command module so local commands can share it.
Change-Id: I22b0f8d2cf69875013cec657b8e6c4385549ccac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297024
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Python 3 has a simpler super() style so switch to it to make the
code a little simpler and to stop pylint warnings.
Change-Id: I1b3ccf57ae968d56a9a0bcfc1258fbd8bfa3afee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297383
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Since the --manifest-url flag is always required when creating a new
checkout, allow the url to be specified via a positional argument.
This brings it a little closer to the `git clone` UI.
Change-Id: Iaf18e794ae2fa38b20579243d067205cae5fae2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297322
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We only provide input to GitCommand in one place, so inline the logic
to be more synchronous and similar to subprocess.run. This makes the
code simpler and easier to understand.
Change-Id: Ibe498fedf608774bae1f807fc301eb67841c468b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297142
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Clean up a few linter warnings.
Change-Id: I531d0263a202435d32d83d87ec24998f4051639c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297062
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The code is a bit simpler & easier to reason about.
Change-Id: I149729c7d01434b08b58cc9715dcf0f0d11201c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297022
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Added --no-use-superproject to repo and init.py to disable use of
manifest superprojects.
Replaced the term "sha" with "commit id".
Added _GetBranch method to Superproject object.
Moved shared code between init and sync into SyncSuperproject function.
This function either does git clone or git fetch. If git fetch fails
it does git clone.
Changed Superproject constructor to accept manifest, repodir and branch
to avoid passing them to multiple functions as argument.
Changed functions that were raising exceptions to return either True
or False.
Saved the --use-superproject option in config as repo.superproject.
Updated internal-fs-layout.md document.
Updated the tests to work with the new API changes in Superproject.
Performance for the first time sync has improved from 20 minutes to
around 15 minutes.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo init took around 20 seconds longer because of cloning of superproject.
$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real 0m35.919s
user 0m21.947s
sys 0m8.977s
First run
$ time repo sync --use-superproject
...
real 16m41.982s
user 100m6.916s
sys 19m18.753s
No difference in repo sync time after the first run.
Bug: [google internal] b/179090734
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
We have access to repodir on the command object itself, so we don't
need to pull it indirectly out of the manifest object.
Change-Id: I8688fb1c84979825efa966dc787e78c6f7ba3823
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296542
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
A little sugar simplifies the code a bit.
Change-Id: Ie2b8a965faa9f9ca05c7be479d03e8e073cd816d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296522
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Changed "git pull" to "git fetch" as we are using --bare option. Used the
following command to fetch:
git fetch origin +refs/heads/*:refs/heads/* --prune
Pass --branch argument to Superproject's UpdateProjectsRevisionId function.
Returned False/None when directories don't exist instead of raise
GitError exception from _Fetch and _LsTree functions. The caller of Fetch
does Clone if Fetch fails.
Tested the code with the following commands.
$ ./run_tests -v
Tested the init and sync code by copying all the repo changes into my Android
AOSP checkout and running repo sync with --use-superproject option.
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222
Reviewed-by: Mike Frysinger <vapier@google.com>
After updating all project’s revsionIds with the SHAs from superproject,
write the updated manifest into superproject_override.xml file. Reload
that file for future Reloads. This file is created in exp-superproject
directory.
Moved most of the code that is superproject specific into
git_superproject.py and wrote test code.
If git pull fails, did a git clone of the superproject.
We saw performance gains for consecutive repo sync's. The time to sync
went down from around 120 secs to 40 secs when repo sync is executed
consecutively.
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
First run
$ time repo sync --use-superproject
...
real 21m3.745s
user 97m59.380s
sys 19m11.286s
After two consecutive sync runs
$ time repo sync -c -j8 --use-superproject
real 0m39.626s
user 0m29.937s
sys 0m38.155s
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.
Tested the code with the following commands.
$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,
.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []
Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
We're committed to Python 3 at this point, so purge all the
is_python3 related dynamic checks.
Bug: https://crbug.com/gerrit/10418
Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Spread the operation of querying which local branches exist across a
pool of processes and build the name map of projects -> branches as
these tasks finish rather than blocking on the entire query. The search
operations are submitted in batches to reduce the overhead of interprocess
communication. The `chunksize` argument used to control this batch size
was selected by incrementing through powers of two until it stopped being
faster.
Change-Id: Ie3d7f799ee8e83e5058536caf53e2979175408b7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291342
Tested-by: Chris Mcdonald <cjmcdonald@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
When updating the tracking ref to whatever the user requested,
make sure we reset state completely rather than trying to update
the ref to it. This avoids confusing git as to the current state
of the tree, and is more inline with user intentions: if they made
a local change to the checkout, but ran repo init with a specific
rev, we shouldn't stay wedged forever until they manually clean it
all up.
Bug: https://crbug.com/gerrit/12801
Change-Id: Ieba8d9c15781b4d0649bf01c7460694da63387b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290923
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The knowledge about running hooks and all its exception handling
is scattered over multiple files. This makes the code harder
to read, but also it requires duplication of logic in case
other RepoHooks are added to different commands.
This refactoring also creates uniform behavior of the hooks
across multiple commands and it guarantees the re-use of the same
arguments on all of them.
Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: Ia4d90eab429e4af00943306e89faec8db35ba29d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/277562
Tested-by: Remy Bohmer <oss@bohmer.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
We conflate the manifest & parsing logic with the management of the
repo client checkout in a single class. This makes testing just one
part (the manifest parsing) hard as it requires a full checkout too.
Start splitting the two apart into separate classes to make it easy
to reason about & test.
Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The --force-sync option was being passed down for all updates except
for the manifest project, so add that there too.
Bug: https://crbug.com/gerrit/11034
Change-Id: I33818b652f828c6b847dbc70f1fedfac5ac17bbe
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228146
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Sometimes parsing JSON is easier than parsing XML, especially when
the XML format is limited (which ours is). Add a --json option to
the manifest command to quickly emit that form.
Bug: https://crbug.com/gerrit/11743
Change-Id: Ia2bb254a78ae2b70a851638b4545fcafe8c1a76b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280436
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Instead of assuming the repo client is tracking the "master" branch
of the manifest repo, use the existing info we have to display the
right info to the user.
Bug: https://crbug.com/gerrit/13339
Change-Id: I8b265f4b2e075fdc41909b1f3dff9aee87384353
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/287279
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The project.py file is huge and contains multiple
classes. By moving it to seperate class files the code
becomes more readable and maintainable.
Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: Ida9d99d31751d627ae1ea0373418080696d2e14b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/281293
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <linux@bohmer.net>
Instead of hardcoding "master" as our default, use the remote server's
default branch instead. For most people, this should be the same as
"master" already. For projects moving to "main", it means we'll use
the new name automatically rather than forcing people to use -b main.
For repositories that never set up a default HEAD, we should still use
the historical "master" default.
Bug: https://crbug.com/gerrit/13339
Change-Id: I4117c81a760c9495f98dbb1111a3e6c127f45eba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280799
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This change increases the speed of the command with parallelization with
processes. The parallelization with threads doesn't work well, and
increasing the number of jobs to many (8 threads ~) didn't increase the speed.
Possibly, the global interpreter lock of Python affects.
Bug: https://crbug.com/gerrit/12389
Change-Id: Icbe5df8ba037dd91422b96f4e43708068d7be924
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279936
Tested-by: Kimiyuki Onaka <kimiyuki@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
We deprecated this 8 years ago. Time to drop it to simplify the code
as it'll help with refactoring in this module to not migrate it.
Change-Id: I2deae5496d1f66a4491408fcdc95cd527062f8b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280798
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
If you pass args to `repo init` when first creating a checkout, the
repo launcher throws an error. But the init subcommand that runs in
an existing checkout silently ignores them. Throw a proper error.
Change-Id: I433bfcc73902d25f6b6a2974e77f6a977a75ed16
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279696
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
For large projects, clone bundle is useful because it provided a way to
efficiently transfer a large portion of git objects through CDN, without
needing to interact with git server. However, with partial clones, the
intention is to not download most of the objects, so the use of clone
bundles would defeat the space savings normally seen with partial
clones, as they are downloaded before the first fetch.
A new option, --clone-bundle is added to override this behavior.
Add a new repo.clonebundle variable which remembers the choice if
explicitly given from command line at repo init.
Change-Id: I03638474af303a82af34579e16cd4700690b5f43
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268452
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
If the dest-branch attribute is set in the project manifest, then
we need to push to that branch. Previously, we would unconditionally
pre-pend the refs/heads prefix to it. The dest-branch attribute is
allowed to be a ref expression though, so it may already have it.
Simple fix is to check if it already has the prefix before adding it.
Bug: crbug.com/gerrit/12770
Change-Id: I45d6107ed6cf305cf223023b0ddad4278f7f4146
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268152
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
When running repo info -d an error would be thrown saying:
fatal: bad revision 'refs/remotes/m/refs/heads/master..'
Using the short branch name here instead, like 'refs/remotes/m/master..'
resolves this issue.
Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: I50ea92c45c011b2c3e3a63803decb88e7837a380
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/266578
Reviewed-by: Mike Frysinger <vapier@google.com>
When generating a revision locked manifest, we need to know what
ref to push changes to when doing 'repo upload'. This information
is lost when we lock the revision attribute to a particular commit
hash, so we need to expose it through the dest-branch attribute.
Bug: https://crbug.com/1005103
Test: manual execution
Change-Id: Ib31fd77ad8c9379759c4181dac1ea97de43eec35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263572
Tested-by: Sean McAllister <smcallis@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Recent changes in ChromeOS Infra to ensure we're reading from
snapshot manifests properly have exposed several bugs in our
assumptions about manifest files. Mainly that the revision field
for a project does _not_ have to refer to a ref, it can just be
a commit hash.
Several places assume that the revision field can be parsed as a
ref to get the branch the project is on, which isn't true. To fix
this we need to be able to look at the upstream and dest-branch
attributes of the repo, so we expose them through the environment
variables set in `repo forall`.
Test: manual 'repo forall' run
Bug: https://crbug.com/1032441
Change-Id: I2c039e0f4b2e0f430602932e91b782edb6f9b1ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263132
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
In order to remove the stream fileno() will be called on the filedescriptor.
If the file is already closed fileno() will raise an error and forall
will fail.
Bug: https://crbug.com/gerrit/12563
Change-Id: Ib7b895fe881c844e3eb3672b011fdcdbdae63024
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/262838
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Karsten Pfeiffer-Raddatz <raddatz.karsten@gmail.com>
Add retries with exponential backoff and jitter to the fetch
operations. By default don't change behavior and enable
behind the new flag '--fetch-retries'.
Bug: https://crbug.com/1061473
Change-Id: I492710843985d00f81cbe3402dc56f2d21a45b35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/261576
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: George Engelbrecht <engeg@google.com>
This is more for users trying to get a sense of how old/new their
current version of repo is when debugging issues.
Change-Id: Ifb413c679bb8c8dbf4f9334137adf086bb000a68
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/261192
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This gets the unittests passing again for now.
Change-Id: Ibed430a305bc26b907ad0ea424c7eec7de37e942
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259994
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We respect this option when running the first `repo init`, but then
silently ignore it once the initial sync is done. Make sure users
are able to change things on the fly.
We refactor the wrapper API to allow reuse between the two init's.
Bug: https://crbug.com/gerrit/11045
Change-Id: Icb89a8cddca32f39a760a6283152457810b2392d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260032
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
This allows people to quickly create new branches when pulling down
changes rather than having to juggle the git calls themselves.
Bug: https://crbug.com/gerrit/11609
Change-Id: Ie6a4d05e9f4e9347fe7f7119c768e6446563ae65
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259855
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We gracefully handle cherry-pick errors, but none of the others
which means people get confusing Python tracebacks. Move the
main logic in a single GitError try block so we can show pretty
error messages for all of them.
Change-Id: I52cdf6468d21a98de7f65b86d5267b3caabd5af8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259854
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This is a pretty common option for people to want too use, so include
it as a pass-thru option when cherry-picking.
Bug: https://crbug.com/gerrit/9418
Change-Id: I2a24c1ed7544541719caa4d3c0574347a151a1b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259853
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The git cherry-pick already supports this, so plumb the existing repo
option down. Otherwise it's confusing when people use -c --ff and it
doesn't use that behavior.
Change-Id: Id68932ffa09204bb30b92a21aff185c00394a520
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259852
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We refer to this as "revision" in help text, and in REPO_REV envvar,
so rename to --repo-rev to be consistent. We keep --repo-branch for
backwards compatibility, but as a hidden option.
Bug: https://crbug.com/gerrit/11045
Change-Id: I1ecc282fba32917ed78a63850360c08469db849a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259352
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We respect this option when running the first `repo init`, but then
silently ignore it once the initial sync is done. Make sure users
are able to change things on the fly.
Bug: https://crbug.com/gerrit/11045
Change-Id: I129ec5fec43a85067d555bb60c0d1ae02465f139
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258893
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Since tracking a branch prevents repo from updating, make sure we
warn people about the situation when using `repo sync`.
Bug: https://crbug.com/gerrit/11045
Change-Id: I966513f510827cc93194f8df176c6745946bd739
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258892
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The current subcmds design has singletons in all_commands. This isn't
exactly unusual, but the fact that our main & help subcommand will then
attach members to the classes before invoking them is. This makes it
hard to keep track of what members a command has access to, and the two
code paths (main & help) attach different members depending on what APIs
they then invoke.
Lets pull this back a step by storing classes in all_commands and leave
the instantiation step to when they're used. This doesn't fully clean
up the confusion, but gets us closer.
Change-Id: I6a768ff97fe541e6f3228358dba04ed66c4b070a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259154
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The branch->branches alias is setup in the main module when that
really belongs in the existing all_commands setup.
For help, rather than monkey patching all_commands to the class,
switch it to use the state directly from the module. This makes
it a bit more obvious where it's coming from rather than this one
subcommand having a |commands| member added externally to it.
Change-Id: I0200def09bf4774cad8012af0f4ae60ea3089dc0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259153
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Enable using --pretty-format to build a custom subject line
even when using the --raw option.
Change-Id: I0c1e682d984e56698fe65939aa6de12a653cd0f1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258565
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Connor Newton <connor@ifthenelse.io>
This is a dict to index, not a function to call.
Change-Id: I0117eeaaa8b2ef4762ab6f0d22f9ffdaee961f52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258132
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We often ask users what OS/version they're running when debugging.
Include that in the version output to simplify the process.
Change-Id: Ie480b6d1c874e6f4c6f4996a03795077b844f858
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256732
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Draft CLs were replaced by private/wip CLs in Gerrit instead years ago.
Change-Id: If4f3d6606aad40a6f1617a49681dfd45c64d3d37
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256673
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Still use the repo manifest by default as before, but gives us
the option of overriding it to support e.g.: using a subset of
the full manifest.
Change-Id: Ia42cd1cb3a0a58929d31bb01c9724e9d31f68730
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256372
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
We produce some simple "Get" messages that aren't super clear as to
what they're doing, especially for people not familiar with repo.
Rephrase these to explicitly state the thing we're doing so it's
clear why we're downloading a particular source.
Bug: https://crbug.com/gerrit/11293
Change-Id: I0749504f17c5385c6c65274a274e0ae25b117413
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256455
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We don't use these for much yet, but init passes it down to the
project sync layers which already has support for verbose mode.
Change-Id: I651794f1b300be1688eeccf3941ba92c776812b5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256454
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Also fix the normal output to write to stdout.
Change-Id: I6c117eea9cec08f5be9a44b90dbe9bf1f824ec95
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256114
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
- E129 visually indented line with same indent as next logical line
- E125 continuation line with same indent as next logical line
Fixed automatically by:
autopep8 --in-place --select E125,E129 subcmds/sync.py
Change-Id: Ia2f82f443e1e6a23ba22c6f9849c8485405aed0e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256092
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
Since deleting a source checkout involves a good bit of internal
knowledge of .repo/, move the DeleteProject helper out of the sync
code and into the Project class itself. This allows us to add git
worktree support to it so we can unlock/unlink project checkouts.
Change-Id: If9af8bd4a9c7e29743827d8166bc3db81547ca50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256072
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When using extensions, make sure we set the git repo format version
so git knows to check the extension compatibility. We can add a
helper to the Project API to simplify this and make it foolproof.
Change-Id: I9ab6c32d92fe2b8e5df6e2b080ca71556332e909
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256035
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
This provides initial support for using git worktrees internally
instead of our own ad-hoc symlink tree. It's been lightly tested
which is why it's not currently exposed via --help.
When people opt-in to worktrees in an existing repo client checkout,
no projects are migrated. Instead, only new projects will use the
worktree method. This allows for limited testing/opting in without
having to completely blow things away or get a second checkout.
Bug: https://crbug.com/gerrit/11486
Change-Id: Ic3ff891b30940a6ba497b406b2a387e0a8517ed8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254075
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Different Python & OS versions have different environ behavior wrt
accepted types & encoding. Since we're migrating to be Python 3 only,
lets change our code to assume strings always work as that's what the
newer Python 3 does. This will fail under Python 2 for some env vars,
mostly on Windows, but the effort of maintaining shim layers that can
handle these edge cases isn't worth it when we're dropping that code.
We leave the logic in the `repo` launcher for now as it is simple, and
we want it to be able to switch versions a bit longer than the rest of
the tree.
Here's the support table:
| *NIX | Windows |
Python 2 | ASCII string | str or bytes, not unicode |
Python 3 | str or bytes | str only |
Windows uses strings natively in its environment all the time. But it
doesn't allow unicode strings under Python 2, so we have to encode.
Python 2 on *NIX is funky in that it always lowers to ASCII, so we had
to manually encode to avoid errors regardless of unicode or str.
Python 3 on Windows & *NIX will accept strings. *NIX will also accept
bytes but Windows will not.
Bug: https://crbug.com/gerrit/12145
Change-Id: I3cf8f95a06902754ea1f08ad4b28503f7063531b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248972
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This adds a CLI option to the existing autoupload gitconfig knob that
allows people to automatically answer "yes" to the various prompts.
Bug: https://crbug.com/gerrit/12368
Change-Id: I819ebca01b9a40240b33866ae05907c7469703e3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255892
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This allows users to specify custom hashtags when uploading, both via
the CLI and via the same gitconfig settings as other upload options.
Bug: https://crbug.com/gerrit/11174
Change-Id: Ia0959e25b463e5f29d704e4d06e0de793d4fc77c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255855
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Show clearer messages and exit non-zero if there's nothing to upload.
Change-Id: Icd9c13b9b1126610a409fc13d1d11bfc66f5e802
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255834
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Trying to use booleans with names like "no_xxx" are hard to follow due
to the double negatives. Invert all of them so we only have positive
meanings to follow.
Change-Id: Ifd37d0368f97034d94aa2cf38db52c723ac0c6ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255493
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Named Temporary file defaults to mode 'w+b' which causes repo sync to
fail. By opening the tmpcookiefile in PersistentTransport.request as
writable, we are able to run sync successfully.
Bug: https://crbug.com/gerrit/12370
Test: Ran smartsync successfully
Change-Id: I01ddf915fc30eb3ff0e4d440a6f1aa261c63e88d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255692
Tested-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
All the stdlib imports are supposed to come before any local imports.
Change-Id: I10c0335ba2ff715fd34c9eb91bfe6560e904df08
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255593
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This allows us to control sync output better by having three levels
of output: quiet (only errors), default (progress bars), verbose (all
the things). For now, we just put the chatty "already have persistent
ref" message behind the verbose level.
Bug: https://crbug.com/gerrit/11293
Change-Id: Ia61333fd8085719f3e99edb7b466cdb04031b67f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255414
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
All of the instances of this are related to Python 2 names that
don't exist in Python 3, and the warnings are raised when running
flake8 on Python 3.
All of these will go away once we completely remove support for
Python 2, so just suppress them inline. We don't globally suppress
the check so that we will still see legitimate errors if/when they
occur in new code.
Change-Id: Iccf955f50abfc9f83b371fc0af6cceb51037456f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255039
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
We increased the max line length to 100 columns which got rid of
the majority of these warnings, but there were still a few lines
that exceeded 100 columns.
Change-Id: Ib3372868ca2297f83073a14f91c8ae3df9d0d0e6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254699
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Some people find the existing output to be a bit confusing. It spews
a lot of git output before exiting, but it's not exactly clear what
the final state is when things pass. Add an explicit message.
Bug: https://crbug.com/gerrit/10501
Change-Id: I9de83b595d3185feb820005b8fc81c6adc55b357
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254732
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We've been requiring git-1.7.2 since Oct 2012, so we can safely drop
the individual checks sprinkled throughout the code base for older.
Change-Id: I1737fff7b3f27f475960b0bff9cb300aefd5d108
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253135
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
flake8 reports:
E722 do not use bare 'except'
Replace them with 'except Exception' per [1] which says:
Bare except will catch exceptions you almost certainly don't want
to catch, including KeyboardInterrupt (the user hitting Ctrl+C) and
Python-raised errors like SystemExit
If you don't have a specific exception you're expecting, at least
except Exception, which is the base type for all "Regular" exceptions.
[1] https://stackoverflow.com/a/54948581
Change-Id: Ic555ea9482645899f5b04040ddb6b24eadbf9062
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254606
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E301,E302,E303,E305,E306
Manually fix issues in project.py caused by misuse of block comments.
Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E121 continuation line under-indented for hanging indent
- E122 continuation line missing indentation or outdented
- E125 continuation line with same indent as next logical line
- E126 continuation line over-indented for hanging indent
- E127 continuation line over-indented for visual indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E121,E122,E125,E126,E127,E128,E129,E131
Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
flake8 reports:
E713 test for membership should be 'not in'
Change-Id: I4446be67c431b7267105b53478d2ceba2af758d7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254451
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
flake8 reports:
E714 test for object identity should be 'is not'
Change-Id: Ib8c4100babaf952bbfe65fd56555ece8a958e4b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254450
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
flake8 reports:
E265 block comment should start with '# '
While we're at it, add a period at the end of the comment sentence.
Change-Id: Icb7119079a1d64e6defafc3f6d24e99dbf16139d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254596
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
When upload hooks fail, people are forced to use --no-verify to upload
CLs anyways. When projects have flaky hooks, this trains people to
always use that option. This is obviously bad: hooks might get fixed,
or some of the hooks are always good & people should review.
Lets add an --ignore-hooks option. This still runs the hooks, but any
failures will be ignored and allow the user to upload anyways.
Bug: https://crbug.com/gerrit/12230
Change-Id: Ide2ac8a40a656bfcd6aae20c3ce8118e06bf909b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254452
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
flake8 reports:
W391 blank line at end of file
Change-Id: I5498b2de2d1268d4f1f4b9e1760f9fa93a6da4cd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254594
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
We were perhaps a bit too hasty to jump to git-2.10. Existing LTS
releases of Ubuntu are quite old still: Trusty has 1.9 while Xenial
has 2.5. While we plan on dropping support for those eventually as
we migrate to Python 3.6, we don't need to be so strict just yet on
the git versions.
We also want to disconnect the version the repo launcher requires
from the version the rest of the source tree requires. The repo
launcher doesn't need as many features, and being flexible there
allows us more freedom to upgrade & rollback as needed.
So we'll allow git-1.7 again, but start warning on any users older
than git-1.9. This aligns better with existing LTS releases, and
gives users a chance to start upgrading before we cut them off.
Change-Id: I140305dd8e42c9719c84e2aee0dc6a5c5b18da25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254573
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Some automatic git operations will prune objects on us, and not just
the gc step. Normally we don't care, but with shared projects, we
will have multiple git checkouts with refs that the others cannot
see, but with a shared object dir. Any pruning of objects based on
refs in just one repo can easily break the others.
git-2.7.0 introduced a preciousObjects setting which tells git to
never prune objects for this exact scenario: there might be refs in
some location that git is unable to see.
Change-Id: I781de27c5bbe1d4c70f0187566141c9cce088bd8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254392
Reviewed-by: Nasser Grainawi <nasser@codeaurora.org>
Reviewed-by: David Riley <davidriley@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Execution of 'repo forall -p -c' doesn't work with Py3 and ends up
with an error:
Got an error, terminating the pool: TypeError: can only concatenate
str (not "bytes") to str
That's fixed by using the decode() method.
Change-Id: Ice01aaa1822dde8d957b5bf096021dd5a2b7dd51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253659
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Jiri Tyr <jiri.tyr@gmail.com>
These files are not directly executable, so drop the +x bits.
Change-Id: Iaf19a03a497686cc21103e7ddf08073173440dd1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254076
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Instead of not using '-c' for '--current-branch' when using gitc, we
were only using '-c' when using gitc, so we still had the conflict with
the gitc option, and other users still couldn't use '-c'.
Test: repo init -u https://android.googlesource.com/platform/manifest; repo init -c
Test: repo gitc-init -u ... -b ... -c testing
Change-Id: I71e4950a49c281418249f0783c6a2ea34f0d3e2b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253795
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Dan Willemsen <dwillemsen@google.com>
Currently our default behavior is:
* Try to sync all repos
* If any errors seen, exit
* Try to garbage collect all repos
* If any errors seen, exit
* Try to update local project list
* If any errors seen, exit
* Try to checkout out all local repos
* If any errors seen, exit
Users find these incomplete syncs confusing, so lets try to complete
as much as possible by default and printing out summaries at the end.
Bug: https://crbug.com/gerrit/11293
Change-Id: Idd17cc9c3bbc574d8a0f08a30225dec7bfe414cb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238554
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The use case is any situation where your manifest does
not exist on server, but where you still want to do
full sync for the projects, without having your
workspace manifest switched to other branch or
forwarded to latest or similar.
This allows syncing to a historical manifest in git log,
that does not have a branch, as well as when integrating
something together that has not been pushed upstream yet.
Changes can also exist locally on a manifest that is
behind head, meaning not requiring rebase to latest.
Tested using:
$ cd .repo/manifests/
$ git checkout <any hash 1>
$ <do local modifications>
$ repo sync --no-manifest-update
$ git checkout <any hash 2>
$ repo sync --no-manifest-update
Change-Id: I0c9773aa8bc5876813a2e7d7fec697abcb2d9e94
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/246445
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
We keep getting requests for init to support -c. This conflicts with
gitc-init which allocates -c for its own use. Lets make this dynamic
so we keep it with "init" but omit it for "gitc-init".
Bug: https://crbug.com/gerrit/10200
Change-Id: Ibf69c2bbeff638e28e63cb08926fea0c622258db
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253252
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This makes it consistent with the short option for current-branch in
repo sync.
Change-Id: I2848e87f45a66ef8d829576d0c0c4c0f7a8636a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/241700
Tested-by: Diogo Ferreira <deovferreira@gmail.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
After Ib546f5ebbc8a23875fbd14bf166fbe95b7dd244e, repo info now displays
the current project revision in the 'Current revision' field.
While the output is more consistent, there are use cases for the
revision expression as shown in the manifest. This patch re-adds the
manifest revision as a new 'Manifest revision' field.
Change-Id: I50c1559dcb7ceb69af07352b956d78f85b8f592e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/240799
Tested-by: Diogo Ferreira <deovferreira@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
No functional changes, just unifying duplicate code paths.
Change-Id: I6afa797ca1e1eb90abdc0236325003ae070cbfb3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/247293
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The upload module tries to turn the strings into bytes before passing
to EditString, but it combines bytes & strings causing an error. The
return value might be bytes or string, but the caller only expects a
string. Lets simplify this by sticking to strings everywhere and have
EditString take care of converting to/from bytes when reading/writing
the underlying files. This also avoids possible locale confusion when
reading the file by forcing UTF-8 everywhere.
Bug: https://crbug.com/gerrit/11929
Change-Id: I07b146170c5e8b5b0500a2c79e4213cd12140a96
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/245621
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Series of steps:
* Create a local "b1" branch with `repo start b1` that tracks a remote
branch (totally fine)
* Manually create a local "b2" branch with `git branch --track b1 b2`
that tracks the local "b1" (uh-oh...)
* Delete the local "b1" branch manually or via `repo prune` (....)
* Try to process the "b2" branch with `repo prune`
Since b2 tracks a branch that no longer exists, everything blows up
at this point as we try to probe the non-existent ref. Instead, we
should flag this as unknown and leave it up to the user to resolve.
This probably could come up if a local branch was tracking a remote
branch that was deleted from the server, and users ran something like
`repo sync --prune` which cleaned up the remote refs.
Bug: https://crbug.com/gerrit/11485
Change-Id: I6b6b6041943944b8efa6e2ad0b8b10f13a75a5c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236793
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Kirtika Ruchandani <kirtika@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The current sync output displays "Fetching project" and "Checking out
project" messages and progress bar updates independently leading to a
lot of spam. Lets merge these periodic outputs with the status bar to
get a little bit tighter output in the normal case. This doesn't solve
all our problems, but gets us closer.
Bug: https://crbug.com/gerrit/11293
Change-Id: Icd627830af4dd934a9355b7ace754b56dc96cfef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244934
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When repo sync fails because some git trees are not in clean state and
as such can not be rebased automatically, it is a pain to figure out
which trees are the culprits.
With this patch the list of offending trees is printed when repo sync
reports checkout errors.
TEST=ran 'repo sync' and observed the proper list of directories show
up after the final error message
Bug: https://crbug.com/gerrit/11293
Change-Id: Icdf1a03e9014ecb184f331f513cc9a2efc7d11ed
Signed-off-by: Vadim Bendebury <vbendeb@google.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244053
Reviewed-by: Mike Frysinger <vapier@google.com>
Use open() as a context manager to simplify the close logic and make
the code easier to read & understand. This is also more Pythonic.
Change-Id: I579d03cca86f99b2c6c6a1f557f6e5704e2515a7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244734
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
In CrOS, our infra has to deal with partial checkouts constantly
(for a variety of reasons). To help reset back to a good state,
we run git commands via `repo forall`, but don't care about the
missing checkouts. Add a flag so we can disambiguate between
missing repos and failing git subcommands.
Bug: https://crbug.com/1013377
Bug: https://crbug.com/1013623
Change-Id: Ie3498c6d111276c60d2ecedbba21bfa778588d50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/241935
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The "Current revision" field shows the revision as listed in the
manifest. I think most users expect this to show the revision
that the git tree is checked out to instead. Switch the output
to show that revision instead, and add a "Current branch" if it
matches a local branch.
Change-Id: Ib546f5ebbc8a23875fbd14bf166fbe95b7dd244e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239240
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If the user passes in bad projects like `repo info asdf`, we currently
silently swallow those and do nothing. Allow NoSuchProjectError to
bubble up to main which will handle & triage this correctly for us.
Change-Id: Ie04528e7b7a164293063a636813a73eaabdd5bc3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239238
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Depending on where/how repo is invoked, the active version might be
from a git tree, and it might be different from the .repo/repo.git/
version in the current repo client checkout. Report both if they're
different so it's clearer.
Lets also include the two different User-Agent's that we set up when
talking to networked services.
Bug: https://crbug.com/gerrit/11144
Change-Id: I2ebb6e3ac30e374a8406cab3e4438087246a8c57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239234
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If `git grep` fails in any project checkout (e.g. an incomplete
sync), make sure we print that error clearly rather than blowing
up, and exit non-zero in the process.
Bug: https://crbug.com/gerrit/11613
Change-Id: I31de1134fdcc7aaa9814cf2eb6a67d398eebf9cf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239237
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If `git diff` fails in any project checkout (e.g. an incomplete
sync), make sure we print that error clearly rather than blowing
up, and exit non-zero in the process.
Bug: https://crbug.com/gerrit/11613
Change-Id: I12f278427cced20f23f8047e7e3dba8f442ee25e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239236
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If a repo doesn't exist (e.g. an incomplete sync), make sure we exit
non-zero when they get skipped.
Change-Id: Ifff711e374416b1e6b9b8da4fdc6f14b27ced450
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239235
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This uses coloring style like we use in grep/forall already.
Change-Id: I317e2e47567a30c513083c48e7c7c40b091bb29a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238555
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Lets switch the default rebase behavior to align with our new sync
behavior: we try to rebase all projects by default and exit/summarize
things at the very end if there were any errors. Or if people want
to exit immediately, they can use the new --fail-fast option.
Change-Id: I436ac563f972b45de6ce9ad74da1e4870e584902
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238553
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Cut out some more standalone code from Execute to make this func a
bit more manageable. The manifest project update is pretty simple
and standalone, but still takes up a good chunk of what's left.
Change-Id: Idc2442d9def495eccd0a49cda203c44aef16f129
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236614
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The smart sync logic takes up about 45% of the overall Execute func
and is about 100 lines of code. The only effect it has on the rest
of the code is to set the manifest_name variable. Since this func
is already quite huge, split the smart sync logic out.
Change-Id: Id861849b0011ab47387d74e92c2ac15afcc938ba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234835
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This makes it way easier to recover from forgetting to run repo start
before committing: just run `repo start -b new-branch`, instead of
all that tedious mucking around with reflogs.
Change-Id: I56d49dce5d027e28fbba0507ac10cd763ccfc36d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232712
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
In commit d9e5cf0e ("sync: invert --force-broken with --fail-fast") the
force-broken option has been deprecated. Accidentally the option has
been changed from Boolean to Value. This breaks all users of repo with:
main.py: error: -f option requires an argument
This is easy to avoid by keeping the type.
Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Change-Id: Ia8b589cf41ac756d10c61e17ec8d76ba8f7031f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/235043
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
A common pattern in our subcommands is to verify the arguments &
options before executing things. For some subcommands, that check
stage is quite long which makes the execution function even bigger.
Lets split that logic out of the execute phase so it's easier to
manage these.
This is most noticeable in the sync subcommand whose Execute func
is quite large, and the option checking makes up ~15% of it.
The manifest command's Execute can be simplified significantly as
the optparse configuration always sets output_file to a string.
Change-Id: I7097847ff040e831345e63de6b467ee17609990e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234834
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This is useful when you want to scan all the possibilities of repo
at once. Like when you're searching for different option names.
Change-Id: I225dfb94d2be78229905b744ecf57eb2829bb52d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232894
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
