Commit Graph

295 Commits

Author SHA1 Message Date
Mike Frysinger
2273f46cb3 sync: fix --tags option
This has been broken since it was added where --tags was actually
the same as --no-tags.  Oddly, it was copied from init where the
logic is correct.

Bug: https://crbug.com/gerrit/12401
Change-Id: I15b89da1a655176a11bebc22573b25c728055328
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322955
Reviewed-by: Andrew Lamb <andrewlamb@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-11-05 20:13:02 +00:00
Raman Tenneti
6251729cb4 superproject: added 'implies -c' in the help of --use-superproject option.
sync.py: deleted unused import errno.

Tested:
$ ./run_tests
$ flake8 repo subcmds/sync.py

Bug: https://crbug.com/gerrit/15208
Change-Id: I2bb3098f5602ded3861e000100766041ad93b53d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322555
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-11-01 22:15:53 +00:00
Jack Neus
03ff276cd7 sync: properly handle standalone manifests for sync command
sync should not attempt to sync the manifest project if it was
created from a standalone manifest. The current work around is to
run sync with --nmu.

BUG=none
TEST=manual runs

Change-Id: I2e121af0badf9642143e77c7af89d1c2d993b0f3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/321195
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-15 17:20:00 +00:00
Allen Webb
4ee4a45d03 subcmds/sync: Use pack-refs instead of gc for redundant gitdirs.
Previously `git gc` was being run on every gitdir even when they shared
the same objects. Instead only call it once and use pack-refs for the
gitdirs that were not gc'ed.

Bug: https://crbug.com/gerrit/15113
Test: repo sync -j # and check that git pack-refs is called
Change-Id: Icff37ab3ec78cfb44391d8cc7f2d875991532320
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/320275
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-14 12:27:12 +00:00
Allen Webb
669efd0fd7 subcmds/sync: Disable autoDetach for git gc.
gc.autoDetach is enabled by default which makes 'git gc --auto' return
immediately and run in background. This can lead to a pile up of
operations all using large amounts of memory at the same time. To avoid
this set gc.autoDetach to false so that the garbage collect task waits
for instances to finish before spawning more.

Bug: https://crbug.com/gerrit/15113
Test: repo sync -j # and check the number of 'git gc' processes
Change-Id: Ic0815156ba3db03972968f33f6f9f51e4928f23b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319835
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-05 14:12:01 +00:00
Mike Frysinger
9d96f58f5f make file removal a bit more robust
Some of the file removal calls are subject to race conditions (if
something else deletes the file), so extend our remove API to have
an option to ignore ENOENT errors.  Then update a bunch of random
call sites to use this new functionality.

Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195
Reviewed-by: Sean McAllister <smcallis@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-09-28 16:06:50 +00:00
Raman Tenneti
6448a4f2af sync: Log repo sync state events as 'data' events.
git_trace2_event_log.py:
+ Added LogDataConfigEvents method to log 'data' events.
  Sync's current_sync_state and previous_sync_state are logged
  as 'data' events in the current log.

  It logs are key/value in the |config| argument. Each key is
  prefixed with |prefix| argument.

  The following are sample events that are logged during repo sync.

   {"event":"data",
   "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
   "thread":"MainThread",
   "time":"2021-09-14T18:16:19.935846Z",
   "key":"previous_sync_state/repo.syncstate.main.synctime",
   "value":"2021-09-14T17:27:11.573717Z"}

   {"event":"data",
   "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
   "thread":"MainThread",
   "time":"2021-09-14T18:16:19.955546Z",
   "key":"current_sync_state/repo.syncstate.main.synctime",
   "value":"2021-09-14T18:16:19.935979Z"}

tests/test_git_trace2_event_log.py:
+ Added unit tests

sync.py:
+ Changed logging calls to LogDataConfigEvents.

Tested:
$ ./run_tests

Tested it by running the following command multiple times.
$ repo_dev sync -j 20
  repo sync has finished successfully

  Verified config data is looged in trace2 event logs.

Bug: [google internal] b/199758376
Change-Id: I75fd830e90c1811ec28510538c99a2632b104e85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317823
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-09-14 21:36:12 +00:00
Raman Tenneti
b55769a5c9 superproject: print messages if the manifest has superproject tag.
1) If the manifest has superproject tag (git_master, etc), then
   display error/warning messages (as it is doing today)
2) If the manifest doesn't have superproject tag (nest, chromeos
   manifests), then don't display any error/warning messages about
   superrproject (behave as though user has specified
   --no-use-superproject).
3) Print error/warning messages if --use-superproject passed as
   argument to repo sync.
4) No change in behavior for the repo init command.

git_superproject.py:
+ Fixed typo in _WriteManifestFile method name
+ Superproject accepts print_message  as an argument and it defaults
  to True. All messages that are printed to stderr are controlled by
  this flag. If it is True, then messages get printed.
+ Added PrintMessages function which return true if either
  --use-superproject is specified on the command line or if the
  manifest has a superproject tag.

sync.py:
+ Displays the warning message if PrintMessgages are enabled and
  passes that as argument to superproject object.
+ Added 'hassuperprojecttag' trace2 log entry for analysis. We can
  find users/branches that are using superproject, but the manifest is
  missing the superproject tag.

Tested:
$ ./run_tests

+ Verified printing of messages with and without superproject tag, with
  with --use-superproject option.

+ aosp-master
  $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
  $ repo_dev sync

+ A manifest without superproject tag.
  $ repo_dev init -m $(pwd)/manifest_7482982.xml
  $ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml

Bug: [google internal] b/196411099
Change-Id: I92166dcad15a4129fab82edcf869e7c8db3efd4b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314982
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-08-13 20:07:40 +00:00
Raman Tenneti
7954de13b7 sync: Added logging of repo sync state and config options for analysis.
git_config.py:
+ Added SyncAnalysisState class, which saves the following data
  into the config object.
  ++ sys.argv, options, superproject's logging data.
  ++ repo.*, branch.* and remote.* parameters from config object.
  ++ current time as synctime.
  ++ Version number of the object.
+ All the keys for the above data are prepended with 'repo.syncstate.'
+ Added GetSyncAnalysisStateData and UpdateSyncAnalysisState methods
  to GitConfig object to save/get the above data.

git_trace2_event_log.py:
+ Added LogConfigEvents method with code from DefParamRepoEvents
  to log events.

sync.py:
+ superproject_logging_data is a dictionary that collects all the
  superproject data that is to be logged as trace2 event.
+ Sync at the end logs the previously saved syncstate.* parameters
  as previous_sync_state. Then it calls config's UpdateSyncAnalysisState
  to save and log all the current options, superproject logged data.

docs/internal-fs-layout.md:
+ Added doc string explaining [repo.syncstate ...] sections of
  .repo/manifests.git/config file.

test_git_config.py:
+ Added unit test for the new methods of GitConfig object.

Tested:
$ ./run_tests

$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest

Tested it by running the following command multiple times.
$ repo_dev sync -j 20
  repo sync has finished successfully

  Verified config file has [syncstate ...] data saved.

Bug: [google internal] b/188573450
Change-Id: I1f914ce50f3382111b72940ca56de7c41b53d460
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313123
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2021-07-29 19:20:57 +00:00
Raman Tenneti
ae86a46022 superproject: Skip updating of superproject when -l is used with sync.
Skip updating the superproject when -l is present and use the existing
superproject, if available (this would make sync -l work as it's
intended to do), and fall back to sync without superproject when not
(this would catch the case when superproject is enabled by automatic
rollout).

Tested:
$ repo sync -j 20 -n
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/android/src/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Fetching: 100% (1032/1032), done in 41.184s
...

$ repo_dev sync -j 20 -l
prebuilts/asuite/: discarding 1 commits
prebuilts/runtime/: discarding 1 commits
...
repo sync has finished successfully.

+ With superproject-override.xml and test it.

  $ ls -l .repo/exp-superproject/
  total 176
  drwxr-xr-x 7 rtenneti primarygroup   4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
  -rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 superproject_override.xml
  rtenneti@rtenneti:~/work/android/src/aosp$ repo_dev sync -j 20 -l
  ...
  repo sync has finished successfully.

+ Rename the file superproject-override.xml and test it.
  $ ls -l .repo/exp-superproject/
  total 176
  drwxr-xr-x 7 rtenneti primarygroup   4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
  -rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 temp.xml

  $ repo_dev sync -j 20 -l
  Checking out:  1% (12/1031) platform/external/rust/crates/fallible-streaming-iteexternal/linux-kselftest/: discarding 1 commits
  prebuilts/remoteexecution-client/: discarding 1 commits
  Checking out: 51% (536/1031) platform/prebuilts/gcc/darwin-x86/aarch64/....
  ....
  Checking out: 100% (1031/1031), done in 5.478s
  repo sync has finished successfully.

Bug: [google internal] b/184368268
Change-Id: I3aba5872e4f7c299977b92c2a39847ef28698c5a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312962
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-07-28 16:12:53 +00:00
Xin Li
927d29a8af Update manifest_name with the returned manifest name for superproject.
This is similar to smart sync, allowing sync to benefit from the patched
manifest.

Bug: [google internal] b/190688390
Change-Id: I158a80afceca606dcd81ec76b2caede369f7ed03
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Xin Li <delphij@google.com>
2021-07-14 22:51:33 +00:00
Raman Tenneti
8db30d686a superproject - More friendly user message when superproject failed.
superproject is going to be default for some users. This change
doesn't fail for repo init or repo sync if source couldn't be synced
because of errors in superproject and superproject=true in the config
file. The commands will fail if --use-superproject is specified on
the command line explicitly.

The error messages are logged with trace2 event logs and will be
monitored.

+ sync - When there are errors with superproject and git_superproject
  says it is fatal failure, sync will exit only when --use-superproject
  option is specified on the command line.

+ init - command doesn't fail *if there are any superproject errors),
  but it will print a warning message and logs message via trace2 event
  logs. For fatal errors, init will exit only when --use-superproject
  option is specified on the command line.

+ All git commands log the command that is being executed so trace2
  event logs will know the manifest, remote url and the branch name.

There is no functional change other than fatal errors are honored with
--use-supeproject option with init/sync commands.

Tested the code with the following commands.

$ ./run_tests -v

Test 1 - sync'ing without errors
--------------------------------
Added the following lines to '~/.repoconfig/config
[repo]
	superproject = true

  $ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2
  NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
  repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2

  warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning

  Your identity is: Raman Tenneti <rtenneti@google.com>
  If you want to change this, please re-run 'repo init' with --config-name

  repo has been initialized in /usr/local/google/home/rtenneti/work/drive2/android/test

$ repo_dev sync
remote: Total 4 (delta 1), reused 4 (delta 1)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/drive2/android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
...

Test 2 - init and sync fail when --use-superproject option is passed
--------------------------------------------------------------------
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 --use-superproject
remote: Total 57 (delta 16), reused 56 (delta 16)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2

warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning

rtenneti@rtenneti2:~/work/drive2/android/test$ repo_dev sync --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2

warning: Cannot get project commit ids from manifest
warning: Update of revisionId from superproject has failed, repo sync will not use superproject to fetch the source.  Please resync with the --no-use-superproject option to avoid this repo warning.

Test 3 - git fetch command fails and git command is printed
-----------------------------------------------------------
With config change
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2
...
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/test/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Performing initial setup for superproject; this might take several minutes.
repo: error: git fetch call failed,command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2

warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning

Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name

repo has been initialized in ....

Test 4 - no superproject tag
-----------------------------
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b pie-dev
...
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: .../android/pie_dev/.repo/manifest.xml
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning

Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name

repo has been initialized in ...

$ repo_dev sync
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: /usr/local/google/home/rtenneti/work/drive2/android/pie_dev/.repo/manifest.xml
warning: Cannot get project commit ids from manifest
warning: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option to avoid the repo warning.

Bug: [google internal] b/192614798
Bug: [google internal] b/Bug: [google internal] b/192614798

Change-Id: I9a97a0e7d9e609fad151bd7dd9cfc523eaa887cd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311502
Reviewed-by: Amith Dsouza <amithds@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-07-14 00:41:36 +00:00
Xin Li
e39d8b36f6 Fix an issue that repo can't see projects declared in a local manifest.
When loading of superproject failed, we were resetting the manifest to
None, and later code would reload the manifest to see if there are
submodules, which would load the non-local manifest, causing sync with
superproject to fail.

Address this by setting the manifest_name to opt.manifest_name instead.

Bug: [google internal] b/189139268
Change-Id: I3616512e1c4b73e7eca0d83fd1fc474b825adbbf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311102
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
2021-07-13 23:35:07 +00:00
Mike Frysinger
5892973212 sync: dump git output on errors
Bug: https://crbug.com/gerrit/14700
Change-Id: I1ae53dc7f3792b7e8f11d73f706864fb6591eee8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-07-04 02:28:00 +00:00
Xin Li
0cb6e92ac5 Add the ability to administratively enroll repo into using superproject.
Repo will remember a choice and an expiration time of the choice, per
user, about whether to use superproject by default. When not specified
from command line and the choice is not expired, repo would use the
user default value.

When a user default value is not present and when the system wide
enable default is provided in git's system configuration, repo would
ask the user for a confirmation which will be valid for two weeks.

git_config.py: Add support for system config. When reading system
	config, we would use --system to avoid hardcoding a path as the
	value may be different on some other distributions.

git_superproject.py: Add a new subroutine, _UseSuperproject(), which
	returns whether superproject should be used and whether it
	is from a user configuration.

	The value is determined in the following order:

	1. If the user specifies either --use-superproject or
	--no-use-superproject, then that choice is being used.

	2. If neither is specified, we would then check the saved value
	(upon repo init) and use that choice when there was a choice.

	3. We then check if there is a saved and unexpired value for
	user's choice in their ~/.gitconfig, and use the unexpired
	choice, if available.

	4. Finally, if all the above didn't give us a decision, and if
	the git system configuration is providing a rollout hint, present
	a prompt to user for their decision and save it in ~/.gitconfig.

subcmds/sync.py: Make use of the new UseSuperproject() provided by
git_superproject.py.

While there also silent stderr from git describe when determining the
version of repo.

Bug: [google internal] b/190688390
Change-Id: Iad3ee03026342ee500e5d65e2f0fa600d7637613
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309762
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
2021-07-01 16:27:43 +00:00
Raman Tenneti
784e16f3aa superproject: Don't exit if superproject tag doesn't exist in manifest.
Don't exit if there are missing commit ids in superproject.

This change implements the following suggestion from delphij@:

"we should note the event (so we know that --use-superproject but there
 were some errors, e.g. manifest didn't specify commit id for some
 reason, or if there is no superproject but --use-superproject is
 used), print out a message telling the use that this is not support,
 but continue as if --no-use-superproject was specified?"

Changes:

superproject:
+ Added git_trace2_event_log as an argument to the constructor.
+ Sync method returns SyncResult a NamedTuple of
  ++ success - True if sync of superproject is successful, or False.
  ++ fatal - True if caller should exit, Or False.
+ UpdateProjectsRevisionId returns UpdateProjectsResult a NamedTuple of
   ++ manifest_path - path name of the overriding manifest file instead
      of None
   ++ fatal - True if caller should exit, Or False
+ _GetAllProjectsCommitIds returns CommitIdsResult a NamedTuple of
  ++ commit_ids - a dictionary with the projects/commit ids on success,
     otherwise None
  ++ fatal - True if caller should exit, Or False
+ Added  _SkipUpdatingProjectRevisionId a helper function to see if a
  project's revision id needs to be updated or not. This function is
  used to exclude projects from local manifest file.
+ Added the following error events into git_trace2_event_log
  ++ If superproject is missing in a manifest
  ++ If there are missing commit ids for projects.

command.py:
+ Deleted unused import - platform
+ Added git_trace2_event_log as a member so all subcmds can log error
  events.

main.py:
+ Initialized git_trace2_event_log as a member of command object.

init.py:
+ Deleted unused import - optparse

init.py:
+ Called sys.exit only if Sync returns exit=True

sync.py:
+ Called sys.exit only if Superproject's UpdateProjectsRevisionId returns
  exit=True
+ Reloaded the manifest only if manifest path is returned by
  UpdateProjectsRevisionId. If not, fall back to the old way of doing
  repo sync.

test_git_superproject:
+ Added code to verify error events are being logged.
+ Added a test for no superproject tag
+ Added test for UpdateProjectsRevisionId not updating the revision id
  with the commit ids.

Tested the code with the following commands.

+ Positive test case with aosp-master.
  $ repo_dev init -u persistent-https://android.git.corp.google.com/platform/manifest -b master --use-superproject
  NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
  .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.

  Your identity is: Raman Tenneti <rtenneti@google.com>
  If you want to change this, please re-run 'repo init' with --config-name

  repo has been initialized in .../android/aosp

  $ repo_dev sync -j40 --use-superproject
  remote: Total 12 (delta 4), reused 12 (delta 4)
  NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
  .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
  ...
  repo sync has finished successfully.

+ Negative test case without superproject tag.
  $ repo_dev sync -j40 --use-superproject
  NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
  repo error: superproject tag is not defined in manifest: .../android/aosp/.repo/manifest.xml
  error: Cannot get project commit ids from manifest
  error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
  ...
  Checking out: 100% (1022/1022), done in 3.589s
  repo sync has finished successfully.

+ Test for missing commit_id for a project.
  $ repo_dev sync -j40 --use-superproject
  NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
  .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
  error: please file a bug using go/repo-bug to report missing commit_ids for: ['build/blueprint']
  error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
  ...
  Checking out: 100% (1022/1022), done in 3.364s
  repo sync has finished successfully.

$ ./run_tests -v
  ...
  ...== 164 passed in 2.87s ==...

Bug: [google internal] b/189371541
Change-Id: I5ea49f87e8fa41be590fc0c914573e16c8cdfcfa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309162
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-06-16 04:48:35 +00:00
Mike Frysinger
4f21054c28 commands: document the "common" class attribute
Switch it to uppercase to make it clear it's a constant, and add
documentation so its usage is clear.

Change-Id: I6d281a66a90b5908b3131585c9945e88cfe815ea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309322
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-06-15 06:07:37 +00:00
Peter Kjellerstedt
d177609cb0 sync: Sync submodules correctly again
Commit b4429439 (sync: refactor main fetch loop) broke syncing of
submodules with sync-s="true". The first `repo sync` would just fetch
the superrepo, while the second `repo sync` would fetch the submodules.
This was due to the new _FetchMain() function not passing back the
all_projects variable that it had modified.

Change-Id: Ie8187cde7bb894e4e9a6b76c2aed83873d9f69a4
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/307065
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-20 16:23:10 +00:00
Mike Frysinger
339f2df1dd ssh: rewrite proxy management for multiprocessing usage
We changed sync to use multiprocessing for parallel work.  This broke
the ssh proxy code as it's all based on threads.  Rewrite the logic to
be multiprocessing safe.

Now instead of the module acting as a stateful object, callers have to
instantiate a new ProxyManager class that holds all the state, an pass
that down to any users.

Bug: https://crbug.com/gerrit/12389
Change-Id: I4b1af116f7306b91e825d3c56fb4274c9b033562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305486
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-05-10 21:16:06 +00:00
Mike Frysinger
19e409c818 ssh: move proxy usage to the sync subcommand
The only time we really need ssh proxies is when we want to run many
connections and reuse them.  That only happens when running sync.
Every other command makes at most two connections, and even then it's
only one or none.  So the effort of setting up & tearing down ssh
proxies isn't worth it most of the time.

The big reason we want to move this logic to sync is that it's now
using multiprocessing for parallel work.  The current ssh proxy code
is all based on threads, which means none of the logic is working
correctly.  The current ssh design makes it hard to fix when all of
the state lives in the global/module scope.

So the first step to fixing this is top move the setup & teardown to
the one place that really needs it: sync.  No other commands will use
proxies anymore, just direct connections.

Bug: https://crbug.com/gerrit/12389
Change-Id: Ibd351acdec39a87562b3013637c5df4ea34e03c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305485
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-10 21:10:29 +00:00
Raman Tenneti
23ea754524 sync: added --no-use-superproject to disable superproject.
Tested the code with the following commands.

$ ./run_tests -v

$ repo_dev sync -c -j8 --no-use-superproject
Fetching: 100% (1041/1041), done in 1m22.743s

$ repo_dev sync -c -j8 --use-superproject
WARNING: --use-superproject is experimental and not for general use
..

Bug: [google internal] b/187459275
Change-Id: I3f4269df38cd24a21723e8b2be5a1f013e7b5a91
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305682
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-08 00:04:00 +00:00
Raman Tenneti
f907ced0fe sync: Recommend using --no-use-superproject if sync fails.
If superproject was not available for a branch, then the next
repo sync would also fail because --use-superproject is
remembered across repo init. In such cases, hoping the hint to
to use --no-use-superproject will help.

Tested the code with the following commands and by forcing
a failure.

$ ./run_tests -v

Bug: [google internal] b/187459275
Change-Id: Ie250812b7ba83afc230b5b1d154ba11f245f8b8a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305622
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-05-07 20:15:50 +00:00
Mike Frysinger
b44294395f sync: refactor main fetch loop
This is a large chunk of code that is largely isolated.  Move it into
a class method to make it easier to manage & reason about, and in a
follow up CL, easier to scope.

Bug: https://crbug.com/gerrit/12389
Change-Id: I0c69d95a9e03478d347b761580b2343bffa012d5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305484
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-05-06 19:46:09 +00:00
Mike Frysinger
219431e1c9 sync: fix recursive fetching
Commit b2fa30a2b8 ("sync: switch network
fetch to multiprocessing") accidentally changed the variable passed to
the 2nd fetch call from |missing| to |to_fetch| due to a copy & paste
of the earlier changed logic.  Undo that to fix git submodule fetching.

Bug: https://crbug.com/gerrit/14489
Change-Id: I627954f80fd2e80d9d5809b530aa6b0ef9260abb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305262
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-05 02:42:23 +00:00
Mike Frysinger
05638bf771 sync: use manifest_name passed in
Commit fb527e3f52 ("sync: create dedicated
manifest project update func") refactored code from the main body into a
dedicated method.  The manifest_name was passed as an argument, but never
used it, and instead reaches back out to the command line options.  This
ignores the logic in the main loop where manifest_name might have changed
(like when using smart sync).

Change-Id: I4b84638fbb10c2b6f8f4b555e1475b0669c2daf4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305148
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 20:00:50 +00:00
Mike Frysinger
c99322a6a9 sync: switch to multiprocessing.Event
We've switched most of this command over to multiprocessing and off
of _threading, so do the Event object too.  The APIs are the same
between the modules, so we shouldn't need to update anything else.

Change-Id: I52d31f1c6ef2bcbe7bbc1dd1add79a8d5d08784a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305147
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 20:00:41 +00:00
Mike Frysinger
14208f4c93 sync: fix logic error with linkfile errors
Make sure err_update_linkfiles is always initalized.

Bug: https://crbug.com/gerrit/11008
Change-Id: I7bdd91f82507608ef967daf0fa0f9c859454e19f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305146
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 20:00:20 +00:00
Mike Frysinger
c177f944d9 subcmds: force consistent help text format
We're inconsistent with help text as to whether it uses title case and
whether it ends in a period.  Add a test to enforce a standard, and use
the style that Python optparse & argparse use themselves (e.g. with the
--help option): always lowercase, and never trailing period.

Change-Id: Ic1defae23daeac0ac9116aaf487427f50b34050d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305144
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 16:40:53 +00:00
Mike Frysinger
d68ed63328 init/sync: add --no-tags to match --tags
While this provides a way to undo earlier command line options (e.g.
`repo sync --tags --no-tags`) which can be helpful for scripting &
automation, this more importantly allows the user to override the
manifest settings for syncing tags from a project.

Bug: https://crbug.com/gerrit/12401
Change-Id: Id4c36cd82e6ca7cb073b5d63a09f6c7ccdebba83
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304904
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 11:32:17 +00:00
Mike Frysinger
7356114d90 add --no-current-branch option to invert --current-branch
For most commands, this is more about providing a way to undo earlier
command line options (e.g. `repo info -c --no-current-branch`) which
can be helpful for scripting & automation.  But for the sync command,
this is helpful to undo the setting that exists in the manifest itself.

With this in place, tweak the sync current_branch_only logic to only
apply the manifest settings when the user hasn't specified a command
line option.

Bug: https://crbug.com/gerrit/12401
Change-Id: I21e2384624680cc740d1b5d1e49c50589d2fe6a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304903
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 11:31:48 +00:00
Raman Tenneti
feb28914bd superproject: Don't update the commit ids of projects if remote is different.
1) Skip setting the revision id (commit id) for the projects whose
   remote doesn't match superproject's remote.
2) exp-superproject/superproject_override.xml includes local_manfiest's
   projects. When we load this XML, don't reload projects from local.xml
   (otherwise we will get duplicate projects errors).

Tested the code with the following commands.

$ ./run_tests -v

+ Test with local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml  .repo/local_manifests/local.xml

  $ repo_dev sync -c -j8

+ Test without local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
  $ repo_dev sync -c -j8

Bug: [google internal] b/186395810
Change-Id: I4e9d4ac2d94a9fc0cef0ccd787b6310758009e86
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304882
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-03 05:13:23 +00:00
jiajia tang
a590e640a6 Update copyfile and linkfile if manifest updated
Currently, copyfiles and linkfiles which marked by
"<copyfile/>" and "<linkfile/>" in manifest will
be created by first exec 'repo sync'.
But if some "<copyfile/>" or "<linkfile/>" are removed
in manifest, then 'repo sync', these removed item
dest can not be removed in the sourcecode workspace.

This patch is intent to fix this issue, by save a
'copy-link-files.json' in .repo and then compared with
new dest path when next sync. If any "<copyfile/>" or
"<linkfile/>" were removed, the dest path will be
removed in sourcecode at the same time.

Bug: https://crbug.com/gerrit/11008
Change-Id: I6b7b41e94df0f9e6e52801ec755951a4c572d05d
Signed-off-by: jiajia tang <tangjiajia@xiaomi.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304202
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-01 13:26:08 +00:00
Mike Frysinger
d41eed0b36 sync: fix missing import for -q
Some refactors during review dropped this import when it was reworked,
but it's still needed when using the --quiet setting.

Change-Id: I6d9302ef5a056e52415ea63f35bad592b9dfa75d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303942
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-21 15:37:16 +00:00
Raman Tenneti
ad8aa69772 sync: only print error.GitError, don't raise that exception.
In _FetchOne & _CheckOne, only print error.GitError exception,
but other exceptions are still thrown

Fixes the GitError exceptions from /usr/lib/python3.8/multiprocessing/pool.py
exiting the repo sync.

Tested the code with the following commands and verified repo sync
continues after fetch error because of an invalid SHA1.

$ ./run_tests -v

$ python3 ~/work/repo/git-repo/repo sync -m manifest_P21623846.xml -j32
...
error.GitError: Cannot fetch platform/vendor/google_devices/redbull/proprietary update-ref: fatal: d5a99e518f09d6abb0c0dfa899594e1ea6232459^0: not a valid SHA1
....

An error like the following when jobs=1
  error.GitError: Cannot checkout platform/vendor/qcom/sdm845/proprietary/qcrilOemHook: Cannot initialize work tree for platform/vendor/qcom/sdm845/proprietary/qcrilOemHook

Bug: https://crbug.com/gerrit/14392
Change-Id: I8922ad6c07c733125419f5698b0f7e32d70c7905
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303544
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-04-15 22:43:07 +00:00
Mike Frysinger
b5d075d04f command: add a helper for the parallel execution boilerplate
Now that we have a bunch of subcommands doing parallel execution, a
common pattern arises that we can factor out for most of them.  We
leave forall alone as it's a bit too complicated atm to cut over.

Change-Id: I3617a4f7c66142bcd1ab030cb4cca698a65010ac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301942
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-04-15 05:10:16 +00:00
Mike Frysinger
151701e85f progress: hide progress bar when --quiet
We want progress bars in the default output mode, but not when the
user specifies --quiet.  Add a setting to the Progress bar class so
it takes care of not displaying anything itself rather than having
to update every subcommand to conditionally setup & call the object.

Change-Id: I1134993bffc5437bc22e26be11a512125f10597f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303225
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-13 22:25:26 +00:00
Mike Frysinger
9180a07b8f command: make --verbose/--quiet available to all subcommands
Add new CommonOptions entry points to move the existing --jobs to,
and relocate all --verbose/--quiet options to that.  This provides
both a consistent interface for users as well as for code.

Change-Id: Ifaf83b88872421f4749b073c472b4a67ca6c0437
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303224
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-13 22:25:17 +00:00
Raman Tenneti
f32f243ff8 init: Added --partial-clone-exclude option.
partial-clone-exclude option excludes projects during
partial clone. This is a comma-delimited project names
(from manifest.xml). This option is persisted and it
is used by the sync command.

A project that has been unparital'ed will remain unpartial if
that project's name is specified in the --partial-clone-exclude
option. The project name should match exactly.

Added
$ ./run_tests -v

Bug: [google internal] b/175712967
"I can't "unpartial" my androidx-main checkout"

$ rm -rf androidx-main/
$ mkdir androidx-main/
$ cd androidx-main/
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml
$ repo_dev sync -c -j8

+ Verify a project is partial
$ cd frameworks/support/
$ git config -l | grep  'partial'

+ Unpartial a project.
$ /google/bin/releases/android/git_repack/git_unpartial

+ Verify project is unpartial
$ git config -l | grep  'partial'
$ cd ../..

+ Exclude the project from being unparial'ed after init and sync.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml

+ Verify project is unpartial
$ cd frameworks/support/
$ git config -l | grep  'partial'
$ cd ../..
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep  'partial'
$ cd ../..

+ Remove the project from exclude list and verify that project is partially cloned.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep  'partial'

Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-04-13 15:47:10 +00:00
Mike Frysinger
49de8ef584 sync: add separate --jobs options for different steps
The number of jobs one wants to run against the network tends to
factor differently from the number of jobs one wants to run when
checking out local projects.  The former is constrained by your
internet connection & server limits while the later is constrained
by your local computer's CPU & storage I/O.  People with beefier
computers probably want to keep the network/server jobs bounded a
bit lower than the local/checkout jobs.

Change-Id: Ia27ab682c62c09d244a8a1427b1c65acf0116c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302804
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-09 15:58:56 +00:00
Mike Frysinger
65af2602b5 sync: add progress bar to garbage collection phase
This can take a few seconds, if not a lot more, so add a progress bar
so users understand what's going on.

Change-Id: I5b4b54c1bbb9ec18728f979521310f7087afaa5c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-09 15:58:21 +00:00
Mike Frysinger
347f9ed393 sync: rework selfupdate logic
The current logic has a downside in that it doesn't sync to the latest
signed version available if the latest commit itself is unsigned.  This
can come up when using the "main" branch as it is sometimes signed, but
often not as it's holding the latest merged commits.  When people use
the main branch, it's to get early testing on versions tagged but not
yet released, and we don't want them to get stuck indefinitely on that
old version of repo.

For example, this series of events:
* "stable" is at v2.12.
* "main" is tagged with v2.13.
* early testers use --repo-rev main to get v2.13.
* new commits are merged to "main".
* "main" is tagged with v2.14.
* new commits are merged to "main".
* devs who had synced in the past to test v2.13 are stuck on v2.13.
  repo sees "main" is unsigned and so doesn't try to upgrade at all.

The only way to get unwedged is to re-run `repo init --repo-rev main`,
or to manually sync once with repo verification disabled, or for us to
leave "main" signed for a while and hope devs will sync in that window.

The new logic is that whenever changes are available, we switch to the
latest signed tag.  We also replace some of the duplicated verification
code in the sync command with the newer wrapper logic.  This handles a
couple of important scenarios inaddition to above:
* rollback (e.g. v2.13.8 -> v2.13.7)
* do not trash uncommitted changes (in case of ad-hoc testing)
* switch tag histories (e.g. v2.13.8 -> v2.13.8-cr1)

Change-Id: I5b45ba1dd26a7c582700ee3711f303dc7538579b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300122
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-09 03:16:45 +00:00
Mike Frysinger
b2fa30a2b8 sync: switch network fetch to multiprocessing
This avoids GIL limitations with using threads for parallel processing.

This reworks the fetch logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.

We have to tweak the chunking logic a little here because multiprocessing
favors batching over returning immediate results when using a larger value
for chunksize.  When a single job can be quite slow, this tradeoff is not
good UX.

Bug: https://crbug.com/gerrit/12389
Change-Id: I0f0512d15ad7332d1eb28aff52c29d378acc9e1d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298642
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:52:57 +00:00
Raman Tenneti
2ae44d7029 sync: imply -c if --use-superproject option is used.
Tested the code with the following commands.

$ ./run_tests -v

Bug: [google internal] b/183232698
Bug: https://crbug.com/gerrit/13707

$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync --use-superproject
$ repo_dev sync
  real	0m8.046s
  user	0m2.866s
  sys	0m2.457s

Second time repo sync took only 8 seconds and verified by printing that
urrent_branch_only is True in project.py's Sync_NetworkHalf function.

Change-Id: Ic48efb23ea427dfa36e12a5c49973d6ae776d818
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301182
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-03-24 15:17:19 +00:00
Mike Frysinger
511a0e54f5 sync: fix reporting of failed local checkouts
The refactor to multiprocessing broke status reporting slightly when
checking out projects.  Make sure we mark the step as failed if any
of the projects failed, not just when --fail-fast is set.

Change-Id: I0efb56ce83b068b2c334046df3fef23d797599c9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299882
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-15 16:54:23 +00:00
Raman Tenneti
ef99ec07b4 superproject: Display status messages during repo init/sync.
Superproject objects accept the optional argument “quiet”.
The following progress messages are displayed if quiet is false.

Displayed the following message whenever we find we have to make a new
folder (aka new remote), because if you started with repo init android
and later do googleplex-android that is when it will be slow.

"<location>: Performing initial setup for superproject; this might take
several minutes.".

After fetch completion, added the following notification:
"<location>: Initial setup for superproject completed."

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL.

$ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev  --partial-clone --clone-filter=blob:limit=10M --repo-rev=main  --use-superproject

Bug: [google internal] b/181178282
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-04 20:07:52 +00:00
Mike Frysinger
ebf04a4404 sync: switch local checkout to multiprocessing
This avoids GIL limitations with using threads for parallel processing.
In a CrOS checkout with ~1000 repos, the nop case goes from ~6 sec down
to ~4 sec with -j8.  Not a big deal, but shows that this actually works
to speed things up unlike the threading model.

This reworks the checkout logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.

Bug: https://crbug.com/gerrit/12389
Change-Id: I143e5e3f7158e83ea67e2d14e5552153a874248a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298063
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-27 19:55:14 +00:00
Mike Frysinger
7b586f231b sync: capture all git output by default
The default sync output should show a progress bar only for successful
commands, and the error output for any commands that fail.  Implement
that policy here.

Bug: https://crbug.com/gerrit/11293
Change-Id: I85716032201b6e2b45df876b07dd79cb2c1447a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297905
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 20:13:18 +00:00
Mike Frysinger
fbb95a4342 progress/sync: include active number of jobs
Provide a bit more info to users that things are actively running.

Bug: https://crbug.com/gerrit/11293
Change-Id: Ie8eeaa8804d1ca71cf5c78ad850fa2d17d26208c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297904
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 20:13:18 +00:00
Mike Frysinger
4e05f650e0 progress: always enable always_print_percentage
The idea for skipping some progress updates was to avoid spending
too much time on the progress bar itself.  Unfortunately, for large
projects (100s if not 1000s) of repos, we get into the situation
with large/slow checkouts that we skip showing updates when a repo
finishes, but not enough repos finished to increase the percent.

Since the progress bar should be relatively fast compared to the
actual network & local dick operations, have it show an update
whenever the caller requests it.  A test with ~1000 repos shows
that the progress bar in total adds <100ms.

Bug: https://crbug.com/gerrit/11293
Change-Id: I708a0c4bd923c59c7691a5b48ae33eb6fca4cd14
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297903
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 20:13:18 +00:00
Raman Tenneti
55d6a5a3a2 sync: use superproject if manifest's config has superproject enabled.
If --use-superproject is passed as argument to "repo init", then
--use-superproject need not be specified during "repo sync".

Tested the code with the following commands.

$ time repo_dev sync -c -j32
...
WARNING: --use-superproject is experimental and not for general use

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: Ibb33f3038a2515f74a6c4f7cb785d354b26ee680
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298102
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Ian Kasprzak <iankaz@google.com>
2021-02-25 16:35:53 +00:00
Mike Frysinger
be24a54d9c sync: update event is_set API
Python 3 renamed this method from isSet to is_set.

Change-Id: I8f9bb0b302d55873bed3cb20f2d994fa2d082157
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297742
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-23 17:56:49 +00:00
Mike Frysinger
6a2400a4d0 command: unify --job option & default values
Extend the Command class to support adding the --jobs option to the
parser if the command declares it supports running in parallel.  Also
pull the default value used for the number of local jobs into the
command module so local commands can share it.

Change-Id: I22b0f8d2cf69875013cec657b8e6c4385549ccac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297024
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-02-22 22:51:07 +00:00
Mike Frysinger
c47a235bc5 trim redundant pass statements
Clean up a few linter warnings.

Change-Id: I531d0263a202435d32d83d87ec24998f4051639c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297062
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-16 19:23:00 +00:00
Mike Frysinger
fb21d6ab64 sync: use subprocess.run to verify tags
The code is a bit simpler & easier to reason about.

Change-Id: I149729c7d01434b08b58cc9715dcf0f0d11201c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297022
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-16 16:26:41 +00:00
Raman Tenneti
21dce3d8b3 init: added --use-superproject option to clone superproject.
Added --no-use-superproject to repo and init.py to disable use of
manifest superprojects.

Replaced the term "sha" with "commit id".

Added _GetBranch method to Superproject object.

Moved shared code between init and sync into SyncSuperproject function.
This function either does git clone or git fetch. If git fetch fails
it does git clone.

Changed Superproject constructor to accept manifest, repodir and branch
to avoid passing them to multiple functions as argument.

Changed functions that were raising exceptions to return either True
or False.

Saved the --use-superproject option in config as repo.superproject.
Updated internal-fs-layout.md document.

Updated the tests to work with the new API changes in Superproject.

Performance for the first time sync has improved from 20 minutes to
around 15 minutes.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL.

$ repo init took around 20 seconds longer because of cloning of superproject.

$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real	0m35.919s
user	0m21.947s
sys	0m8.977s

First run
$ time repo sync --use-superproject
...
real	16m41.982s
user	100m6.916s
sys	19m18.753s

No difference in repo sync time after the first run.

Bug: [google internal] b/179090734
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707

Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-11 18:59:29 +00:00
Mike Frysinger
e3315bb49a diffmanifests/sync: simplify repodir lookup
We have access to repodir on the command object itself, so we don't
need to pull it indirectly out of the manifest object.

Change-Id: I8688fb1c84979825efa966dc787e78c6f7ba3823
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296542
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-11 02:38:53 +00:00
Raman Tenneti
8d43dea6ea sync: pass --bare option when doing git clone of superproject.
Changed "git pull" to "git fetch" as we are using --bare option. Used the
following command to fetch:
  git fetch origin +refs/heads/*:refs/heads/* --prune

Pass --branch argument to Superproject's UpdateProjectsRevisionId function.

Returned False/None when directories don't exist instead of raise
GitError exception from _Fetch and _LsTree functions. The caller of Fetch
does Clone if Fetch fails.

Tested the code with the following commands.

$ ./run_tests -v

Tested the init and sync code by copying all the repo changes into my Android
AOSP checkout and running repo sync with --use-superproject option.

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-02-08 17:34:55 +00:00
Raman Tenneti
1fd7bc2438 sync: superproject performance changes.
After updating all project’s revsionIds with the SHAs from superproject,
write the updated manifest into superproject_override.xml file. Reload
that file for future Reloads. This file is created in exp-superproject
directory.

Moved most of the code that is superproject specific into
git_superproject.py and wrote test code.

If git pull fails, did a git clone of the superproject.

We saw performance gains for consecutive repo sync's. The time to sync
went down from around 120 secs to 40 secs when repo sync is executed
consecutively.

Tested the code with the following commands.

$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.

First run
$ time repo sync --use-superproject
...
real	21m3.745s
user	97m59.380s
sys	19m11.286s

After two consecutive sync runs
$ time repo sync -c -j8 --use-superproject
real	0m39.626s
user	0m29.937s
sys	0m38.155s

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>

Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-07 22:25:38 +00:00
Anders Björklund
2a2da80ba6 sync: Disable info about disabling pruning when quiet
If you have a lot of shared projects, it spams.

Bug: https://crbug.com/gerrit/13961
Change-Id: If3f5baef65930830af9a2cd01a1b593dd518ab09
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294049
Tested-by: Anders Björklund <anders.bjorklund.2@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-22 11:43:13 +00:00
Raman Tenneti
6a872c9dae sync: Added --use-superproject option and support for superproject.
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,

.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-21 19:41:52 +00:00
Mike Frysinger
acf63b2892 drop pyversion & is_python3 checking
We're committed to Python 3 at this point, so purge all the
is_python3 related dynamic checks.

Bug: https://crbug.com/gerrit/10418
Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:58 +00:00
Mike Frysinger
784ccfc040 strip python2-only coding:utf-8 & print_function settings
We're committed to Python 3 at this point, so clean up boilerplate.

Bug: https://crbug.com/gerrit/10418
Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:05 +00:00
Erwan Yvin
dc5c4d1d11 sync: respect --force-sync when fetching manifest project updates
The --force-sync option was being passed down for all updates except
for the manifest project, so add that there too.

Bug: https://crbug.com/gerrit/11034
Change-Id: I33818b652f828c6b847dbc70f1fedfac5ac17bbe
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228146
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-11-17 03:06:06 +00:00
Roger Shimizu
ac29ac397f subcmds/sync.py: Fix typo in help
Change-Id: I70b63477241284249e395b8b0a220cb6f44f836b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/270183
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
Tested-by: David Pursehouse <dpursehouse@digital.ai>
2020-06-06 23:46:00 +00:00
Mike Frysinger
7ae210a15b sync: fix duplicate word in description
Bug: https://crbug.com/gerrit/12814
Change-Id: Id722eec9a59dded588f13bc605ce2c94b4047265
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268739
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
2020-05-24 23:51:28 +00:00
Xin Li
d79a4bc51b Make partial clone imply no-clone-bundle by default.
For large projects, clone bundle is useful because it provided a way to
efficiently transfer a large portion of git objects through CDN, without
needing to interact with git server. However, with partial clones, the
intention is to not download most of the objects, so the use of clone
bundles would defeat the space savings normally seen with partial
clones, as they are downloaded before the first fetch.

A new option, --clone-bundle is added to override this behavior.
Add a new repo.clonebundle variable which remembers the choice if
explicitly given from command line at repo init.

Change-Id: I03638474af303a82af34579e16cd4700690b5f43
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268452
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-05-21 19:47:36 +00:00
George Engelbrecht
9bc283e49b sync: add retry to fetch operations
Add retries with exponential backoff and jitter to the fetch
operations. By default don't change behavior and enable
behind the new flag '--fetch-retries'.

Bug: https://crbug.com/1061473

Change-Id: I492710843985d00f81cbe3402dc56f2d21a45b35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/261576
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: George Engelbrecht <engeg@google.com>
2020-04-02 21:17:54 +00:00
Mike Frysinger
58ac1678e8 init: rename --repo-branch to --repo-rev
We refer to this as "revision" in help text, and in REPO_REV envvar,
so rename to --repo-rev to be consistent.  We keep --repo-branch for
backwards compatibility, but as a hidden option.

Bug: https://crbug.com/gerrit/11045
Change-Id: I1ecc282fba32917ed78a63850360c08469db849a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259352
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-03-18 00:24:43 +00:00
Mike Frysinger
23c900f105 sync: warn if not tracking a branch
Since tracking a branch prevents repo from updating, make sure we
warn people about the situation when using `repo sync`.

Bug: https://crbug.com/gerrit/11045
Change-Id: I966513f510827cc93194f8df176c6745946bd739
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258892
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-03-17 05:38:19 +00:00
Mike Frysinger
e20da3eeed sync: fix os.environ logic errors
This is a dict to index, not a function to call.

Change-Id: I0117eeaaa8b2ef4762ab6f0d22f9ffdaee961f52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258132
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-03-07 13:10:14 +00:00
David Pursehouse
aa611a2ca2 sync: Fix flake8 E125 and E129 warnings
- E129 visually indented line with same indent as next logical line
- E125 continuation line with same indent as next logical line

Fixed automatically by:

 autopep8 --in-place --select E125,E129 subcmds/sync.py

Change-Id: Ia2f82f443e1e6a23ba22c6f9849c8485405aed0e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256092
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-20 02:17:08 +00:00
Mike Frysinger
c0d1866b35 project/sync: move DeleteProject helper to Project
Since deleting a source checkout involves a good bit of internal
knowledge of .repo/, move the DeleteProject helper out of the sync
code and into the Project class itself.  This allows us to add git
worktree support to it so we can unlock/unlink project checkouts.

Change-Id: If9af8bd4a9c7e29743827d8166bc3db81547ca50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256072
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-20 00:51:42 +00:00
Mike Frysinger
f81c72ed77 project: set core.repositoryFormatVersion=1 when using extensions
When using extensions, make sure we set the git repo format version
so git knows to check the extension compatibility.  We can add a
helper to the Project API to simplify this and make it foolproof.

Change-Id: I9ab6c32d92fe2b8e5df6e2b080ca71556332e909
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256035
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-02-19 23:44:10 +00:00
Mike Frysinger
979d5bdc3e add experimental git worktree support
This provides initial support for using git worktrees internally
instead of our own ad-hoc symlink tree.  It's been lightly tested
which is why it's not currently exposed via --help.

When people opt-in to worktrees in an existing repo client checkout,
no projects are migrated.  Instead, only new projects will use the
worktree method.  This allows for limited testing/opting in without
having to completely blow things away or get a second checkout.

Bug: https://crbug.com/gerrit/11486
Change-Id: Ic3ff891b30940a6ba497b406b2a387e0a8517ed8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254075
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-19 18:11:33 +00:00
Mike Frysinger
56ce3468b4 assume environment always accepts strings
Different Python & OS versions have different environ behavior wrt
accepted types & encoding.  Since we're migrating to be Python 3 only,
lets change our code to assume strings always work as that's what the
newer Python 3 does.  This will fail under Python 2 for some env vars,
mostly on Windows, but the effort of maintaining shim layers that can
handle these edge cases isn't worth it when we're dropping that code.

We leave the logic in the `repo` launcher for now as it is simple, and
we want it to be able to switch versions a bit longer than the rest of
the tree.

Here's the support table:
          |    *NIX      |         Windows           |
 Python 2 | ASCII string | str or bytes, not unicode |
 Python 3 | str or bytes | str only                  |

Windows uses strings natively in its environment all the time.  But it
doesn't allow unicode strings under Python 2, so we have to encode.

Python 2 on *NIX is funky in that it always lowers to ASCII, so we had
to manually encode to avoid errors regardless of unicode or str.

Python 3 on Windows & *NIX will accept strings.  *NIX will also accept
bytes but Windows will not.

Bug: https://crbug.com/gerrit/12145
Change-Id: I3cf8f95a06902754ea1f08ad4b28503f7063531b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248972
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-19 18:03:46 +00:00
Mike Frysinger
c58ec4dba1 avoid negative variables
Trying to use booleans with names like "no_xxx" are hard to follow due
to the double negatives.  Invert all of them so we only have positive
meanings to follow.

Change-Id: Ifd37d0368f97034d94aa2cf38db52c723ac0c6ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255493
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-19 00:24:43 +00:00
Collin Fijalkovich
e1191b3adb Open temporary cookie file as writable in sync.py
Named Temporary file defaults to mode 'w+b' which causes repo sync to
fail. By opening the tmpcookiefile in PersistentTransport.request as
writable, we are able to run sync successfully.

Bug: https://crbug.com/gerrit/12370
Test: Ran smartsync successfully
Change-Id: I01ddf915fc30eb3ff0e4d440a6f1aa261c63e88d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255692
Tested-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-02-18 19:20:53 +00:00
Mike Frysinger
521d01b2e0 sync: introduce --verbose option
This allows us to control sync output better by having three levels
of output: quiet (only errors), default (progress bars), verbose (all
the things).  For now, we just put the chatty "already have persistent
ref" message behind the verbose level.

Bug: https://crbug.com/gerrit/11293
Change-Id: Ia61333fd8085719f3e99edb7b466cdb04031b67f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255414
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-17 17:02:27 +00:00
Mike Frysinger
e19d9e1a65 sync: add a "finished" message
Some people find the existing output to be a bit confusing.  It spews
a lot of git output before exiting, but it's not exactly clear what
the final state is when things pass.  Add an explicit message.

Bug: https://crbug.com/gerrit/10501
Change-Id: I9de83b595d3185feb820005b8fc81c6adc55b357
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254732
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 20:54:57 +00:00
Mike Frysinger
6f1c626a9b drop old git_require checks
We've been requiring git-1.7.2 since Oct 2012, so we can safely drop
the individual checks sprinkled throughout the code base for older.

Change-Id: I1737fff7b3f27f475960b0bff9cb300aefd5d108
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253135
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 11:44:59 +00:00
David Pursehouse
145e35b805 Fix usage of bare 'except'
flake8 reports:

  E722 do not use bare 'except'

Replace them with 'except Exception' per [1] which says:

  Bare except will catch exceptions you almost certainly don't want
  to catch, including KeyboardInterrupt (the user hitting Ctrl+C) and
  Python-raised errors like SystemExit

  If you don't have a specific exception you're expecting, at least
  except Exception, which is the base type for all "Regular" exceptions.

[1] https://stackoverflow.com/a/54948581

Change-Id: Ic555ea9482645899f5b04040ddb6b24eadbf9062
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254606
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:49:25 +00:00
David Pursehouse
819827a42d Fix blank line issues reported by flake8
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E301,E302,E303,E305,E306

Manually fix issues in project.py caused by misuse of block comments.

Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:36:40 +00:00
David Pursehouse
abdf750061 Fix indentation issues reported by flake8
- E121 continuation line under-indented for hanging indent
- E122 continuation line missing indentation or outdented
- E125 continuation line with same indent as next logical line
- E126 continuation line over-indented for hanging indent
- E127 continuation line over-indented for visual indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E121,E122,E125,E126,E127,E128,E129,E131

Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:36:22 +00:00
David Pursehouse
54a4e6007a Fix various whitespace issues reported by pyflakes
- E201 whitespace after '['
- E202 whitespace before '}'
- E221 multiple spaces before operator
- E222 multiple spaces after operator
- E225 missing whitespace around operator
- E226 missing whitespace around arithmetic operator
- E231 missing whitespace after ','
- E261 at least two spaces before inline comment
- E271 multiple spaces after keyword

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E201,E202,E221,E222,E225,E226,E231,E261,E271

Change-Id: I367113eb8c847eb460532c7c2f8643f33040308c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254601
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:00:16 +00:00
David Pursehouse
42339d7e52 Remove redundant backslashes
fleka8 reports:

  E502 the backslash is redundant between brackets

Fixed automatically with autopep8:

  git-repo $ git ls-files | grep py$ | xargs autopep8 --in-place --select E502

Change-Id: I1486ae1d17206918474363daf518274c5be8daed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254602
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:45:44 +00:00
David Pursehouse
eeff3537de Fix tests for membership to use 'not in'
flake8 reports:

  E713 test for membership should be 'not in'

Change-Id: I4446be67c431b7267105b53478d2ceba2af758d7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254451
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:18:17 +00:00
David Pursehouse
e5913ae410 Fix flake8 E251 unexpected spaces around keyword / parameter equals
Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place --select E251

Change-Id: I58009e1c8c91c39745d559ac919be331d4cd9e77
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254598
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:17:08 +00:00
David Pursehouse
8f9e02231a Remove trailing blank lines
flake8 reports:

  W391 blank line at end of file

Change-Id: I5498b2de2d1268d4f1f4b9e1760f9fa93a6da4cd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254594
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 02:58:17 +00:00
Mike Frysinger
5f2b045195 sync: change how we preserve objects in shared repos
Some automatic git operations will prune objects on us, and not just
the gc step.  Normally we don't care, but with shared projects, we
will have multiple git checkouts with refs that the others cannot
see, but with a shared object dir.  Any pruning of objects based on
refs in just one repo can easily break the others.

git-2.7.0 introduced a preciousObjects setting which tells git to
never prune objects for this exact scenario: there might be refs in
some location that git is unable to see.

Change-Id: I781de27c5bbe1d4c70f0187566141c9cce088bd8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254392
Reviewed-by: Nasser Grainawi <nasser@codeaurora.org>
Reviewed-by: David Riley <davidriley@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-11 23:58:43 +00:00
Mike Frysinger
5a03308c5c sync: try to checkout repos across sync failures
Currently our default behavior is:
* Try to sync all repos
  * If any errors seen, exit
* Try to garbage collect all repos
  * If any errors seen, exit
* Try to update local project list
  * If any errors seen, exit
* Try to checkout out all local repos
  * If any errors seen, exit

Users find these incomplete syncs confusing, so lets try to complete
as much as possible by default and printing out summaries at the end.

Bug: https://crbug.com/gerrit/11293
Change-Id: Idd17cc9c3bbc574d8a0f08a30225dec7bfe414cb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238554
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-05 21:37:20 +00:00
Fredrik de Groot
cc960971f4 sync: add option to skip manifest update
The use case is any situation where your manifest does
not exist on server, but where you still want to do
full sync for the projects, without having your
workspace manifest switched to other branch or
forwarded to latest or similar.
This allows syncing to a historical manifest in git log,
that does not have a branch, as well as when integrating
something together that has not been pushed upstream yet.
Changes can also exist locally on a manifest that is
behind head, meaning not requiring rebase to latest.

Tested using:
  $ cd .repo/manifests/
  $ git checkout <any hash 1>
  $ <do local modifications>
  $ repo sync --no-manifest-update
  $ git checkout <any hash 2>
  $ repo sync --no-manifest-update

Change-Id: I0c9773aa8bc5876813a2e7d7fec697abcb2d9e94
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/246445
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-05 18:57:58 +00:00
Mike Frysinger
3538dd224d sync: merge project updates with status bar
The current sync output displays "Fetching project" and "Checking out
project" messages and progress bar updates independently leading to a
lot of spam.  Lets merge these periodic outputs with the status bar to
get a little bit tighter output in the normal case.  This doesn't solve
all our problems, but gets us closer.

Bug: https://crbug.com/gerrit/11293
Change-Id: Icd627830af4dd934a9355b7ace754b56dc96cfef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244934
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-11-12 23:33:51 +00:00
Vadim Bendebury
dff919493a sync: report list of failing git trees
When repo sync fails because some git trees are not in clean state and
as such can not be rebased automatically, it is a pain to figure out
which trees are the culprits.

With this patch the list of offending trees is printed when repo sync
reports checkout errors.

TEST=ran 'repo sync' and observed the proper list of directories show
     up after the final error message

Bug: https://crbug.com/gerrit/11293
Change-Id: Icdf1a03e9014ecb184f331f513cc9a2efc7d11ed
Signed-off-by: Vadim Bendebury <vbendeb@google.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244053
Reviewed-by: Mike Frysinger <vapier@google.com>
2019-11-12 21:08:54 +00:00
Mike Frysinger
3164d40e22 use open context managers in more places
Use open() as a context manager to simplify the close logic and make
the code easier to read & understand.  This is also more Pythonic.

Change-Id: I579d03cca86f99b2c6c6a1f557f6e5704e2515a7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244734
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-11-12 03:44:39 +00:00
Mike Frysinger
fb527e3f52 sync: create dedicated manifest project update func
Cut out some more standalone code from Execute to make this func a
bit more manageable.  The manifest project update is pretty simple
and standalone, but still takes up a good chunk of what's left.

Change-Id: Idc2442d9def495eccd0a49cda203c44aef16f129
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236614
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2019-09-19 01:52:08 +00:00
Mike Frysinger
01d6c3c0c5 sync: create dedicated smart sync func
The smart sync logic takes up about 45% of the overall Execute func
and is about 100 lines of code.  The only effect it has on the rest
of the code is to set the manifest_name variable.  Since this func
is already quite huge, split the smart sync logic out.

Change-Id: Id861849b0011ab47387d74e92c2ac15afcc938ba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234835
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2019-09-11 23:24:11 +00:00
Stefan Müller-Klieser
46702eddc7 sync: fix deprecated command line option -f
In commit d9e5cf0e ("sync: invert --force-broken with --fail-fast") the
force-broken option has been deprecated. Accidentally the option has
been changed from Boolean to Value. This breaks all users of repo with:

  main.py: error: -f option requires an argument

This is easy to avoid by keeping the type.

Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Change-Id: Ia8b589cf41ac756d10c61e17ec8d76ba8f7031f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/235043
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-30 16:29:01 +00:00
Mike Frysinger
ae6cb08ae5 split out cli validation from execution
A common pattern in our subcommands is to verify the arguments &
options before executing things.  For some subcommands, that check
stage is quite long which makes the execution function even bigger.
Lets split that logic out of the execute phase so it's easier to
manage these.

This is most noticeable in the sync subcommand whose Execute func
is quite large, and the option checking makes up ~15% of it.

The manifest command's Execute can be simplified significantly as
the optparse configuration always sets output_file to a string.

Change-Id: I7097847ff040e831345e63de6b467ee17609990e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234834
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-28 03:54:11 +00:00
Mike Frysinger
d9e5cf0ee7 sync: invert --force-broken with --fail-fast
People seem to not expect the sync process to halt immediately if an
error is encountered.  It's also basically guaranteed to leave their
tree in an incomplete state.  Lets invert the default behavior so we
attempt to sync (both fetch & checkout) all projects.  If an error is
hit, we still exit(1) and show it at the end.

If people want the sync to abort quickly, they can use the new option
--fail-fast.

Bug: https://crbug.com/gerrit/11293
Change-Id: I49dd6c4dc8fd5cce8aa905ee169ff3cbe230eb3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234812
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-27 01:20:44 +00:00
Mike Frysinger
a850ca2712 rebase/sync: use exit(1) for errors instead of exit(-1)
Callers don't actually see -1 (they'll usually see 255, but the exact
answer here is complicated).  Just switch to 1 as that's the standard
value tools use to indicate an error.

Change-Id: Ib712db1924bc3e5f7920bafd7bb5fb61f3bda44f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233553
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-08 05:07:31 +00:00