Commit Graph

1563 Commits

Author SHA1 Message Date
Mike Frysinger
c99322a6a9 sync: switch to multiprocessing.Event
We've switched most of this command over to multiprocessing and off
of _threading, so do the Event object too.  The APIs are the same
between the modules, so we shouldn't need to update anything else.

Change-Id: I52d31f1c6ef2bcbe7bbc1dd1add79a8d5d08784a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305147
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 20:00:41 +00:00
Mike Frysinger
14208f4c93 sync: fix logic error with linkfile errors
Make sure err_update_linkfiles is always initalized.

Bug: https://crbug.com/gerrit/11008
Change-Id: I7bdd91f82507608ef967daf0fa0f9c859454e19f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305146
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 20:00:20 +00:00
Mike Frysinger
2ee0a62db0 release-process: document the rate limiting in automatic updates
We check for updates only once per day, so clarify the docs.

Change-Id: Ib669ca6ebc67bc13204996fa40e1a3a82012295e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305145
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 19:37:48 +00:00
Mike Frysinger
0ae9503a86 sync: fix print error when handling server error
When converting this logic from print() to the output buffer, this
error codepath should have dropped the use of the file= redirect.

Bug: https://crbug.com/gerrit/14482
Change-Id: Ib484924a2031ba3295c1c1a5b9a2d816b9912279
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 12:48:42 -04:00
Mike Frysinger
c177f944d9 subcmds: force consistent help text format
We're inconsistent with help text as to whether it uses title case and
whether it ends in a period.  Add a test to enforce a standard, and use
the style that Python optparse & argparse use themselves (e.g. with the
--help option): always lowercase, and never trailing period.

Change-Id: Ic1defae23daeac0ac9116aaf487427f50b34050d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305144
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 16:40:53 +00:00
Mike Frysinger
aedd1e5ef0 sync: fix print error when handling server error
When converting this logic from print() to the output buffer, this
error codepath should have dropped the use of the file= redirect.

Bug: https://crbug.com/gerrit/14482
Change-Id: Ib484924a2031ba3295c1c1a5b9a2d816b9912279
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 16:40:28 +00:00
Raman Tenneti
5a41b0be01 superproject: skip updating commit ids if remote's fetchUrl don't match.
Tested the code with the following commands.

$ ./run_tests -v

+ Test with local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml  .repo/local_manifests/local.xml

  $ repo_dev sync -c -j8

+ Test without local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
  $ repo_dev sync -c -j8

Bug: [google internal] b/186395810
Change-Id: Id618113a91c12bcb90a30a3c23d3d6842bcb49e1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304942
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-05-04 15:32:23 +00:00
Mike Frysinger
d68ed63328 init/sync: add --no-tags to match --tags
While this provides a way to undo earlier command line options (e.g.
`repo sync --tags --no-tags`) which can be helpful for scripting &
automation, this more importantly allows the user to override the
manifest settings for syncing tags from a project.

Bug: https://crbug.com/gerrit/12401
Change-Id: Id4c36cd82e6ca7cb073b5d63a09f6c7ccdebba83
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304904
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 11:32:17 +00:00
Mike Frysinger
7356114d90 add --no-current-branch option to invert --current-branch
For most commands, this is more about providing a way to undo earlier
command line options (e.g. `repo info -c --no-current-branch`) which
can be helpful for scripting & automation.  But for the sync command,
this is helpful to undo the setting that exists in the manifest itself.

With this in place, tweak the sync current_branch_only logic to only
apply the manifest settings when the user hasn't specified a command
line option.

Bug: https://crbug.com/gerrit/12401
Change-Id: I21e2384624680cc740d1b5d1e49c50589d2fe6a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304903
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 11:31:48 +00:00
Mike Frysinger
b8e09ea1d6 harmonize --current-branch short option across subcommands
We're inconsistent with the short option for this flag:
* gitc-init: <none as -c is already used>
* info:      -b
* init:      -c
* overview:  -b
* sync:      -c
* upload:   --cbr

Since info & overview are not as heavily used as the others, switch
them from -b to -c.  We leave -b in as a hidden alias for now.

Similarly, switch upload from --cbr to just -c.  A lot of people
use --cbr, so we leave this as a hidden alias for now too.

Ideally gitc-init wouldn't use -c, but that ship has sailed, and
we're more likely to deprecate gitc entirely at this point.

This provides a consistent set of options across subcommands.

Bug: https://crbug.com/gerrit/12401
Change-Id: Iec249729223866fe1ea0ebabed12ca851cc38b35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304902
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-03 16:57:23 +00:00
Raman Tenneti
feb28914bd superproject: Don't update the commit ids of projects if remote is different.
1) Skip setting the revision id (commit id) for the projects whose
   remote doesn't match superproject's remote.
2) exp-superproject/superproject_override.xml includes local_manfiest's
   projects. When we load this XML, don't reload projects from local.xml
   (otherwise we will get duplicate projects errors).

Tested the code with the following commands.

$ ./run_tests -v

+ Test with local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml  .repo/local_manifests/local.xml

  $ repo_dev sync -c -j8

+ Test without local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
  $ repo_dev sync -c -j8

Bug: [google internal] b/186395810
Change-Id: I4e9d4ac2d94a9fc0cef0ccd787b6310758009e86
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304882
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-03 05:13:23 +00:00
Mike Frysinger
d1f3e149df upload: search local projects in parallel
Search for project branches to upload in parallel.  This can cut the
lookup time in half for large projects.  We still run the actual hooks
in serial once we have the list of projects to process, but we would
need to rethink things quite a bit before we could handle running them
in parallel too.

Change-Id: I8da0cbc5010566aa860e1a158f3dc07f0709dcff
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304842
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-02 00:06:32 +00:00
Mike Frysinger
29626b4f46 project: fix m/ generation when switching manifest branches
We were updating the per-checkout m/ pseudo ref when syncing, but we
only created the common m/ redirect when initializing a project for
the first time.  This is fine unless the user switches the manifest
branch in an existing project, then we never create that redirect.

Bug: https://crbug.com/gerrit/14468
Change-Id: I5325e7e602dcb4ce150bef258901ba5e9fdea461
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304822
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-02 00:05:55 +00:00
Mike Frysinger
3b038cecc4 upload: include the project in error messages
When running upload across multiple projects, include the project in
any error messages that come up.  This lets users figure out where
the problem might be.

Change-Id: I09470c9a1b512baf910d6d97b747816d1a6f3a87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304783
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-02 00:05:49 +00:00
jiajia tang
a590e640a6 Update copyfile and linkfile if manifest updated
Currently, copyfiles and linkfiles which marked by
"<copyfile/>" and "<linkfile/>" in manifest will
be created by first exec 'repo sync'.
But if some "<copyfile/>" or "<linkfile/>" are removed
in manifest, then 'repo sync', these removed item
dest can not be removed in the sourcecode workspace.

This patch is intent to fix this issue, by save a
'copy-link-files.json' in .repo and then compared with
new dest path when next sync. If any "<copyfile/>" or
"<linkfile/>" were removed, the dest path will be
removed in sourcecode at the same time.

Bug: https://crbug.com/gerrit/11008
Change-Id: I6b7b41e94df0f9e6e52801ec755951a4c572d05d
Signed-off-by: jiajia tang <tangjiajia@xiaomi.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304202
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-01 13:26:08 +00:00
Mike Frysinger
f69c7ee318 manifest_xml: ban use of newlines in paths
There should be no valid use of these anywhere, so just ban them
to make things easier for people.

Bug: https://crbug.com/gerrit/14156
Bug: https://crbug.com/gerrit/14200
Change-Id: I8d2cf988c510c98194c43a329a2b9bf313a3f0a8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304662
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-30 05:54:11 +00:00
Xin Li
aabf79d3f0 sync: Fix a corner case when both superproject and depth used.
When depth is used, we would fetch only SHA1 when superproject is
used, as the result, only the manifest branch is being recorded,
and commands like repo start would fail.

Fix this by saving the upstream branch value in the overlay
manifest and add the upstream branch to fetch list.

Bug: [google internal] b/185951360
Change-Id: Ib36f56067723f2572ed817785b31cc928ddfec0a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304562
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Xin Li <delphij@google.com>
2021-04-29 19:05:47 +00:00
Mike Frysinger
a1cd770d56 help/version: sprinkle bug report URL around
Make it a bit easier for people to locate bug reporting info.

Change-Id: If9c8939c84ebd52eb96b353c1797afa25868bb85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303943
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
2021-04-26 21:43:43 +00:00
Raman Tenneti
cd89ec147a sync: Fix exception in an exsiting clone (without partial-clone).
Default the partial_clone_exclude argument to an empty set.

Fixes the following report by Emil Medve.

With this change (up to v2.14.1), on an existing "normal" clone (without partial-clone options) I'm seeing this traceback during `repo selfupdate`:

Traceback (most recent call last):

  File ".../.repo/repo/main.py", line 630, in <module>
    _Main(sys.argv[1:])
  File ".../.repo/repo/main.py", line 604, in _Main
    result = run()
  File ".../.repo/repo/main.py", line 597, in <lambda>
    run = lambda: repo._Run(name, gopts, argv) or 0
  File ".../.repo/repo/main.py", line 261, in _Run
    result = cmd.Execute(copts, cargs)
  File ".../.repo/repo/subcmds/selfupdate.py", line 54, in Execute
    if not rp.Sync_NetworkHalf():
  File ".../.repo/repo/project.py", line 1091, in Sync_NetworkHalf
    if self.name in partial_clone_exclude:
TypeError: argument of type 'NoneType' is not iterable

$ ./run_tests -v

Change-Id: I71e744e4ef2a37b13aa9ba42eba3935e78c4e40a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304082
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-04-22 18:00:32 +00:00
Mike Frysinger
d41eed0b36 sync: fix missing import for -q
Some refactors during review dropped this import when it was reworked,
but it's still needed when using the --quiet setting.

Change-Id: I6d9302ef5a056e52415ea63f35bad592b9dfa75d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303942
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-21 15:37:16 +00:00
Mike Frysinger
d2b086bea9 init: restore default --manifest-name
The merge of the repo & init parser missed this default.

When running `repo init ...` in an existing checkout but w/out the -m
option, then repo would error out complaining that -m is required when
it didn't do this before.

Change-Id: I58035d48cc413b5d373702b9dc3b9ecd3fd1e900
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303945
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2021-04-21 04:45:59 +00:00
Mike Frysinger
6823bc269d sync: cleanup sleep+retry logic a bit
Make sure we print a message whenever we retry so it's clear to the
user why repo is pausing for a long time, and why repo might have
passed even though it displayed some errors earlier.

Also unify the sleep logic so we don't have two independent methods.
This makes it easier to reason about.

Also don't sleep if we're in the last iteration of the for loop.  It
doesn't make sense to and needlessly slows things down when there are
real errors.

Bug: https://crbug.com/gerrit/12494
Change-Id: Ifceace5b2dde75c2dac39ea5388527dd37376336
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303402
Reviewed-by: Sam Saccone 🐐 <samccone@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-19 16:42:06 +00:00
Raman Tenneti
ad8aa69772 sync: only print error.GitError, don't raise that exception.
In _FetchOne & _CheckOne, only print error.GitError exception,
but other exceptions are still thrown

Fixes the GitError exceptions from /usr/lib/python3.8/multiprocessing/pool.py
exiting the repo sync.

Tested the code with the following commands and verified repo sync
continues after fetch error because of an invalid SHA1.

$ ./run_tests -v

$ python3 ~/work/repo/git-repo/repo sync -m manifest_P21623846.xml -j32
...
error.GitError: Cannot fetch platform/vendor/google_devices/redbull/proprietary update-ref: fatal: d5a99e518f09d6abb0c0dfa899594e1ea6232459^0: not a valid SHA1
....

An error like the following when jobs=1
  error.GitError: Cannot checkout platform/vendor/qcom/sdm845/proprietary/qcrilOemHook: Cannot initialize work tree for platform/vendor/qcom/sdm845/proprietary/qcrilOemHook

Bug: https://crbug.com/gerrit/14392
Change-Id: I8922ad6c07c733125419f5698b0f7e32d70c7905
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303544
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-04-15 22:43:07 +00:00
Mike Frysinger
b5d075d04f command: add a helper for the parallel execution boilerplate
Now that we have a bunch of subcommands doing parallel execution, a
common pattern arises that we can factor out for most of them.  We
leave forall alone as it's a bit too complicated atm to cut over.

Change-Id: I3617a4f7c66142bcd1ab030cb4cca698a65010ac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301942
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-04-15 05:10:16 +00:00
Peter Kjellerstedt
b8bf291ddb tests: Make ReviewableBranchTests.test_smoke work with git < 2.28.0
Bug: https://crbug.com/gerrit/14380
Change-Id: Id015bd98b008e1530ada2c7e4332c67e8e208e25
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303325
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-04-14 16:22:52 +00:00
Mike Frysinger
233badcdd1 list: fix help grammar
Change-Id: Ia642e38532173d59868e0101cc098eab706d715e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303302
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-14 15:25:53 +00:00
Mike Frysinger
0888a083ec help: switch from formatter module to textwrap
Since Python has deprecated the formatter module, switch to the textwrap
module instead for reflowing text.  We weren't really using any other
feature anyways.

Verified by diffing the output before & after the change and making sure
it was the same.

Then made a few tweaks to tighten up the output.

Change-Id: I0be1bc2a6661a311b1a4693c80d0f8366320ba55
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303282
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-14 01:00:51 +00:00
Mike Frysinger
e2effe11a5 list: add option to show non-checkedout projects too
Currently, list only shows projects that exist in the checkout, and
doesn't offer any way to list all projects in the manifest (based on
the current settings, or on the options passed to list).  This seems
to be the opposite of what (at least some) users expect, so let's
add an option to show all of them regardless of checkout state.

Change-Id: I94bbdc5bd0ff2a411704fa215e7fc2b60fa3360e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301263
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-13 22:42:32 +00:00
Mike Frysinger
151701e85f progress: hide progress bar when --quiet
We want progress bars in the default output mode, but not when the
user specifies --quiet.  Add a setting to the Progress bar class so
it takes care of not displaying anything itself rather than having
to update every subcommand to conditionally setup & call the object.

Change-Id: I1134993bffc5437bc22e26be11a512125f10597f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303225
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-13 22:25:26 +00:00
Mike Frysinger
9180a07b8f command: make --verbose/--quiet available to all subcommands
Add new CommonOptions entry points to move the existing --jobs to,
and relocate all --verbose/--quiet options to that.  This provides
both a consistent interface for users as well as for code.

Change-Id: Ifaf83b88872421f4749b073c472b4a67ca6c0437
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303224
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-13 22:25:17 +00:00
Raman Tenneti
f32f243ff8 init: Added --partial-clone-exclude option.
partial-clone-exclude option excludes projects during
partial clone. This is a comma-delimited project names
(from manifest.xml). This option is persisted and it
is used by the sync command.

A project that has been unparital'ed will remain unpartial if
that project's name is specified in the --partial-clone-exclude
option. The project name should match exactly.

Added
$ ./run_tests -v

Bug: [google internal] b/175712967
"I can't "unpartial" my androidx-main checkout"

$ rm -rf androidx-main/
$ mkdir androidx-main/
$ cd androidx-main/
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml
$ repo_dev sync -c -j8

+ Verify a project is partial
$ cd frameworks/support/
$ git config -l | grep  'partial'

+ Unpartial a project.
$ /google/bin/releases/android/git_repack/git_unpartial

+ Verify project is unpartial
$ git config -l | grep  'partial'
$ cd ../..

+ Exclude the project from being unparial'ed after init and sync.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml

+ Verify project is unpartial
$ cd frameworks/support/
$ git config -l | grep  'partial'
$ cd ../..
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep  'partial'
$ cd ../..

+ Remove the project from exclude list and verify that project is partially cloned.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep  'partial'

Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-04-13 15:47:10 +00:00
Mike Frysinger
49de8ef584 sync: add separate --jobs options for different steps
The number of jobs one wants to run against the network tends to
factor differently from the number of jobs one wants to run when
checking out local projects.  The former is constrained by your
internet connection & server limits while the later is constrained
by your local computer's CPU & storage I/O.  People with beefier
computers probably want to keep the network/server jobs bounded a
bit lower than the local/checkout jobs.

Change-Id: Ia27ab682c62c09d244a8a1427b1c65acf0116c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302804
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-09 15:58:56 +00:00
Mike Frysinger
a1051d8baa init: organize command line options a bit
We've grown a lot of options in here and it's hard to make sense of
them.  Add more groups to try and make it easier to pick things out.

Change-Id: I6b9dc0e83f96137f974baf82d3fb86992b857bd2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302803
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-09 15:58:45 +00:00
Mike Frysinger
65af2602b5 sync: add progress bar to garbage collection phase
This can take a few seconds, if not a lot more, so add a progress bar
so users understand what's going on.

Change-Id: I5b4b54c1bbb9ec18728f979521310f7087afaa5c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-09 15:58:21 +00:00
Mike Frysinger
347f9ed393 sync: rework selfupdate logic
The current logic has a downside in that it doesn't sync to the latest
signed version available if the latest commit itself is unsigned.  This
can come up when using the "main" branch as it is sometimes signed, but
often not as it's holding the latest merged commits.  When people use
the main branch, it's to get early testing on versions tagged but not
yet released, and we don't want them to get stuck indefinitely on that
old version of repo.

For example, this series of events:
* "stable" is at v2.12.
* "main" is tagged with v2.13.
* early testers use --repo-rev main to get v2.13.
* new commits are merged to "main".
* "main" is tagged with v2.14.
* new commits are merged to "main".
* devs who had synced in the past to test v2.13 are stuck on v2.13.
  repo sees "main" is unsigned and so doesn't try to upgrade at all.

The only way to get unwedged is to re-run `repo init --repo-rev main`,
or to manually sync once with repo verification disabled, or for us to
leave "main" signed for a while and hope devs will sync in that window.

The new logic is that whenever changes are available, we switch to the
latest signed tag.  We also replace some of the duplicated verification
code in the sync command with the newer wrapper logic.  This handles a
couple of important scenarios inaddition to above:
* rollback (e.g. v2.13.8 -> v2.13.7)
* do not trash uncommitted changes (in case of ad-hoc testing)
* switch tag histories (e.g. v2.13.8 -> v2.13.8-cr1)

Change-Id: I5b45ba1dd26a7c582700ee3711f303dc7538579b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300122
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-09 03:16:45 +00:00
Mike Frysinger
9a734a3975 init: merge subcmd & wrapper parsers
These are manually kept in sync which is a pain.  Have the init
subcmd reuse the wrapper code directly.

Change-Id: Ica73211422c64377bacc9bb3b1d1a8d9d5f7f4ca
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302762
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-09 01:04:32 +00:00
Raman Tenneti
6a2f4fb390 repo init: Added --no-partial-clone and made it persist. Bumped version to 2.14.
Saved the repo.partialclone when --no-partial-clone option is passed
to init, so repo sync will honor the no-partial-clone option.

$ ./run_tests -v

Bug: [google internal] b/175712967

$ mkdir androidx-main && cd androidx-main
$ repo init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M
$ repo sync -c -j32
$ cd frameworks/support/ && /google/bin/releases/android/git_repack/git_unpartial
$ git config -l | grep  'partialclonefilter=blob'

Observe partialclone is not enabled.

$ cd ../..
$ repo init -u https://android.googlesource.com/platform/manifest -b androidx-main
$ repo sync -c -j32
$ cd frameworks/support/ && git config -l | grep  'partialclonefilter=blob'

Observe partialclone is enabled.

$ /google/bin/releases/android/git_repack/git_unpartial

Observe partialclone is not enabled.

$ cd ../..
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --no-partial-clone
$ repo sync -c -j32
$ cd frameworks/support/ && git config -l | grep  'partialclonefilter=blob'

Observe partialclone is not enabled.

Change-Id: I4400ad7803b106319856bcd0fffe00bafcdf014e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302122
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-04-05 05:53:19 +00:00
Mike Frysinger
beea5de842 tox: enable python 3.5 & 3.9 testing
We still support Python 3.5, so make sure it keeps working.

Change-Id: I150158a656b26de6d733316a68a2cbb8b5b99716
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299625
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:57:05 +00:00
Mike Frysinger
bfbcfd9045 run_tests: fix exit code handling
We need to pass back an int, not a CompletedProcess object.  Switch to
check=False so we don't throw an exception on failure -- we're already
showing pytest's stderr, and will return the non-zero status.

Change-Id: Ib0d3862a09a3963f25025f39a8e34419cf2a54df
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299624
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:56:57 +00:00
Mike Frysinger
74317d3b01 setup: bump major version
We don't keep this up-to-date in general, but might as well keep
the major version in sync.

Change-Id: I20908005b3b393d384da0ef9342d7c9d094550cb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299622
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:56:47 +00:00
Mike Frysinger
b2fa30a2b8 sync: switch network fetch to multiprocessing
This avoids GIL limitations with using threads for parallel processing.

This reworks the fetch logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.

We have to tweak the chunking logic a little here because multiprocessing
favors batching over returning immediate results when using a larger value
for chunksize.  When a single job can be quite slow, this tradeoff is not
good UX.

Bug: https://crbug.com/gerrit/12389
Change-Id: I0f0512d15ad7332d1eb28aff52c29d378acc9e1d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298642
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:52:57 +00:00
Mike Frysinger
d246d1fee7 grep: add --jobs support
Use multiprocessing to run in parallel.  When operating on multiple
projects, this can greatly speed things up.  Across 1000 repos, it
goes from ~40sec to ~16sec with the default -j8.

The output processing does not appear to be a significant bottle
neck -- it accounts for <1sec out of the ~16sec runtime.  Thus we
leave it in the main thread to simplify the code.

Change-Id: I750b72c7711b0c5d26e65d480738fbaac3a69971
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297984
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:43:19 +00:00
Mike Frysinger
bec4fe8aa3 prune: add --jobs support
Use multiprocessing to run in parallel.  When operating on multiple
projects, this can greatly speed things up.  Across 1000 repos, it
goes from ~10sec to ~4sec with the default -j8.

This only does a simple conversion over to get an easy speedup.  It
is currently written to collect all results before displaying them.
If we refactored this module more, we could have it display results
as they came in.

Change-Id: I5caf4ca51df0b7f078f0db104ae5232268482c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298643
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-31 16:28:24 +00:00
Mike Frysinger
ddab0604ee forall: handle missing project refs better
If the project exists, but the ref the manifest wants doesn't exist,
don't throw an error (and abort the process in general).  This can
come up with a partially synced tree: the manifest is up-to-date,
but not all the projects have yet been synced.

Bug: https://crbug.com/gerrit/14289
Change-Id: Iba97413c476544223ffe518198c900c2193a00ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301262
Reviewed-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-25 23:08:51 +00:00
Raman Tenneti
2ae44d7029 sync: imply -c if --use-superproject option is used.
Tested the code with the following commands.

$ ./run_tests -v

Bug: [google internal] b/183232698
Bug: https://crbug.com/gerrit/13707

$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync --use-superproject
$ repo_dev sync
  real	0m8.046s
  user	0m2.866s
  sys	0m2.457s

Second time repo sync took only 8 seconds and verified by printing that
urrent_branch_only is True in project.py's Sync_NetworkHalf function.

Change-Id: Ic48efb23ea427dfa36e12a5c49973d6ae776d818
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301182
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-03-24 15:17:19 +00:00
Mike Frysinger
d1e4fa7015 start: add a --HEAD alias
This makes it easy to use --H as a shortcut, and kind of matches the
use of storing HEAD as the revision.

Change-Id: I590bf488518f313e7a593853140316df98262d7e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301163
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-24 00:32:04 +00:00
Mike Frysinger
323b113f55 forall/list: delete spurious "
Change-Id: I6995d48be1d8fc5d93f4b9fa617fad70b5b3429f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300902
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-19 21:13:49 +00:00
Raman Tenneti
8367096d02 superproject: Added --depth=1 argument to git fetch command.
Tested the code with the following commands.

$ ./run_tests -v

Without --depth

$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject
real    6m48.086s
user    3m27.281s
sys     1m1.386s

With --depth=1
$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject

real    2m49.637s
user    2m51.458s
sys     0m39.108s

From dwillemsen@:
 "For me it's the difference between 9m28s using the current code and
  16s using --depth=1 while fetching the superproject."

Bug: [google internal] b/180451672
Bug: https://crbug.com/gerrit/13707
Change-Id: I1c3e4aef4414c4e9dd259fb6e4619da0421896b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300922
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-19 21:10:39 +00:00
Mike Frysinger
d34af28ac2 forall: allow interactive commands with -j1
Historically forall has been interactive since it ran in serial.
Recent rework in here dropped that to enable parallel processing.
Restore support for interactive commands when running -j1 or with
an explicit --interactive option.

Bug: https://crbug.com/gerrit/14256
Change-Id: I502007186f771914cfd7830846a4e1938b5e1f38
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300722
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-18 22:13:01 +00:00
Raman Tenneti
a5b40a2845 repo: Add a new "command" event type to git trace2 logging in repo.
Add a new "event": "command", which is emitted at when all command
arguments have been processed.

Additional fields:

"name": Name of the primary command (ex: repo, git)

"subcommands"': List of the sub-commands once command-line arguments
                are processed

Examples:

Command: repo --version

Event: {"event": "command", <common fields>,
        "name": "repo",
	"subcommands": ["version"]
	}

Bug: [google internal] b/178507266
Testing:
- Unit tests
- Verified repo git trace2 logs had expected data

Change-Id: I825bd0ecedee45135382461a4ba10f987f09aef3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300343
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-18 14:58:24 +00:00