Commit Graph

1826 Commits

Author SHA1 Message Date
Mike Frysinger
beea5de842 tox: enable python 3.5 & 3.9 testing
We still support Python 3.5, so make sure it keeps working.

Change-Id: I150158a656b26de6d733316a68a2cbb8b5b99716
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299625
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:57:05 +00:00
Mike Frysinger
bfbcfd9045 run_tests: fix exit code handling
We need to pass back an int, not a CompletedProcess object.  Switch to
check=False so we don't throw an exception on failure -- we're already
showing pytest's stderr, and will return the non-zero status.

Change-Id: Ib0d3862a09a3963f25025f39a8e34419cf2a54df
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299624
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:56:57 +00:00
Mike Frysinger
74317d3b01 setup: bump major version
We don't keep this up-to-date in general, but might as well keep
the major version in sync.

Change-Id: I20908005b3b393d384da0ef9342d7c9d094550cb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299622
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:56:47 +00:00
Mike Frysinger
b2fa30a2b8 sync: switch network fetch to multiprocessing
This avoids GIL limitations with using threads for parallel processing.

This reworks the fetch logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.

We have to tweak the chunking logic a little here because multiprocessing
favors batching over returning immediate results when using a larger value
for chunksize.  When a single job can be quite slow, this tradeoff is not
good UX.

Bug: https://crbug.com/gerrit/12389
Change-Id: I0f0512d15ad7332d1eb28aff52c29d378acc9e1d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298642
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:52:57 +00:00
Mike Frysinger
d246d1fee7 grep: add --jobs support
Use multiprocessing to run in parallel.  When operating on multiple
projects, this can greatly speed things up.  Across 1000 repos, it
goes from ~40sec to ~16sec with the default -j8.

The output processing does not appear to be a significant bottle
neck -- it accounts for <1sec out of the ~16sec runtime.  Thus we
leave it in the main thread to simplify the code.

Change-Id: I750b72c7711b0c5d26e65d480738fbaac3a69971
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297984
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:43:19 +00:00
Mike Frysinger
bec4fe8aa3 prune: add --jobs support
Use multiprocessing to run in parallel.  When operating on multiple
projects, this can greatly speed things up.  Across 1000 repos, it
goes from ~10sec to ~4sec with the default -j8.

This only does a simple conversion over to get an easy speedup.  It
is currently written to collect all results before displaying them.
If we refactored this module more, we could have it display results
as they came in.

Change-Id: I5caf4ca51df0b7f078f0db104ae5232268482c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298643
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-31 16:28:24 +00:00
Mike Frysinger
ddab0604ee forall: handle missing project refs better
If the project exists, but the ref the manifest wants doesn't exist,
don't throw an error (and abort the process in general).  This can
come up with a partially synced tree: the manifest is up-to-date,
but not all the projects have yet been synced.

Bug: https://crbug.com/gerrit/14289
Change-Id: Iba97413c476544223ffe518198c900c2193a00ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301262
Reviewed-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-25 23:08:51 +00:00
Raman Tenneti
2ae44d7029 sync: imply -c if --use-superproject option is used.
Tested the code with the following commands.

$ ./run_tests -v

Bug: [google internal] b/183232698
Bug: https://crbug.com/gerrit/13707

$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync --use-superproject
$ repo_dev sync
  real	0m8.046s
  user	0m2.866s
  sys	0m2.457s

Second time repo sync took only 8 seconds and verified by printing that
urrent_branch_only is True in project.py's Sync_NetworkHalf function.

Change-Id: Ic48efb23ea427dfa36e12a5c49973d6ae776d818
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301182
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-03-24 15:17:19 +00:00
Mike Frysinger
d1e4fa7015 start: add a --HEAD alias
This makes it easy to use --H as a shortcut, and kind of matches the
use of storing HEAD as the revision.

Change-Id: I590bf488518f313e7a593853140316df98262d7e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301163
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-24 00:32:04 +00:00
Mike Frysinger
323b113f55 forall/list: delete spurious "
Change-Id: I6995d48be1d8fc5d93f4b9fa617fad70b5b3429f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300902
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-19 21:13:49 +00:00
Raman Tenneti
8367096d02 superproject: Added --depth=1 argument to git fetch command.
Tested the code with the following commands.

$ ./run_tests -v

Without --depth

$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject
real    6m48.086s
user    3m27.281s
sys     1m1.386s

With --depth=1
$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject

real    2m49.637s
user    2m51.458s
sys     0m39.108s

From dwillemsen@:
 "For me it's the difference between 9m28s using the current code and
  16s using --depth=1 while fetching the superproject."

Bug: [google internal] b/180451672
Bug: https://crbug.com/gerrit/13707
Change-Id: I1c3e4aef4414c4e9dd259fb6e4619da0421896b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300922
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-19 21:10:39 +00:00
Mike Frysinger
d34af28ac2 forall: allow interactive commands with -j1
Historically forall has been interactive since it ran in serial.
Recent rework in here dropped that to enable parallel processing.
Restore support for interactive commands when running -j1 or with
an explicit --interactive option.

Bug: https://crbug.com/gerrit/14256
Change-Id: I502007186f771914cfd7830846a4e1938b5e1f38
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300722
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-18 22:13:01 +00:00
Raman Tenneti
a5b40a2845 repo: Add a new "command" event type to git trace2 logging in repo.
Add a new "event": "command", which is emitted at when all command
arguments have been processed.

Additional fields:

"name": Name of the primary command (ex: repo, git)

"subcommands"': List of the sub-commands once command-line arguments
                are processed

Examples:

Command: repo --version

Event: {"event": "command", <common fields>,
        "name": "repo",
	"subcommands": ["version"]
	}

Bug: [google internal] b/178507266
Testing:
- Unit tests
- Verified repo git trace2 logs had expected data

Change-Id: I825bd0ecedee45135382461a4ba10f987f09aef3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300343
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-18 14:58:24 +00:00
Mike Frysinger
511a0e54f5 sync: fix reporting of failed local checkouts
The refactor to multiprocessing broke status reporting slightly when
checking out projects.  Make sure we mark the step as failed if any
of the projects failed, not just when --fail-fast is set.

Change-Id: I0efb56ce83b068b2c334046df3fef23d797599c9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299882
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-15 16:54:23 +00:00
Mike Frysinger
8da7b6fc65 bash-completion: initial import based on CrOS version
We've had a limited version of this in CrOS for a long time.  There's
nothing CrOS specific about it, so lets move it to the repo project so
everyone can utilize it.

Change-Id: I04cd94610c1100f3afcd2baf8c8e7ab13e589490
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299202
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-15 16:54:21 +00:00
Mike Frysinger
0458faa502 manifest: allow toplevel project checkouts
Re-allow checking out projects to the top of the repo client checkout.
We add checks to prevent checking out files under .repo/ as that path
is only managed by us, and projects cannot inject content or settings
into it.

Bug: https://crbug.com/gerrit/14156
Bug: https://crbug.com/gerrit/14200
Change-Id: Id6bf9e882f5be748442b2c35bbeaee3549410b25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299623
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-12 16:31:14 +00:00
Mike Frysinger
68d5d4dfe5 document the new manifest restrictions on name & path settings
Bug: https://crbug.com/gerrit/14156
Change-Id: I473edab1173e6a266d0754c29d5dc7ff761f1359
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299403
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-12 16:30:37 +00:00
Mike Frysinger
a3794e9c6f prune: minor optimization & robustification
If the current project doesn't have any local branches, then there's
nothing to prune, so return right away.  This avoids running a few
git commands when we aren't actually going to use the results, and
it avoids checking repository validity.  Since we aren't going to do
anything in here, no need to check it.

Change-Id: Ie9d5c75a954e42807477299f3e5a63a92fac138b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299742
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-12 05:28:06 +00:00
Raman Tenneti
080877e413 superproject: pass groups to ToXml method.
Added the following methods to XmlManifest class.
+ GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower()
+ GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func.

+ Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr.
+ Used the above methods to get groups in command.py::GetProjects
  and part of init.py.

TODO: clean up these funcs to take structured group data more instead
      of passing strings around everywhere that need parsing.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL
and verified prebuilts/fullsdk-linux directory has all the folders.

Tested repo init and repo sync with --use-superproject and without
--use-superproject argument.

$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main  --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject

$ repo_dev sync -c -j32

Bug: [google internal] b/181804931
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-11 01:24:52 +00:00
Mike Frysinger
9888accb0c project: fix diff printing with embedded %
The recent commit 84230009ee ("project:
make diff tools synchronous") broke repo diff if it includes % formats.
Add an explicit format string to fix.

Bug: https://crbug.com/gerrit/14208
Change-Id: Ie255a43c5b767488616b2b3dd15abc18f93bfab2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299402
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-09 17:00:02 +00:00
Mike Frysinger
5a4c8fde17 init: expose --worktree option
There's a few rough edges here still, but no known corruption ones,
so open it up a bit for people to experiment with.

Bug: https://crbug.com/gerrit/11486
Change-Id: I81e0122ab6d3e032c546c8239dd4f03740676e80
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299242
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-09 16:59:59 +00:00
Ian Kasprzak
835a34bdb9 Log repo.* config variables in git trace2 logger.
Bug: [google internal] b/181758736
Testing:
- Unit tests
- Verified repo git trace2 logs had expected data

Change-Id: I9af8a574377bd91115f085808c1271e9dee16a36
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299182
Tested-by: Ian Kasprzak <iankaz@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
2021-03-08 17:32:09 +00:00
Raman Tenneti
ef99ec07b4 superproject: Display status messages during repo init/sync.
Superproject objects accept the optional argument “quiet”.
The following progress messages are displayed if quiet is false.

Displayed the following message whenever we find we have to make a new
folder (aka new remote), because if you started with repo init android
and later do googleplex-android that is when it will be slow.

"<location>: Performing initial setup for superproject; this might take
several minutes.".

After fetch completion, added the following notification:
"<location>: Initial setup for superproject completed."

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL.

$ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev  --partial-clone --clone-filter=blob:limit=10M --repo-rev=main  --use-superproject

Bug: [google internal] b/181178282
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-04 20:07:52 +00:00
Mike Frysinger
934cb0a849 tests: fix duplicate method from copy & paste error
Change-Id: Ib748c61b1e65aee6dff8b97a9753d14c470a827f
Reported-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299002
Reviewed-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-04 16:17:11 +00:00
Mike Frysinger
3c0931285c project: fix variable typo
Bug: https://crbug.com/gerrit/11293
Reported-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: I37bac58aa1dc9ecc10e29253d14ff9e6fb42427c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298942
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-03 16:45:21 +00:00
Mike Frysinger
5413397204 manifest: relax include name rules for user-specified path
Allow the user to specify relative or absolute or any other funky
path that they want when using `repo init` or `repo sync`.  Our
goal is to restrict the paths in the remote manifest git repo we
cloned from the network, not protect the user from themselves.

Bug: https://crbug.com/gerrit/14156
Change-Id: I1ccfb2a6bd1dce2bd765e261bef0bbf0f8a9beb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298823
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-02 03:18:57 +00:00
Mike Frysinger
13cb7f799d forall: greatly speed up processing overhead
With the recent commit 0501b29e7a
("status: Use multiprocessing for `repo status -j<num>` instead of
threading"), the limitation with project serialization no longer
applies.  It turns out that ad-hoc logic is expensive.  In the CrOS
checkout (~1000 projects w/8 jobs by default), it adds about ~7sec
overhead to all invocations.  With a fast nop run:
	time repo forall -j8 -c true
This goes from ~11sec to ~4sec -- more than 50% speedup.

Change-Id: Ie6bcccd21eef20440692751b7ebd36c890d5bbcc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298724
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-01 15:58:06 +00:00
Mike Frysinger
819c73954f forall: simplify arg passing to worker children
The ProjectArgs function can be inlined which simplifies it quite a
bit.  We shouldn't need the custom exception handling here either.
This also makes the next commit easier to review.

Change-Id: If3be04f58c302c36a0f20b99de0f67e78beac141
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298723
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-01 15:58:06 +00:00
Mike Frysinger
179a242caa forall: move nested func out to the class
This is in preparation for simplifying the jobs support.  The nested
function is referenced in the options object which can't be pickled,
so pull it out into a static method instead.

Change-Id: I01d3c4eaabcb8b8775ddf22312a6e142c84cb77d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298722
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-01 15:57:32 +00:00
Mike Frysinger
31fabeed54 download: handle shared projects a bit better
If a manifest checksout a project multiple times, repo download isn't
able to accurately pick the right project.  We were just picking the
first result which could be a bit random for the user.  If we hit that
situation, check if the cwd is one of the projects, and if it isn't,
we emit an error and tell the user it's an ambiguous request.

Bug: https://crbug.com/gerrit/13070
Change-Id: Id1059b81330229126b48c7312569b37504808383
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298702
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-01 15:57:17 +00:00
Mike Frysinger
76844ba292 project: skip clone bundles when we've already initialized the object dir
The clone bundle logic assumes there is a one-to-one mapping between the
projects/ and project-objects/ trees.  When using shared projects (where
we checkout different branches from the same project), this would lead us
to fetching the same clone bundle multiple times.  Automatically skip the
clone bundle logic if the project-objects/ dir already exists.

Bug: https://crbug.com/gerrit/10993
Change-Id: I82c6fa1faf8605fd56c104fcea2a43dd4eecbce4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-01 15:57:12 +00:00
Mike Frysinger
6d1faa1db3 git_refs: fix crash with binary . files in .git/refs/
On macOS, the Finder app likes to poop .DS_Store files in every path
that the user browses.  If the user pokes around the .git/ tree, it
could generate a .DS_Store file in there too.  When repo goes to read
all the local refs, it tries to decode this binary file as UTF-8 and
subsequently crashes.

Since paths that begin with . are not valid refs, ignore them like we
already do with paths that end in .lock.  Also bump the check up to
ignore dirs that match since that follows the git rules: they apply
to any component in its path, not just the final path (name).

We don't implement the full valid ref algorithm that git employs as
it's a bit complicated, and we only really need to focus on what will
practically show up locally.

Bug: https://crbug.com/gerrit/14162
Change-Id: I6519f990e33cc58a72fcb00c0f983ad3285ace3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298662
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
2021-02-28 16:07:24 +00:00
Mike Frysinger
4510be51c1 git_command: pass GIT_DIR on Windows with /
When using Git under Windows, it seems that Git doesn't always parse
GIT_DIR correctly when it uses the Windows \ form, but does when it
uses / only.

For example, when using worktrees:
$ GIT_DIR='C:\Users\vapier\Desktop\repo\breakpad\tools\test\.git' git worktree list
fatal: not a git repository: ..\..\.repo\worktrees\linux-syscall-support.git\worktrees\test
$ GIT_DIR='C:/Users/vapier/Desktop/repo/breakpad/tools/test/.git' git worktree list
C:/Users/vapier/Desktop/repo/breakpad/.repo/worktrees/linux-syscall-support.git  fd00dbbd0c06 (detached HEAD)
..\..\..\..\..\src\src\third_party\lss\.git                                      fd00dbbd0c06 (detached HEAD)
..\..\..\..\..\tools\test\.git                                                   fd00dbbd0c06 (detached HEAD)

Change-Id: I666c03ae845ecb55d7f9800731ea6987d3e7f401
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298622
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-28 16:07:20 +00:00
Mike Frysinger
a29424ea6d manifest: validate project name & path and include name attributes
These attribute values are used to construct local filesystem paths,
so apply the existing filesystem checks to them.

Bug: https://crbug.com/gerrit/14156
Change-Id: Ibcceecd60fa74f0eb97cd9ed1a9792e139534ed4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298443
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-28 16:07:12 +00:00
Mike Frysinger
a00c5f40e7 manifest: refactor the filesystem checking logic for more reuse
This function is currently written with copyfile & linkfile in mind.
Generalize the logic & function arguments slightly so we can reuse
in more places that make sense.

This changes the validation logic slightly too in that we no longer
allow "." for the dest attribute with copyfile & linkfile, nor for
the src attribute with copyfile.  We already rejected those later on
when checking against the active filesystem, but now we reject them
a little sooner when parsing.

The empty path check isn't a new requirement exactly -- repo used to
crash on it, so it was effectively blocked, but now we diagnosis it.

Bug: https://crbug.com/gerrit/14156
Change-Id: I0fdb42a3da60ed149ff1997c5dd4b85da70eec3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-28 16:07:12 +00:00
Mike Frysinger
6093d99d13 checkout: add --jobs support
Use multiprocessing to run in parallel.  When operating on multiple
projects, this can speed things up.  Across 1000 repos, it goes from
~9sec to ~5sec with the default -j8.

Change-Id: Ida6dd565db78ff7bac0ecb25d2805e8a1bf78048
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297982
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-27 19:56:24 +00:00
Mike Frysinger
ebf04a4404 sync: switch local checkout to multiprocessing
This avoids GIL limitations with using threads for parallel processing.
In a CrOS checkout with ~1000 repos, the nop case goes from ~6 sec down
to ~4 sec with -j8.  Not a big deal, but shows that this actually works
to speed things up unlike the threading model.

This reworks the checkout logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.

Bug: https://crbug.com/gerrit/12389
Change-Id: I143e5e3f7158e83ea67e2d14e5552153a874248a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298063
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-27 19:55:14 +00:00
Mike Frysinger
8dbc07aced abandon/start: add --jobs support
Use multiprocessing to run in parallel.  When operating on multiple
projects, this can greatly speed things up.  Across 1000 repos, it
goes from ~30sec to ~3sec with the default -j8.

Change-Id: I0dc62d704c022dd02cac0bd67fe79224f4e34095
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297484
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-02-27 19:45:14 +00:00
Mike Frysinger
8d2a6df1fd progress: include execution time summary
We're already keeping tracking of the start time, so might as
well use it to display overall execution time for steps.

Change-Id: Ib4cf8b2b0dfcdf7b776a84295d59cc569971bdf5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298482
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-26 17:16:29 +00:00
Raman Tenneti
ceba2ddc13 sync: superproject - support for switching hosts and switching branches.
+ superproject will be fetched into a directory with the name
  “<remote name>-superproject.git” instead of the current
  “superproject.git” folder.

+ Deleted  _Clone method and added _Init method.

+ _Init method will do “git init --bare <remote>-superproject.git”.
  It will create the folder and set up a bare repository in
  <remote>-superproject.git folder.

+ _Fetch method, will pass <remote url>, <branch> arguments.
  Moved the --filter argument from “git clone” to “git fetch”.
  _Fetch method will execute the following command to fetch
  superproject. Added --no-tags argument.

  master:  git fetch <remote url> --force --no-tags --filter blob:none
  branch:  git fetch <remote url> --force --no-tags --filter blob:none \
           <branch>:<branch>

+ Performance improvements for aosp-master
  ++ repo init performance improved from 35 seconds to 17 seconds.
  ++ repo init --use-superproject is around 5 to 7 secsonds slower.
  ++ repo sync --use-superproject is around 3 to 4 minutes faster.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL.

$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
  real	0m20.648s
  user	0m8.046s
  sys	0m3.271s

+ Without superproject
$ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main
  real	0m13.078s
  user	0m9.783s
  sys	0m2.528s

$ time repo_dev sync -c -j32 --use-superproject
...
  real	15m7.072s
  user	110m7.216s
  sys	20m17.559s

+ Without superproject
$ time repo sync -c -j32
...
  real	19m25.644s
  user	91m56.331s
  sys	20m59.170s

Bug: [google internal] b/180492484
Bug: [google internal] b/179470886
Bug: [google internal] b/180124069
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707

Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-25 20:45:26 +00:00
Mike Frysinger
45ad1541c5 grep: move nested func out to the class
This is in preparation for adding jobs support.  The nested function
is referenced in the options object which can't be pickled, so pull
it out into a static method instead.

Change-Id: I280ed2bf26390a0203925517a0d17c13053becaa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297983
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 20:13:33 +00:00
Mike Frysinger
7b586f231b sync: capture all git output by default
The default sync output should show a progress bar only for successful
commands, and the error output for any commands that fail.  Implement
that policy here.

Bug: https://crbug.com/gerrit/11293
Change-Id: I85716032201b6e2b45df876b07dd79cb2c1447a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297905
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 20:13:18 +00:00
Mike Frysinger
fbb95a4342 progress/sync: include active number of jobs
Provide a bit more info to users that things are actively running.

Bug: https://crbug.com/gerrit/11293
Change-Id: Ie8eeaa8804d1ca71cf5c78ad850fa2d17d26208c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297904
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 20:13:18 +00:00
Mike Frysinger
4e05f650e0 progress: always enable always_print_percentage
The idea for skipping some progress updates was to avoid spending
too much time on the progress bar itself.  Unfortunately, for large
projects (100s if not 1000s) of repos, we get into the situation
with large/slow checkouts that we skip showing updates when a repo
finishes, but not enough repos finished to increase the percent.

Since the progress bar should be relatively fast compared to the
actual network & local dick operations, have it show an update
whenever the caller requests it.  A test with ~1000 repos shows
that the progress bar in total adds <100ms.

Bug: https://crbug.com/gerrit/11293
Change-Id: I708a0c4bd923c59c7691a5b48ae33eb6fca4cd14
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297903
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 20:13:18 +00:00
Mike Frysinger
23882b33fe init: support -b HEAD as a shortcut to "the default"
When people switch to non-default branches, they sometimes want to
switch back to the default, but don't know the exact name for that
branch.  Add a -b HEAD shortcut for that.

Change-Id: I090230da25f9f5a169608115d483f660f555624f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297843
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 20:12:51 +00:00
Mike Frysinger
92304bff00 project: fix http error retry logic
When sync moved to consume clone output, it merged stdout & stderr,
but the retry logic in this function is based on stderr only.  Move
it over to checking stdout.

Change-Id: I71bdc18ed25c978055952721e3a768289d7a3bd2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297902
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 20:12:42 +00:00
Mike Frysinger
adbd01e0d3 tests: fix init subcmd after url change
My recent 401c6f0725 ("init: make
--manifest-url flag optional") commit broke the unittest.

Change-Id: I19ad0e8c8cbb84ab5474ebc370e00acfe957e136
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298223
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 17:07:11 +00:00
Mike Frysinger
37ac3d626f tests: refactor manifest tests
The XmlManifestTests class is getting to be large and we're only
adding more to it.  Factor out the core logic into a new TestCase
so we can reuse it to better group more tests.

Change-Id: I5113444a4649a70ecfa8d83d3305959a953693f7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298222
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 17:06:56 +00:00
Raman Tenneti
55d6a5a3a2 sync: use superproject if manifest's config has superproject enabled.
If --use-superproject is passed as argument to "repo init", then
--use-superproject need not be specified during "repo sync".

Tested the code with the following commands.

$ time repo_dev sync -c -j32
...
WARNING: --use-superproject is experimental and not for general use

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: Ibb33f3038a2515f74a6c4f7cb785d354b26ee680
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298102
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Ian Kasprzak <iankaz@google.com>
2021-02-25 16:35:53 +00:00
Mike Frysinger
6db4097f31 docs: add warnings about repos data model
For people coming across these docs and thinking that repo's methods
are good to replicate, add a note warning them against doing so.

Change-Id: I443a783794313851a6e7ba1c39baebac988bff9a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298164
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 15:48:03 +00:00