Commit Graph

25 Commits

Author SHA1 Message Date
Kuang-che Wu
8da4861b38 subcmds: reduce multiprocessing serialization overhead
Follow the same approach as 39ffd9977e to reduce serialization overhead.

Below benchmarks are tested with 2.7k projects on my workstation
(warm cache). git tracing is disabled for benchmark.

(seconds)              | v2.48 | v2.48 | this CL | this CL
	               |       |  -j32 |         |    -j32
-----------------------------------------------------------
with clean tree state:
branches (none)        |   5.6 |   5.9 |    1.0  |    0.9
status (clean)         |  21.3 |   9.4 |   19.4  |    4.7
diff (none)            |   7.6 |   7.2 |    5.7  |    2.2
prune (none)           |   5.7 |   6.1 |    1.3  |    1.2
abandon (none)         |  19.4 |  18.6 |    0.9  |    0.8
upload (none)          |  19.7 |  18.7 |    0.9  |    0.8
forall -c true         |   7.5 |   7.6 |    0.6  |    0.6
forall -c "git log -1" |  11.3 |  11.1 |    0.6  |    0.6

with branches:
start BRANCH --all     |  21.9 |  20.3 |   13.6  |    2.6
checkout BRANCH        |  29.1 |  27.8 |    1.1  |    1.0
branches (2)           |  28.0 |  28.6 |    1.5  |    1.3
abandon BRANCH         |  29.2 |  27.5 |    9.7  |    2.2

Bug: b/371638995
Change-Id: I53989a3d1e43063587b3f52f852b1c2c56b49412
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440221
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Kuang-che Wu <kcwu@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>
2024-10-23 23:34:34 +00:00
Aravind Vasudevan
c993c5068e subcmds: Use repo logger
Bug: b/292704435
Change-Id: Ia3a45d87fc0bf0d4a1ba53050d9c3cd2dba20e55
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/386236
Reviewed-by: Jason Chang <jasonnc@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Aravind Vasudevan <aravindvasudev@google.com>
2023-09-14 17:13:37 +00:00
Mike Frysinger
6447733eb2 isort: format codebase
Change-Id: I6f11d123b68fd077f558d3c21349c55c5f251019
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/383715
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2023-08-22 18:32:22 +00:00
Jason Chang
1a3612fe6d Raise RepoExitError in place of sys.exit
Bug: b/293344017
Change-Id: Icae4932b00e4068cba502a5ab4a0274fd7854d9d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/382214
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Jason Chang <jasonnc@google.com>
2023-08-10 23:46:31 +00:00
Gavin Mak
ea2e330e43 Format codebase with black and check formatting in CQ
Apply rules set by https://gerrit-review.googlesource.com/c/git-repo/+/362954/ across the codebase and fix any lingering errors caught
by flake8. Also check black formatting in run_tests (and CQ).

Bug: b/267675342
Change-Id: I972d77649dac351150dcfeb1cd1ad0ea2efc1956
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/363474
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2023-03-22 17:46:28 +00:00
LaMont Jones
cc879a97c3 Add multi-manifest support with <submanifest> element
To be addressed in another change:
 - a partial `repo sync` (with a list of projects/paths to sync)
   requires `--this-tree-only`.

Change-Id: I6c7400bf001540e9d7694fa70934f8f204cb5f57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322657
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-02-17 21:57:55 +00:00
Mike Frysinger
4f21054c28 commands: document the "common" class attribute
Switch it to uppercase to make it clear it's a constant, and add
documentation so its usage is clear.

Change-Id: I6d281a66a90b5908b3131585c9945e88cfe815ea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309322
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-06-15 06:07:37 +00:00
Mike Frysinger
b5d075d04f command: add a helper for the parallel execution boilerplate
Now that we have a bunch of subcommands doing parallel execution, a
common pattern arises that we can factor out for most of them.  We
leave forall alone as it's a bit too complicated atm to cut over.

Change-Id: I3617a4f7c66142bcd1ab030cb4cca698a65010ac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301942
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-04-15 05:10:16 +00:00
Mike Frysinger
9180a07b8f command: make --verbose/--quiet available to all subcommands
Add new CommonOptions entry points to move the existing --jobs to,
and relocate all --verbose/--quiet options to that.  This provides
both a consistent interface for users as well as for code.

Change-Id: Ifaf83b88872421f4749b073c472b4a67ca6c0437
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303224
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-13 22:25:17 +00:00
Mike Frysinger
d246d1fee7 grep: add --jobs support
Use multiprocessing to run in parallel.  When operating on multiple
projects, this can greatly speed things up.  Across 1000 repos, it
goes from ~40sec to ~16sec with the default -j8.

The output processing does not appear to be a significant bottle
neck -- it accounts for <1sec out of the ~16sec runtime.  Thus we
leave it in the main thread to simplify the code.

Change-Id: I750b72c7711b0c5d26e65d480738fbaac3a69971
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297984
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-01 14:43:19 +00:00
Mike Frysinger
45ad1541c5 grep: move nested func out to the class
This is in preparation for adding jobs support.  The nested function
is referenced in the options object which can't be pickled, so pull
it out into a static method instead.

Change-Id: I280ed2bf26390a0203925517a0d17c13053becaa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297983
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 20:13:33 +00:00
Mike Frysinger
784ccfc040 strip python2-only coding:utf-8 & print_function settings
We're committed to Python 3 at this point, so clean up boilerplate.

Bug: https://crbug.com/gerrit/10418
Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:05 +00:00
Mike Frysinger
6f1c626a9b drop old git_require checks
We've been requiring git-1.7.2 since Oct 2012, so we can safely drop
the individual checks sprinkled throughout the code base for older.

Change-Id: I1737fff7b3f27f475960b0bff9cb300aefd5d108
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253135
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 11:44:59 +00:00
David Pursehouse
819827a42d Fix blank line issues reported by flake8
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E301,E302,E303,E305,E306

Manually fix issues in project.py caused by misuse of block comments.

Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:36:40 +00:00
Mike Frysinger
72ab852ca5 grep: handle errors gracefully
If `git grep` fails in any project checkout (e.g. an incomplete
sync), make sure we print that error clearly rather than blowing
up, and exit non-zero in the process.

Bug: https://crbug.com/gerrit/11613
Change-Id: I31de1134fdcc7aaa9814cf2eb6a67d398eebf9cf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239237
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-10-01 05:45:58 +00:00
Mike Frysinger
f601376e13 set default file encoding to utf-8
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.

Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
2019-06-13 14:30:52 +00:00
Mike Frysinger
b8f7bb04d0 update markdown/help header format
Since gitiles recommends using # headers over ---/=== underlines,
change the manifest-format.md over and all our help texts.

Change-Id: I96391d41fba769e9f26870d497cf7cf01c8d8ab3
2018-10-10 01:28:43 -04:00
David Pursehouse
8f62fb7bd3 Tidy up code formatting a bit more
Enable the following Pylint warnings:

  C0322: Operator not preceded by a space
  C0323: Operator not followed by a space
  C0324: Comma not followed by a space

And make the necessary fixes.

Change-Id: I74d74283ad5138cbaf28d492b18614eb355ff9fe
2012-11-14 12:09:38 +09:00
Sarah Owens
cecd1d864f Change print statements to work in python3
This is part of a series of changes to introduce Python3 support.

Change-Id: I373be5de7141aa127d7debdbce1df39148dbec32
2012-11-13 17:33:56 -08:00
David Pursehouse
1d947b3034 Even more coding style cleanup
Fixing some more pylint warnings:

W1401: Anomalous backslash in string
W0623: Redefining name 'name' from outer scope
W0702: No exception type(s) specified
E0102: name: function already defined line n

Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
2012-10-30 10:28:20 +09:00
David Pursehouse
e15c65abc2 Remove unused imports
There are several imports that are not used.  Remove them.

Change-Id: I2ac3be66827bd68d3faedcef7d6bbf30ea01d3f2
2012-08-23 12:15:26 +02:00
Sebastian Schmidt
feb39d61ef Fix format string bugs in grep
This fixes some format string bugs in grep which cause repo to with
"TypeError: not enough arguments for format string" when grepping and
the output contains a valid Python format string.

Change-Id: Ice8968ea106148d409490e4f71a2833b0cc80816
2010-06-17 19:09:37 -07:00
Shawn O. Pearce
f0d4c36701 grep: Only use --color on git 1.6.3 and later
The --color flag wasn't introduced until git 1.6.3.  Prior to that
version, `git grep --color` just produces a fatal error, as it is
an unsupported option.  Since this is just pretty output and is not
critical to execution, we can simply omit the option if the version
of git we are running on doesn't support it.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-06-12 09:33:48 -07:00
Shawn O. Pearce
50fa1ac6db Clarify the option section header in 'repo help grep'
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 11:44:33 -07:00
Shawn O. Pearce
b812a36236 Add 'repo grep' to support searching all projects
Users can now use 'repo grep' to search all projects, rather than
'repo forall -c git grep'.  Its not only shorter to type, but it
also filters results better by highlighting which projects matched
in the client workspace.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-10 20:37:47 -07:00