This is in preparation for simplifying the jobs support. The nested
function is referenced in the options object which can't be pickled,
so pull it out into a static method instead.
Change-Id: I01d3c4eaabcb8b8775ddf22312a6e142c84cb77d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298722
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This fixes intermingling of parallel jobs and simplifies the code
by switching to subprocess.run. This also provides stable output
in the order of projects by returning the output as a string that
the main loop outputs.
This drops support for interactive commands, but it's unclear if
anyone was relying on that, and the default behavior (-j2) made
that unreliable. If it turns out someone still wants this, we can
look at readding it.
Change-Id: I7555b4e7a15aad336667292614f730fb7a90bd26
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297482
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use a pool contextmanager to take care of the messy details like
properly cleaning it up when aborting.
Change-Id: I264ebb591c2e67c9a975b6dcc0f14b29cc66a874
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297243
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Extend the Command class to support adding the --jobs option to the
parser if the command declares it supports running in parallel. Also
pull the default value used for the number of local jobs into the
command module so local commands can share it.
Change-Id: I22b0f8d2cf69875013cec657b8e6c4385549ccac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297024
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Clean up a few linter warnings.
Change-Id: I531d0263a202435d32d83d87ec24998f4051639c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297062
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Recent changes in ChromeOS Infra to ensure we're reading from
snapshot manifests properly have exposed several bugs in our
assumptions about manifest files. Mainly that the revision field
for a project does _not_ have to refer to a ref, it can just be
a commit hash.
Several places assume that the revision field can be parsed as a
ref to get the branch the project is on, which isn't true. To fix
this we need to be able to look at the upstream and dest-branch
attributes of the repo, so we expose them through the environment
variables set in `repo forall`.
Test: manual 'repo forall' run
Bug: https://crbug.com/1032441
Change-Id: I2c039e0f4b2e0f430602932e91b782edb6f9b1ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263132
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
In order to remove the stream fileno() will be called on the filedescriptor.
If the file is already closed fileno() will raise an error and forall
will fail.
Bug: https://crbug.com/gerrit/12563
Change-Id: Ib7b895fe881c844e3eb3672b011fdcdbdae63024
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/262838
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Karsten Pfeiffer-Raddatz <raddatz.karsten@gmail.com>
Different Python & OS versions have different environ behavior wrt
accepted types & encoding. Since we're migrating to be Python 3 only,
lets change our code to assume strings always work as that's what the
newer Python 3 does. This will fail under Python 2 for some env vars,
mostly on Windows, but the effort of maintaining shim layers that can
handle these edge cases isn't worth it when we're dropping that code.
We leave the logic in the `repo` launcher for now as it is simple, and
we want it to be able to switch versions a bit longer than the rest of
the tree.
Here's the support table:
| *NIX | Windows |
Python 2 | ASCII string | str or bytes, not unicode |
Python 3 | str or bytes | str only |
Windows uses strings natively in its environment all the time. But it
doesn't allow unicode strings under Python 2, so we have to encode.
Python 2 on *NIX is funky in that it always lowers to ASCII, so we had
to manually encode to avoid errors regardless of unicode or str.
Python 3 on Windows & *NIX will accept strings. *NIX will also accept
bytes but Windows will not.
Bug: https://crbug.com/gerrit/12145
Change-Id: I3cf8f95a06902754ea1f08ad4b28503f7063531b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248972
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
We increased the max line length to 100 columns which got rid of
the majority of these warnings, but there were still a few lines
that exceeded 100 columns.
Change-Id: Ib3372868ca2297f83073a14f91c8ae3df9d0d0e6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254699
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E301,E302,E303,E305,E306
Manually fix issues in project.py caused by misuse of block comments.
Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E121 continuation line under-indented for hanging indent
- E122 continuation line missing indentation or outdented
- E125 continuation line with same indent as next logical line
- E126 continuation line over-indented for hanging indent
- E127 continuation line over-indented for visual indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E121,E122,E125,E126,E127,E128,E129,E131
Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
Execution of 'repo forall -p -c' doesn't work with Py3 and ends up
with an error:
Got an error, terminating the pool: TypeError: can only concatenate
str (not "bytes") to str
That's fixed by using the decode() method.
Change-Id: Ice01aaa1822dde8d957b5bf096021dd5a2b7dd51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253659
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Jiri Tyr <jiri.tyr@gmail.com>
In CrOS, our infra has to deal with partial checkouts constantly
(for a variety of reasons). To help reset back to a good state,
we run git commands via `repo forall`, but don't care about the
missing checkouts. Add a flag so we can disambiguate between
missing repos and failing git subcommands.
Bug: https://crbug.com/1013377
Bug: https://crbug.com/1013623
Change-Id: Ie3498c6d111276c60d2ecedbba21bfa778588d50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/241935
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If a repo doesn't exist (e.g. an incomplete sync), make sure we exit
non-zero when they get skipped.
Change-Id: Ifff711e374416b1e6b9b8da4fdc6f14b27ced450
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239235
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
A common pattern in our subcommands is to verify the arguments &
options before executing things. For some subcommands, that check
stage is quite long which makes the execution function even bigger.
Lets split that logic out of the execute phase so it's easier to
manage these.
This is most noticeable in the sync subcommand whose Execute func
is quite large, and the option checking makes up ~15% of it.
The manifest command's Execute can be simplified significantly as
the optparse configuration always sets output_file to a string.
Change-Id: I7097847ff040e831345e63de6b467ee17609990e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234834
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
%prog represents the full subcommand ("repo" + subcommand name), not a
Windows-style environment variable for "repo". The current help output
shows
repo forall% forall ...
Correct the variable usage so it shows "repo forall ..." instead.
Change-Id: I1fea55572428cc922ddf24ace1168a3d8f82dad0
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.
Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
Since gitiles recommends using # headers over ---/=== underlines,
change the manifest-format.md over and all our help texts.
Change-Id: I96391d41fba769e9f26870d497cf7cf01c8d8ab3
pylint is not used since bb5b1a0. The pyflakes cleanup mentioned in that
commit has not been done, but given that this project is no longer being
actively developed I don't think it's worth spending time doing it.
Leaving the pylint suppressions causes confusion because it leads people
to think that we are still using pylint.
Change-Id: If7d9f280a0f408c780f15915ffdb80579ae21f69
If a user executes:
repo forall -c echo $REPO_PROJECT
then $REPO_NAME is expanded by the user's shell first, and passed
as $1 to the shell that executes echo. This will either result in
no output, or output of whatever REPO_NAME is set to in the user's
shell. Either way, this is an unexpected result.
The correct way to do it is:
repo forall -c 'echo $REPO_PROJECT'
such that $REPO_NAME is passed in to the shell literally, and then
expanded to the value set in the environment that was passed to
the shell.
Update the documentation to make this clearer.
Change-Id: I713caee914172ad8d8f0fafacd27026502436f0d
Python on Windows does not support non blocking file operations.
To workaround this issue, we instead use Threads and a Queue to
simulate non-blocking calls. This is happens only when running
with the native Windows version of Python, meaning Linux and Cygwin
are not affected by this change.
Change-Id: I4ce23827b096c5138f67a85c721f58a12279bb6f
Make it possible to exclude projects using regex/wildcard.
The syntax is similar to that of the -r option, e.g.:
repo forall -i ^platform/ ^device/ -c 'echo $REPO_PROJECT'
Change-Id: Id250de5665152228c044c79337d3ac15b5696484
It was only displaying 'Project list error: GitError()'
without any useful info about the project nor the error
Change-Id: Iad66cbaa03cad1053b5ae9ecc90d7772aa42ac13
Enable operating against groups of repositories. As it stands, it isn't
compatible with `-r/--regex`.
`repo forall -g groupname -c pwd` will run `pwd` for all projects in
groupname.
`repo forall -g thisgroup,-butnotthisone -c pwd` will run `pwd` for all
projects in `thisgroup` but not `butnotthisone`.
`repo list -g groupname -n` will list all the names of repos in
`groupname`.
Change-Id: Ia75c50ce52541d1c8cea2874b20a4db2e0e54960
If a workspace is synced with the -s or -t option, the included projects
may be different to those in the original manifest. However, when using
the forall command, the list of the projects from the original manifest
is used.
If the smart sync manifest file exists, use it to override the original
manifest.
Change-Id: Iaefcbe148d2158ac046f158d98bbd8b5a5378ce7
git rev-parse fails for projects that don't have an explicit revision
specified, and don't have a branch of the same name as the default
revision. This can be the case in a workspace synced with the smart
sync (-s) or smart tag (-t) option.
Change-Id: I19bfe9fe7396170379415d85f10f6440dc6ea08f
If the generator that produces per-project worker arguments raises an
exception it triggers python bug http://bugs.python.org/issue8296.
Rewrite the generator expression as a generator function, and catch
Exceptions and KeyboardInterrupts to end the iteration.
Also add a pool worker initializer to disable SIGINT to prevent
KeyboardInterrupts inside multiprocessing.Pool in the worker threads
causing the same problem.
Fixes easy-to-reproduce hangs when hitting ctrl-c during
repo forall -c echo
Change-Id: Ie4a65b3e1e07a64ed6bb6ff20f3912c4326718ca
Before, a list was generated, which is why there was a massive delay.
Using a generator will allow processes to start straight away
Change-Id: Ia325b0b340cc328c08c9bcc92a6709bbdaf6a664
Enable '--jobs' ('-j') option in the forall subcommand. For -jn
where n > 1, the '-p' option can no longer guarantee the
continuity of console output between the project header and the
output from the worker process.
SIG_INT is sent to all worker processes upon keyboard interrupt
(Ctrl+C).
Bug: Issue 105
Change-Id: If09afa2ed639d481ede64f28b641dc80d0b89a5c
For long-running forall commands sometimes it's useful to know which
iteration is currently running. Add REPO_I and REPO_COUNT environment
variables to reflect the current iteration count as well as the total
number of iterations so that the user can build simple status
indicators.
Example:
$ repo forall -c 'echo $REPO_I / $REPO_COUNT; git gc'
1 / 579
Counting objects: 41, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (19/19), done.
Writing objects: 100% (41/41), done.
Total 41 (delta 21), reused 41 (delta 21)
2 / 579
Counting objects: 53410, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (10423/10423), done.
Writing objects: 100% (53410/53410), done.
Total 53410 (delta 42513), reused 53410 (delta 42513)
3 / 579
...
Change-Id: I9f28b0d8b7debe423eed3b4bc1198b23e40c0c50
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It doesn't make sense to print the relpath, since there's nothing
checked out there and the dir shouldn't even exist.
Change-Id: Id43631a8e0895929d3a5ad4ca8c2dc9e3d233e70
Filter the project list based on regex or wildcard matching
of strings, then we can handle subset of all projects.
Change-Id: Ib6c23aec79e7d981f7b6a5eb0ae93c44effec467
Signed-off-by: Zhiguang Li <muzili@gmail.com>
Add a new option (-e, --abort-on-errors) which will cause forall to
abort without iterating through remaining projects if a command
exits unsuccessfully.
Bug: Issue 17
Change-Id: Ibea405e0d98b575ad3bda719d511f6982511c19c
Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
pylint configuration file (.pylintrc) is added, and submission
instructions are updated to include pylint usage steps.
Deprecated pylint suppression (`disable-msg`) is updated in a few
modules to make it work properly with the latest version (0.26).
Change-Id: I4ec2ef318e23557a374ecdbf40fe12645766830c
Fixing more issues found with pylint. Some that were supposed to
have been fixed in the previous sweep (Ie0db839e) but were missed:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
And some more:
W0631: Using possibly undefined loop variable 'name'
W0223: Method 'name' is abstract in class 'name' but is not overridden
W0231: __init__ method from base class 'name' is not called
Change-Id: Ie119183708609d6279e973057a385fde864230c3
Fix the following issues reported by pylint:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
W0612: Unused variable 'name'
W0613: Unused argument 'name'
W0102: Dangerous default value 'value' as argument
W0105: String statement has no effect
Also fixed a few cases of inconsistent indentation.
Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
Allow the optional addition of "annotation" nodes nested under
projects. Each annotation node must have "name" and "value"
attributes. These name/value pairs will be exported into the
environment during any forall command, prefixed with "REPO__"
In addition, an optional "keep" attribute with case insensitive "true"
or "false" values can be included to determine whether the annotation
will be exported with 'repo manifest'
Change-Id: Icd7540afaae02c958f769ce3d25661aa721a9de8
Signed-off-by: James W. Mills <jameswmills@gmail.com>
If a project is missing locally, it might be OK to skip over it
and continue running the same command in other projects.
Bug: REPO-43
Change-Id: I64f97eb315f379ab2c51fc53d24ed340b3d09250
Signed-off-by: Shawn O. Pearce <sop@google.com>
(cherry picked from commit d4cd69bdef)
Windows allows the environment to have unicode values.
This will cause Python to fail to execute the command.
Change-Id: I37d922c3d7ced0d5b4883f0220346ac42defc5e9
Signed-off-by: Shawn O. Pearce <sop@google.com>
The revisionExpr field now holds an expression from the manifest,
such as "refs/heads/master", while revisionId holds the current
commit-ish SHA-1 of the revisionExpr. Currently that is only
filled in if the manifest points directly to a SHA-1.
Signed-off-by: Shawn O. Pearce <sop@google.com>
When trying to read log output from many projects at once it can
be difficult to make sense of which messages came from where.
For many professional developers it is common to want to view the
last week's worth of your work, so you can write a weekly summary
of your activity for your status report.
This is easier with the new -p option:
repo forall -pc git log --reverse --since=1.week.ago --author=sop
produces a report of all commits written by me in the last week,
formatted in a paged output display, with headers inserted in
front of each project's output.
Where this can be even more useful is with git log's pickaxe,
e.g. now we can use:
repo forall -pc git log -Sbar v1.0..v1.1
to locate all additions or removals of the symbol 'bar' since v1.0,
up to and including v1.1. Before displaying the matching commits in
a project, a project header is shown, giving the user some context
information for the matching results.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the value obtained is None we now set the variable to
'' instead, in an attempt to make execve() happier about
our 3rd argument, the env dictionary.
Signed-off-by: Shawn O. Pearce <sop@google.com>
REPO_PATH is the path relative the the root of the client.
REPO_REMOTE is the name of the remote system from the manifest.
REPO_LREV is the name of the revision from the manifest, but
translated to something the local repository knows.
REPO_RREV is the name of the revision from the manifest.
This allows us to do commands like:
repo forall -c 'echo "(cd $REPO_PATH && git checkout `git rev-parse HEAD`)"'
