Compare commits

..

59 Commits

Author SHA1 Message Date
0b57eed8f0 repo: bump launcher version for accumulated fixes
Change-Id: I5d9b866cc53d3824a01f5f0af127cf0c3ff97366
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254757
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 23:27:11 +00:00
72b6dc8891 repo: avoid bare excepts to allow SystemExit to bubble
Bug: https://crbug.com/gerrit/12327
Change-Id: I4ce1142379b111f9ba3a2e5a437026e5c0378a9e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254756
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 22:21:52 +00:00
e19d9e1a65 sync: add a "finished" message
Some people find the existing output to be a bit confusing.  It spews
a lot of git output before exiting, but it's not exactly clear what
the final state is when things pass.  Add an explicit message.

Bug: https://crbug.com/gerrit/10501
Change-Id: I9de83b595d3185feb820005b8fc81c6adc55b357
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254732
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 20:54:57 +00:00
8ddff5c74f repo: add --version support to the launcher
We can get version info when in a checkout, but it'd be helpful
to show that info at all times.

Change-Id: Ieeb44a503c9d7d8c487db4810bdcf3d5f6656c82
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254712
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 20:54:50 +00:00
8409410aa2 repo: export GIT_TRACE2_PARENT_SID
This helps with people tracing repo/git execution.  We use a similar
format to git, but a little simpler since we always initialize the
env var setting, and we want to avoid too much overhead.

Bug: https://crbug.com/gerrit/12314
Change-Id: I75675b6cc4c6f7c4f5e09f54128eba9456364d04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254331
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 19:57:42 +00:00
dc63181fcd flake8: Add comments in config to explain suppressed checks
Change-Id: Ib5c09b36d40a96ba9167b42b3bd2f1ed072660b7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254611
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 12:33:02 +00:00
f700ac79c3 repo: move parser init out of module scope
We import the wrapper on the fly, so minimize how much code we run
in module scope.  It's pointless/wasted when importing.

Change-Id: I4a71c2030325d0a639585671cd7ebe8f22687ecd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254072
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 11:46:30 +00:00
6f1c626a9b drop old git_require checks
We've been requiring git-1.7.2 since Oct 2012, so we can safely drop
the individual checks sprinkled throughout the code base for older.

Change-Id: I1737fff7b3f27f475960b0bff9cb300aefd5d108
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253135
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 11:44:59 +00:00
77479863da flake8: Ignore 'line break before/after binary operator'
- W503 line break before binary operator
- W504 line break after binary operator

There doesn't seem to be a nice way of fixing all of these without
replacing W503 with W504 or vice-versa, or unwrapping them resulting
in excessively long lines. Let's just suppress them.

Change-Id: I7846d0124054f58e1cb480d4459cd9c86b737a50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254608
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 07:29:25 +00:00
16a5c3ac51 git_config: Stop using backslash to wrap lines
Unwrap one unnecessarily wrapped line, and use parentheses on
a wrapped condition instead of wrapping with backslashes.

Change-Id: I12679a0547dd822b15a6551e0f6c308239ff7b2d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254607
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 07:29:25 +00:00
145e35b805 Fix usage of bare 'except'
flake8 reports:

  E722 do not use bare 'except'

Replace them with 'except Exception' per [1] which says:

  Bare except will catch exceptions you almost certainly don't want
  to catch, including KeyboardInterrupt (the user hitting Ctrl+C) and
  Python-raised errors like SystemExit

  If you don't have a specific exception you're expecting, at least
  except Exception, which is the base type for all "Regular" exceptions.

[1] https://stackoverflow.com/a/54948581

Change-Id: Ic555ea9482645899f5b04040ddb6b24eadbf9062
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254606
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:49:25 +00:00
819827a42d Fix blank line issues reported by flake8
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E301,E302,E303,E305,E306

Manually fix issues in project.py caused by misuse of block comments.

Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:36:40 +00:00
abdf750061 Fix indentation issues reported by flake8
- E121 continuation line under-indented for hanging indent
- E122 continuation line missing indentation or outdented
- E125 continuation line with same indent as next logical line
- E126 continuation line over-indented for hanging indent
- E127 continuation line over-indented for visual indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E121,E122,E125,E126,E127,E128,E129,E131

Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:36:22 +00:00
0ab95ba6d0 git_config: Unwrap unnecessarily wrapped line
Change-Id: I56806e8b9b09cd0f7fb834d7edc412682f2af1db
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254604
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:32:47 +00:00
5a2517f411 command: Add parentheses on wrapped condition
Surround the condition with parentheses rather than using
backslashes. This prevents confusion about indentation when
running flake8/autoflake8.

Change-Id: I01775b96f817ee616f545b55369a4864fa1d6712
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254603
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:32:47 +00:00
54a4e6007a Fix various whitespace issues reported by pyflakes
- E201 whitespace after '['
- E202 whitespace before '}'
- E221 multiple spaces before operator
- E222 multiple spaces after operator
- E225 missing whitespace around operator
- E226 missing whitespace around arithmetic operator
- E231 missing whitespace after ','
- E261 at least two spaces before inline comment
- E271 multiple spaces after keyword

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E201,E202,E221,E222,E225,E226,E231,E261,E271

Change-Id: I367113eb8c847eb460532c7c2f8643f33040308c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254601
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:00:16 +00:00
42339d7e52 Remove redundant backslashes
fleka8 reports:

  E502 the backslash is redundant between brackets

Fixed automatically with autopep8:

  git-repo $ git ls-files | grep py$ | xargs autopep8 --in-place --select E502

Change-Id: I1486ae1d17206918474363daf518274c5be8daed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254602
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:45:44 +00:00
03ae99290a pager: Remove unnecessary semicolons
flake8 reports:

  E703 statement ends with a semicolon

Change-Id: Ia63fc9efb04425e425c0f289272db76ff1ceeb34
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254600
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:40:33 +00:00
9090e804ab Remove unused imports
flake8 reports:

  F401 'name' imported but unused

Change-Id: Id45d6efa87ddf53f2c4a0f0c4136ea361ab1b746
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254592
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 05:40:10 +00:00
eeff3537de Fix tests for membership to use 'not in'
flake8 reports:

  E713 test for membership should be 'not in'

Change-Id: I4446be67c431b7267105b53478d2ceba2af758d7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254451
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:18:17 +00:00
8f78a83083 upload: Fix tests for object identity to use 'is not'
flake8 reports:

  E714 test for object identity should be 'is not'

Change-Id: Ib8c4100babaf952bbfe65fd56555ece8a958e4b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254450
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:17:49 +00:00
e5913ae410 Fix flake8 E251 unexpected spaces around keyword / parameter equals
Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place --select E251

Change-Id: I58009e1c8c91c39745d559ac919be331d4cd9e77
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254598
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:17:08 +00:00
119085e6b1 flake8: Increase max line length from 80 to 100
The Google style guide for python [1] says the maximum line length
should be 80, but there are several lines in the code base that
exceed it:

  git ls-files | grep py$ | xargs flake8 | grep E501 | wc -l
  64

I don't think it's worth going through and re-wrapping all those,
so just increase the limit to 100 which seems to be a reasonable
compromise:

  git ls-files | grep py$ | xargs flake8 | grep E501 | wc -l
  6

Leave the re-rewrapping of those lines for a follow-up commit,
though.

[1] http://google.github.io/styleguide/pyguide.html#32-line-length

Change-Id: Ia37c34301163431fd1fb4fb6697a4a482d6be077
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254595
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:16:11 +00:00
086710465e upload: Fix flake8 E241 multiple spaces after ','
Change-Id: I3a65869f9d006027270a7826d7982950c0e6759a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254597
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:00:36 +00:00
ed4f2113d2 project: make syncing a little more self-healing
We have a few files that we optionally symlink from the work tree
.git/ to the .repo/projects/ path.  If they don't exist when we
first initialize, then we skip creating symlinks.  If the files
are created later on under the work tree .git/, repo gets upset.

This can happen with the packed-refs file: if we don't have any
packed refs initially, we don't symlink it.  But if git tries to
pack refs later on and creates the file, the project gets wedged.

We could create an empty file initially and then symlink it, but
for some files, it's not clear we want to always do that (e.g.
the .git/shallow setting).  Instead, lets make handling of these
paths more dynamic.  If they show up later on in the work tree
.git/ only, we'll take care of relocating & symlinking.  This
also makes repo a little more robust and autorecovers incase a
path goes missing in one of the dirs.

Ideally we wouldn't monkey around at all here, but considering
the only option we give to users currently is to blow things
away with --force-sync, this seems a bit better.

Bug: https://crbug.com/gerrit/12324
Change-Id: Ia6960f1896ac6d890c762d7d053684a1c6ab2c87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254632
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 04:48:36 +00:00
719675bcec info: Fix formatting of block comment
flake8 reports:

  E265 block comment should start with '# '

While we're at it, add a period at the end of the comment sentence.

Change-Id: Icb7119079a1d64e6defafc3f6d24e99dbf16139d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254596
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 04:31:40 +00:00
21c1575ee4 upload: add a --ignore-hooks option
When upload hooks fail, people are forced to use --no-verify to upload
CLs anyways.  When projects have flaky hooks, this trains people to
always use that option.  This is obviously bad: hooks might get fixed,
or some of the hooks are always good & people should review.

Lets add an --ignore-hooks option.  This still runs the hooks, but any
failures will be ignored and allow the user to upload anyways.

Bug: https://crbug.com/gerrit/12230
Change-Id: Ide2ac8a40a656bfcd6aae20c3ce8118e06bf909b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254452
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 04:18:49 +00:00
8f9e02231a Remove trailing blank lines
flake8 reports:

  W391 blank line at end of file

Change-Id: I5498b2de2d1268d4f1f4b9e1760f9fa93a6da4cd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254594
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 02:58:17 +00:00
348e218d5b test_project.py: Remove unused variable in 'with' statement
flake8 reports:

 F841 local variable 'f' is assigned to but never used

Change-Id: If808eb381ee44c7da71e6281615a06a6723cf945
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254593
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 02:56:32 +00:00
4bbba7d627 Fix duplicate method name in test_project.py
flake8 reports:

  F811 redefinition of unused 'test_src_block_dir' from line 259

which is caused by having two methods with the same name. Rename
them both to better desribe their purpose.

Change-Id: If7612a42001776d71bb1a6a80fc631d3d262e6ce
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254449
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 02:55:51 +00:00
dc1d0e0c7f Revert "Save cookies back to jar when fetching clone.bundle"
This reverts commit 4abf8e6ef8.

The curl process for updating the cookie file is not atomic.  When
fetching many bundles in parallel, we can sometimes corrupt the file
causing it to be cleared.  Since users should manage gitcookies on
their own, leave it read-only.

Bug: https://crbug.com/gerrit/12300
Change-Id: Id472c99b197bc4cf8533c649f8881509f38643c1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254092
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 02:00:16 +00:00
82caef67a1 repo: lower min version of git a bit
We were perhaps a bit too hasty to jump to git-2.10.  Existing LTS
releases of Ubuntu are quite old still: Trusty has 1.9 while Xenial
has 2.5.  While we plan on dropping support for those eventually as
we migrate to Python 3.6, we don't need to be so strict just yet on
the git versions.

We also want to disconnect the version the repo launcher requires
from the version the rest of the source tree requires.  The repo
launcher doesn't need as many features, and being flexible there
allows us more freedom to upgrade & rollback as needed.

So we'll allow git-1.7 again, but start warning on any users older
than git-1.9.  This aligns better with existing LTS releases, and
gives users a chance to start upgrading before we cut them off.

Change-Id: I140305dd8e42c9719c84e2aee0dc6a5c5b18da25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254573
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 00:28:03 +00:00
3645bd2420 docs: document git/python/Ubuntu/Debian release schedules
Going purely on upstream package release cycles doesn't tell the whole
story: a lot of people run LTS distros which will have older versions
of software we want to support.

Build out a table for us to quickly reference when making decisions as
to what versions of git/python we want to support, and when we can drop
them.  This will also help to refer users to as why we made a specific
decision that might be affecting them.

Change-Id: I7aea24bbefd50e358aeacf11e8c15a346c8fb8a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254572
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 00:27:59 +00:00
5f2b045195 sync: change how we preserve objects in shared repos
Some automatic git operations will prune objects on us, and not just
the gc step.  Normally we don't care, but with shared projects, we
will have multiple git checkouts with refs that the others cannot
see, but with a shared object dir.  Any pruning of objects based on
refs in just one repo can easily break the others.

git-2.7.0 introduced a preciousObjects setting which tells git to
never prune objects for this exact scenario: there might be refs in
some location that git is unable to see.

Change-Id: I781de27c5bbe1d4c70f0187566141c9cce088bd8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254392
Reviewed-by: Nasser Grainawi <nasser@codeaurora.org>
Reviewed-by: David Riley <davidriley@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-11 23:58:43 +00:00
163d42eb43 project: fix bytes/str encoding when updating git submodules
Since tempfile.mkstemp() returns a file handle in binary mode,
make sure we turn our strings into bytes before writing.

Bug: https://crbug.com/gerrit/12043
Change-Id: I3e84d595e84b8bc12a1fbc7fd0bb3ea0ba2832b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254393
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-11 18:49:47 +00:00
07392ed326 project: allow src=. with symlinks
Some Android/Nest manifests are using <linkfile> with src="." to
create stable paths to specific projects.  Allow that specific
use case as it seems reasonable to support.

Bug: https://crbug.com/gerrit/11218
Change-Id: I16dbe8d9fe42ea45440afcb61404c753bff1930d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254330
Reviewed-by: Chanho Park <parkch98@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-11 04:23:26 +00:00
3285e4b436 main: rework launcher version checking
The code has an ad-hoc check in that it requires the launcher major
version to not be less than the source code version.  We don't really
care about that requirement, and it doesn't fit with our other version
checks.  Rework it so we explicitly declare the min launcher version
that is supported.

We'll start with requiring repo launcher 1.15 which was released back
in 2012.  Hopefully no one has anything older than that, although it's
not clear we work with even newer versions than that :).  But let's be
a little conservative with the first update to this logic.

Bug: https://crbug.com/gerrit/10418
Change-Id: I611d70c60324d313c76874e978b8499a491a5d00
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254278
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-10 23:20:55 +00:00
ae62541005 manifest_xml: allow src=. with symlinks
Some Android/Nest manifests are using <linkfile> with src="." to
create stable paths to specific projects.  Allow that specific
use case as it seems reasonable to support.

Bug: https://crbug.com/gerrit/11218
Change-Id: I5eadec257cd58ba0f8687c590ddc250a7a414a85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254276
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-10 23:19:31 +00:00
83a3227b62 Fixing forall subcommand for Py3
Execution of 'repo forall -p -c' doesn't work with Py3 and ends up
with an error:

Got an error, terminating the pool: TypeError: can only concatenate
str (not "bytes") to str

That's fixed by using the decode() method.

Change-Id: Ice01aaa1822dde8d957b5bf096021dd5a2b7dd51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253659
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Jiri Tyr <jiri.tyr@gmail.com>
2020-02-10 10:52:27 +00:00
09dd9bda38 docs: document internal manifests.git/config settings
Change-Id: I6b32d925756375a9335522ff33376cb5f7ed1157
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254073
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-10 00:12:17 +00:00
f914edca53 project: unify HEAD path management
Add a helper function to unify the duplication of finding the full
path to the symbolic HEAD ref.  This makes it easy to handle git
worktrees where .git is a file rather than a dir/symlink.

Bug: https://crbug.com/gerrit/11486
Change-Id: I9f794f1295ad0d98c7c13622f01ded51e4ba7846
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254074
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-09 23:26:05 +00:00
e7c91889a6 remove spurious +x bits
These files are not directly executable, so drop the +x bits.

Change-Id: Iaf19a03a497686cc21103e7ddf08073173440dd1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254076
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-09 23:24:03 +00:00
1b117db767 find python via env
This allows these scripts to run through the active version of the
virtualenv python when invoked via tox.

Change-Id: Ib52f475b7b20c34d62cfd179a1341da1a08a8b5c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253974
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-09 04:02:45 +00:00
563f1a6512 repo: allow REPO_REV to be an env var
We do this for REPO_URL already.

Bug: https://crbug.com/gerrit/10233
Change-Id: I53410645474b00d900467c96fa5d8446f3a607d3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253552
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-09 04:02:26 +00:00
b4687ad862 docs: add a developer reference for .repo/ paths
Currently the only reference for these is the source which can be a
pita when needing to refer to something quickly.

Change-Id: I52baeb9a4935814cf99fa9a9b3102e8e46cddb0d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253972
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-09 03:55:47 +00:00
ded477dbb9 git_config: fix encoding handling in GetUrlCookieFile
Make sure we decode the bytes coming from the subprocess.Popen as
we're treating them as strings.

Change-Id: I44100ca5cd94f68a35d489936292eb641006edbe
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253973
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-08 06:26:36 +00:00
93293ca47f Fix inverted logic around [gitc-]init and -c
Instead of not using '-c' for '--current-branch' when using gitc, we
were only using '-c' when using gitc, so we still had the conflict with
the gitc option, and other users still couldn't use '-c'.

Test: repo init -u https://android.googlesource.com/platform/manifest; repo init -c
Test: repo gitc-init -u ... -b ... -c testing
Change-Id: I71e4950a49c281418249f0783c6a2ea34f0d3e2b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253795
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Dan Willemsen <dwillemsen@google.com>
2020-02-07 20:54:34 +00:00
dbd277ce50 [Win32] Make platform_utils compatible for Python3
On Python 3 several imports are to be imported from
different locations.

Signed-off-by: Remy Böhmer <linux@bohmer.net>
Change-Id: I4f243d145f65e38f74743a742583cfc5c5d76deb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/249610
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-06 14:29:15 +00:00
5a03308c5c sync: try to checkout repos across sync failures
Currently our default behavior is:
* Try to sync all repos
  * If any errors seen, exit
* Try to garbage collect all repos
  * If any errors seen, exit
* Try to update local project list
  * If any errors seen, exit
* Try to checkout out all local repos
  * If any errors seen, exit

Users find these incomplete syncs confusing, so lets try to complete
as much as possible by default and printing out summaries at the end.

Bug: https://crbug.com/gerrit/11293
Change-Id: Idd17cc9c3bbc574d8a0f08a30225dec7bfe414cb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238554
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-05 21:37:20 +00:00
3ba716f382 repo: try to reexec self with Python 3 as needed
We want to start warning about Python 2 usage, but we can't do it
simply because the shebang is /usr/bin/python which might be an old
version like python2.7.

We can't change the shebang because program name usage is spotty at
best: on some platforms (like macOS), it's not uncommon to not have
a `python3` wrapper, only a major.minor one like `python3.6`.  Using
python3 wouldn't guarantee a new enough version of Python 3 anyways,
and we don't want to require Python 3.6 exactly, just that minimum.

So we check the current Python version.  If it's older than the ver
of Python 3 we want, we search for a `python3.X` version to run.  If
those don't work, we see if `python3` exists and is a new enough ver.
If it's not, we die if the current Python 3 is too old, and we start
issuing warnings if the current Python version is 2.7.  This should
allow the user to take a bit more action by installing Python 3 on
their system without having to worry about changing /usr/bin/python.

Once we require Python 3 completely, we can simplify this logic a bit
by always bootstrapping up to Python 3 and failing with Python 2.

We have a few KI with Windows atm though, so keep it disabled there
until the fixes are merged.

Bug: https://crbug.com/gerrit/10418
Change-Id: I5e157defc788e31efb3e21e93f53fabdc7d75a3c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253136
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-05 21:32:02 +00:00
655aedd7f3 repo: raise min version of git
The git-2.10 series was released in 2016.  Since we're moving to
require Python 3.6 which was also released in 2016, bumping up the
git version seems reasonable.  Also we don't really test any git
versions close to as old as 1.7.2 which was released in 2010.

Change-Id: Ib71b714de6cd0b7dd50d0b300b108a560ee27331
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253134
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-05 19:01:40 +00:00
cc960971f4 sync: add option to skip manifest update
The use case is any situation where your manifest does
not exist on server, but where you still want to do
full sync for the projects, without having your
workspace manifest switched to other branch or
forwarded to latest or similar.
This allows syncing to a historical manifest in git log,
that does not have a branch, as well as when integrating
something together that has not been pushed upstream yet.
Changes can also exist locally on a manifest that is
behind head, meaning not requiring rebase to latest.

Tested using:
  $ cd .repo/manifests/
  $ git checkout <any hash 1>
  $ <do local modifications>
  $ repo sync --no-manifest-update
  $ git checkout <any hash 2>
  $ repo sync --no-manifest-update

Change-Id: I0c9773aa8bc5876813a2e7d7fec697abcb2d9e94
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/246445
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-05 18:57:58 +00:00
66098f707a init: handle -c conflicts with gitc-init
We keep getting requests for init to support -c.  This conflicts with
gitc-init which allocates -c for its own use.  Lets make this dynamic
so we keep it with "init" but omit it for "gitc-init".

Bug: https://crbug.com/gerrit/10200
Change-Id: Ibf69c2bbeff638e28e63cb08926fea0c622258db
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253252
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-05 16:00:10 +00:00
f7b64e3350 Do not try to fetch default revision for mirrors always
* Mirrors may contain multiple projects, some of which may not
  always contain the default revision.
* Only fetch the default revision explicitly if
  '--current-branch' is set.
* Fixes breakage casued by
  commit 6856f98467
  "Fix repo mirror with --current-branch"

Bug: https://crbug.com/gerrit/12274
Change-Id: Iaafabe2992f76f3644b841f24245d3e19c9515a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253093
Reviewed-by: Kuang-che Wu <kcwu@chromium.org>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Chirayu Desai <chirayudesai1@gmail.com>
2020-02-05 15:51:18 +00:00
bd0aae95f5 Add a way to override the remote using <extend-project>
This commit supports for the 'remote' attribute in
<extend-project>. This avoids the need to perform a <remove-project>
followed by a <project> in local manifests.

Change-Id: I9f9347913337ec9d159bc264d15ce97881ae5398
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253092
Tested-by: Kyunam Jo <kyunam.jo@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-04 22:42:28 +00:00
e6a202f790 project: add basic path checks for <copyfile> & <linkfile>
Reject paths in <copyfile> & <linkfile> that try to use symlinks or
non-file or non-dirs.

We don't fully validate <linkfile> when src is a glob as it's a bit
complicated -- any component in the src could be the glob.  We make
sure the destination is a directory, and that any paths in that dir
are created as symlinks.  So while this can be used to read any path,
it can't be abused to write to any paths.

Bug: https://crbug.com/gerrit/11218
Change-Id: I68b6d789b5ca4e43f569e75e8b293b3e13d3224b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233074
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
2020-02-04 20:34:23 +00:00
04122b7261 manifest: add basic path checks for <copyfile> & <linkfile>
Reject paths in <copyfile> & <linkfile> that point outside of their
respective scopes.  This validates paths while parsing the manifest
as this should be quick & cheap: we don't access the filesystem as
this code runs before we've synced.

Bug: https://crbug.com/gerrit/11218
Change-Id: I8e17bb91f3f5b905a9d76391b29fbab4cb77aa58
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232932
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
2020-02-04 20:34:01 +00:00
f5525fb310 repo: drop old signing key
This hasn't been used in many years to sign a release, so drop it
from the keyring to avoid confusing people.

Bug: https://crbug.com/gerrit/12229
Change-Id: Ifca7eee713d167c11f32252975724e5858e4c007
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253133
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-04 06:39:42 +00:00
ee451f035d repo: bump launcher version to 2.0
This reflects the transition to the new 2.x series which will be
migrating to Python 3-only.

Bug: https://crbug.com/gerrit/10418
Change-Id: I6355ac955d26b930f8a3721d3526eec5bed92400
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253132
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-04 06:15:26 +00:00
59 changed files with 1683 additions and 573 deletions

14
.flake8
View File

@ -1,3 +1,13 @@
[flake8]
max-line-length=80
ignore=E111,E114,E402
max-line-length=100
ignore=
# E111: Indentation is not a multiple of four
E111,
# E114: Indentation is not a multiple of four (comment)
E114,
# E402: Module level import not at top of file
E402,
# W503: Line break before binary operator
W503,
# W504: Line break after binary operator
W504

View File

@ -84,6 +84,7 @@ def _Color(fg=None, bg=None, attr=None):
code = ''
return code
DEFAULT = None

View File

@ -123,9 +123,9 @@ class Command(object):
project = None
if os.path.exists(path):
oldpath = None
while path and \
path != oldpath and \
path != manifest.topdir:
while (path and
path != oldpath and
path != manifest.topdir):
try:
project = self._by_path[path]
break
@ -236,6 +236,7 @@ class InteractiveCommand(Command):
"""Command which requires user interaction on the tty and
must not run within a pager, even if the user asks to.
"""
def WantPager(self, _opt):
return False
@ -244,6 +245,7 @@ class PagedCommand(Command):
"""Command which defaults to output in a pager, as its
display tends to be larger than one screen full.
"""
def WantPager(self, _opt):
return True

145
docs/internal-fs-layout.md Normal file
View File

@ -0,0 +1,145 @@
# Repo internal filesystem layout
A reference to the `.repo/` tree in repo client checkouts.
Hopefully it's complete & up-to-date, but who knows!
*** note
**Warning**:
This is meant for developers of the repo project itself as a quick reference.
**Nothing** in here must be construed as ABI, or that repo itself will never
change its internals in backwards incompatible ways.
***
[TOC]
## .repo/ layout
All content under `.repo/` is managed by `repo` itself with few exceptions.
In general, you should not make manual changes in here.
If a setting was initialized using an option to `repo init`, you should use that
command to change the setting later on.
It is always safe to re-run `repo init` in existing repo client checkouts.
For example, if you want to change the manifest branch, you can simply run
`repo init --manifest-branch=<new name>` and repo will take care of the rest.
### repo/ state
* `repo/`: A git checkout of the repo project. This is how `repo` re-execs
itself to get the latest released version.
It tracks the git repository at `REPO_URL` using the `REPO_REV` branch.
Those are specified at `repo init` time using the `--repo-url=<REPO_URL>`
and `--repo-branch=<REPO_REV>` options.
Any changes made to this directory will usually be automatically discarded
by repo itself when it checks for updates. If you want to update to the
latest version of repo, use `repo selfupdate` instead. If you want to
change the git URL/branch that this tracks, re-run `repo init` with the new
settings.
* `.repo_fetchtimes.json`: Used by `repo sync` to record stats when syncing
the various projects.
### Manifests
For more documentation on the manifest format, including the local_manifests
support, see the [manifest-format.md] file.
* `manifests/`: A git checkout of the manifest project. Its `.git/` state
points to the `manifest.git` bare checkout (see below). It tracks the git
branch specified at `repo init` time via `--manifest-branch`.
The local branch name is always `default` regardless of the remote tracking
branch. Do not get confused if the remote branch is not `default`, or if
there is a remote `default` that is completely different!
No manual changes should be made in here as it will just confuse repo and
it won't automatically recover causing no new changes to be picked up.
* `manifests.git/`: A bare checkout of the manifest project. It tracks the
git repository specified at `repo init` time via `--manifest-url`.
No manual changes should be made in here as it will just confuse repo.
If you want to switch the tracking settings, re-run `repo init` with the
new settings.
* `manifest.xml -> manifests/<manifest-name>.xml`: A symlink to the manifest
that the user wishes to sync. It is specified at `repo init` time via
`--manifest-name`.
Do not try to repoint this symlink to other files as it will confuse repo.
If you want to switch manifest files, re-run `repo init` with the new
setting.
* `manifests.git/.repo_config.json`: JSON cache of the `manifests.git/config`
file for repo to read/process quickly.
* `local_manifest.xml` (*Deprecated*): User-authored tweaks to the manifest
used to sync. See [local manifests] for more details.
* `local_manifests/`: Directory of user-authored manifest fragments to tweak
the manifest used to sync. See [local manifests] for more details.
### Project objects
* `project.list`: Tracking file used by `repo sync` to determine when projects
are added or removed and need corresponding updates in the checkout.
* `projects/`: Bare checkouts of every project synced by the manifest. The
filesystem layout matches the `<project path=...` setting in the manifest
(i.e. where it's checked out in the repo client source tree). Those
checkouts will symlink their `.git/` state to paths under here.
Some git state is further split out under `project-objects/`.
* `project-objects/`: Git objects that are safe to share across multiple
git checkouts. The filesystem layout matches the `<project name=...`
setting in the manifest (i.e. the path on the remote server). This allows
for multiple checkouts of the same remote git repo to share their objects.
For example, you could have different branches of `foo/bar.git` checked
out to `foo/bar-master`, `foo/bar-release`, etc... There will be multiple
trees under `projects/` for each one, but only one under `project-objects/`.
This can run into problems if different remotes use the same path on their
respective servers ...
* `subprojects/`: Like `projects/`, but for git submodules.
* `subproject-objects/`: Like `project-objects/`, but for git submodules.
### Settings
The `.repo/manifests.git/config` file is used to track settings for the entire
repo client checkout.
Most settings use the `[repo]` section to avoid conflicts with git.
User controlled settings are initialized when running `repo init`.
| Setting | `repo init` Option | Use/Meaning |
|-------------------|---------------------------|-------------|
| manifest.groups | `--groups` & `--platform` | The manifest groups to sync |
| repo.archive | `--archive` | Use `git archive` for checkouts |
| repo.clonefilter | `--clone-filter` | Filter setting when using [partial git clones] |
| repo.depth | `--depth` | Create shallow checkouts when cloning |
| repo.dissociate | `--dissociate` | Dissociate from any reference/mirrors after initial clone |
| repo.mirror | `--mirror` | Checkout is a repo mirror |
| repo.partialclone | `--partial-clone` | Create [partial git clones] |
| repo.reference | `--reference` | Reference repo client checkout |
| repo.submodules | `--submodules` | Sync git submodules |
| user.email | `--config-name` | User's e-mail address; Copied into `.git/config` when checking out a new project |
| user.name | `--config-name` | User's name; Copied into `.git/config` when checking out a new project |
[partial git clones]: https://git-scm.com/docs/gitrepository-layout#_code_partialclone_code
## ~/ dotconfig layout
Repo will create & maintain a few files in the user's home directory.
* `.repoconfig/`: Repo's per-user directory for all random config files/state.
* `.repoconfig/keyring-version`: Cache file for checking if the gnupg subdir
has all the same keys as the repo launcher. Used to avoid running gpg
constantly as that can be quite slow.
* `.repoconfig/gnupg/`: GnuPG's internal state directory used when repo needs
to run `gpg`. This provides isolation from the user's normal `~/.gnupg/`.
* `.repo_.gitconfig.json`: JSON cache of the `.gitconfig` file for repo to
read/process quickly.
[manifest-format.md]: ./manifest-format.md
[local manifests]: ./manifest-format.md#Local-Manifests

View File

@ -89,6 +89,7 @@ following DTD:
<!ATTLIST extend-project path CDATA #IMPLIED>
<!ATTLIST extend-project groups CDATA #IMPLIED>
<!ATTLIST extend-project revision CDATA #IMPLIED>
<!ATTLIST extend-project remote CDATA #IMPLIED>
<!ELEMENT remove-project EMPTY>
<!ATTLIST remove-project name CDATA #REQUIRED>
@ -306,6 +307,9 @@ belongs. Same syntax as the corresponding element of `project`.
Attribute `revision`: If specified, overrides the revision of the original
project. Same syntax as the corresponding element of `project`.
Attribute `remote`: If specified, overrides the remote of the original
project. Same syntax as the corresponding element of `project`.
### Element annotation
Zero or more annotation elements may be specified as children of a
@ -338,7 +342,7 @@ It's just like copyfile and runs at the same time as copyfile but
instead of copying it creates a symlink.
The symlink is created at "dest" (relative to the top of the tree) and
points to the path specified by "src".
points to the path specified by "src" which is a path in the project.
Parent directories of "dest" will be automatically created if missing.

View File

@ -161,7 +161,91 @@ You can create a short changelog using the command:
$ git log --format="%h (%aN) %s" --no-merges origin/stable..$r
```
## Project References
Here's a table showing the relationship of major tools, their EOL dates, and
their status in Ubuntu & Debian.
Those distros tend to be good indicators of how long we need to support things.
Things in bold indicate stuff to take note of, but does not guarantee that we
still support them.
Things in italics are things we used to care about but probably don't anymore.
| Date | EOL | [Git][rel-g] | [Python][rel-p] | [Ubuntu][rel-u] / [Debian][rel-d] | Git | Python |
|:--------:|:------------:|--------------|-----------------|-----------------------------------|-----|--------|
| Oct 2008 | *Oct 2013* | | 2.6.0 | *10.04 Lucid* - 10.10 Maverick / *Squeeze* |
| Dec 2008 | *Feb 2009* | | 3.0.0 |
| Feb 2009 | *Mar 2012* | | | Debian 5 Lenny | 1.5.6.5 | 2.5.2 |
| Jun 2009 | *Jun 2016* | | 3.1.0 | *10.04 Lucid* - 10.10 Maverick / *Squeeze* |
| Feb 2010 | *Oct 2012* | 1.7.0 | | *10.04 Lucid* - *12.04 Precise* - 12.10 Quantal |
| Apr 2010 | *Apr 2015* | | | *10.04 Lucid* | 1.7.0.4 | 2.6.5 3.1.2 |
| Jul 2010 | *Dec 2019* | | **2.7.0** | 11.04 Natty - **<current>** |
| Oct 2010 | | | | 10.10 Maverick | 1.7.1 | 2.6.6 3.1.3 |
| Feb 2011 | *Feb 2016* | | | Debian 6 Squeeze | 1.7.2.5 | 2.6.6 3.1.3 |
| Apr 2011 | | | | 11.04 Natty | 1.7.4 | 2.7.1 3.2.0 |
| Oct 2011 | *Feb 2016* | | 3.2.0 | 11.04 Natty - 12.10 Quantal |
| Oct 2011 | | | | 11.10 Ocelot | 1.7.5.4 | 2.7.2 3.2.2 |
| Apr 2012 | *Apr 2019* | | | *12.04 Precise* | 1.7.9.5 | 2.7.3 3.2.3 |
| Sep 2012 | *Sep 2017* | | 3.3.0 | 13.04 Raring - 13.10 Saucy |
| Oct 2012 | *Dec 2014* | 1.8.0 | | 13.04 Raring - 13.10 Saucy |
| Oct 2012 | | | | 12.10 Quantal | 1.7.10.4 | 2.7.3 3.2.3 |
| Apr 2013 | | | | 13.04 Raring | 1.8.1.2 | 2.7.4 3.3.1 |
| May 2013 | *May 2018* | | | Debian 7 Wheezy | 1.7.10.4 | 2.7.3 3.2.3 |
| Oct 2013 | | | | 13.10 Saucy | 1.8.3.2 | 2.7.5 3.3.2 |
| Feb 2014 | *Dec 2014* | **1.9.0** | | **14.04 Trusty** |
| Mar 2014 | *Mar 2019* | | **3.4.0** | **14.04 Trusty** - 15.10 Wily / **Jessie** |
| Apr 2014 | **Apr 2022** | | | **14.04 Trusty** | 1.9.1 | 2.7.5 3.4.0 |
| May 2014 | *Dec 2014* | 2.0.0 |
| Aug 2014 | *Dec 2014* | **2.1.0** | | 14.10 Utopic - 15.04 Vivid / **Jessie** |
| Oct 2014 | | | | 14.10 Utopic | 2.1.0 | 2.7.8 3.4.2 |
| Nov 2014 | *Sep 2015* | 2.2.0 |
| Feb 2015 | *Sep 2015* | 2.3.0 |
| Apr 2015 | *May 2017* | 2.4.0 |
| Apr 2015 | **Jun 2020** | | | **Debian 8 Jessie** | 2.1.4 | 2.7.9 3.4.2 |
| Apr 2015 | | | | 15.04 Vivid | 2.1.4 | 2.7.9 3.4.3 |
| Jul 2015 | *May 2017* | 2.5.0 | | 15.10 Wily |
| Sep 2015 | *May 2017* | 2.6.0 |
| Sep 2015 | **Sep 2020** | | **3.5.0** | **16.04 Xenial** - 17.04 Zesty / **Stretch** |
| Oct 2015 | | | | 15.10 Wily | 2.5.0 | 2.7.9 3.4.3 |
| Jan 2016 | *Jul 2017* | **2.7.0** | | **16.04 Xenial** |
| Mar 2016 | *Jul 2017* | 2.8.0 |
| Apr 2016 | **Apr 2024** | | | **16.04 Xenial** | 2.7.4 | 2.7.11 3.5.1 |
| Jun 2016 | *Jul 2017* | 2.9.0 | | 16.10 Yakkety |
| Sep 2016 | *Sep 2017* | 2.10.0 |
| Oct 2016 | | | | 16.10 Yakkety | 2.9.3 | 2.7.11 3.5.1 |
| Nov 2016 | *Sep 2017* | **2.11.0** | | 17.04 Zesty / **Stretch** |
| Dec 2016 | **Dec 2021** | | **3.6.0** | 17.10 Artful - **18.04 Bionic** - 18.10 Cosmic |
| Feb 2017 | *Sep 2017* | 2.12.0 |
| Apr 2017 | | | | 17.04 Zesty | 2.11.0 | 2.7.13 3.5.3 |
| May 2017 | *May 2018* | 2.13.0 |
| Jun 2017 | **Jun 2022** | | | **Debian 9 Stretch** | 2.11.0 | 2.7.13 3.5.3 |
| Aug 2017 | *Dec 2019* | 2.14.0 | | 17.10 Artful |
| Oct 2017 | *Dec 2019* | 2.15.0 |
| Oct 2017 | | | | 17.10 Artful | 2.14.1 | 2.7.14 3.6.3 |
| Jan 2018 | *Dec 2019* | 2.16.0 |
| Apr 2018 | *Dec 2019* | 2.17.0 | | **18.04 Bionic** |
| Apr 2018 | **Apr 2028** | | | **18.04 Bionic** | 2.17.0 | 2.7.15 3.6.5 |
| Jun 2018 | *Dec 2019* | 2.18.0 |
| Jun 2018 | **Jun 2023** | | 3.7.0 | 19.04 Disco - **20.04 Focal** / **Buster** |
| Sep 2018 | *Dec 2019* | 2.19.0 | | 18.10 Cosmic |
| Oct 2018 | | | | 18.10 Cosmic | 2.19.1 | 2.7.15 3.6.6 |
| Dec 2018 | *Dec 2019* | **2.20.0** | | 19.04 Disco / **Buster** |
| Feb 2019 | *Dec 2019* | 2.21.0 |
| Apr 2019 | | | | 19.04 Disco | 2.20.1 | 2.7.16 3.7.3 |
| Jun 2019 | | 2.22.0 |
| Jul 2019 | **Jul 2024** | | | **Debian 10 Buster** | 2.20.1 | 2.7.16 3.7.3 |
| Aug 2019 | | 2.23.0 |
| Oct 2019 | **Oct 2024** | | 3.8.0 |
| Oct 2019 | | | | 19.10 Eoan | 2.20.1 | 2.7.17 3.7.5 |
| Nov 2019 | | 2.24.0 |
| Jan 2020 | | 2.25.0 | | **20.04 Focal** |
| Apr 2020 | **Apr 2030** | | | **20.04 Focal** | 2.25.0 | 2.7.17 3.7.5 |
[rel-d]: https://en.wikipedia.org/wiki/Debian_version_history
[rel-g]: https://en.wikipedia.org/wiki/Git#Releases
[rel-p]: https://en.wikipedia.org/wiki/History_of_Python#Table_of_versions
[rel-u]: https://en.wikipedia.org/wiki/Ubuntu_version_history#Table_of_versions
[example announcement]: https://groups.google.com/d/topic/repo-discuss/UGBNismWo1M/discussion
[repo-discuss@googlegroups.com]: https://groups.google.com/forum/#!forum/repo-discuss
[go/repo-release]: https://goto.google.com/repo-release

View File

@ -24,6 +24,7 @@ import tempfile
from error import EditorError
import platform_utils
class Editor(object):
"""Manages the user's preferred text editor."""
@ -57,7 +58,7 @@ class Editor(object):
if os.getenv('TERM') == 'dumb':
print(
"""No editor specified in GIT_EDITOR, core.editor, VISUAL or EDITOR.
"""No editor specified in GIT_EDITOR, core.editor, VISUAL or EDITOR.
Tried to fall back to vi but terminal is dumb. Please configure at
least one of these before using this command.""", file=sys.stderr)
sys.exit(1)
@ -104,10 +105,10 @@ least one of these before using this command.""", file=sys.stderr)
rc = subprocess.Popen(args, shell=shell).wait()
except OSError as e:
raise EditorError('editor failed, %s: %s %s'
% (str(e), editor, path))
% (str(e), editor, path))
if rc != 0:
raise EditorError('editor failed with exit status %d: %s %s'
% (rc, editor, path))
% (rc, editor, path))
with open(path, mode='rb') as fd2:
return fd2.read().decode('utf-8')

View File

@ -14,17 +14,26 @@
# See the License for the specific language governing permissions and
# limitations under the License.
class ManifestParseError(Exception):
"""Failed to parse the manifest file.
"""
class ManifestInvalidRevisionError(Exception):
"""The revision value in a project is incorrect.
"""
class ManifestInvalidPathError(Exception):
"""A path used in <copyfile> or <linkfile> is incorrect.
"""
class NoManifestException(Exception):
"""The required manifest does not exist.
"""
def __init__(self, path, reason):
super(NoManifestException, self).__init__()
self.path = path
@ -33,9 +42,11 @@ class NoManifestException(Exception):
def __str__(self):
return self.reason
class EditorError(Exception):
"""Unspecified error from the user's text editor.
"""
def __init__(self, reason):
super(EditorError, self).__init__()
self.reason = reason
@ -43,9 +54,11 @@ class EditorError(Exception):
def __str__(self):
return self.reason
class GitError(Exception):
"""Unspecified internal error from git.
"""
def __init__(self, command):
super(GitError, self).__init__()
self.command = command
@ -53,9 +66,11 @@ class GitError(Exception):
def __str__(self):
return self.command
class UploadError(Exception):
"""A bundle upload to Gerrit did not succeed.
"""
def __init__(self, reason):
super(UploadError, self).__init__()
self.reason = reason
@ -63,9 +78,11 @@ class UploadError(Exception):
def __str__(self):
return self.reason
class DownloadError(Exception):
"""Cannot download a repository.
"""
def __init__(self, reason):
super(DownloadError, self).__init__()
self.reason = reason
@ -73,9 +90,11 @@ class DownloadError(Exception):
def __str__(self):
return self.reason
class NoSuchProjectError(Exception):
"""A specified project does not exist in the work tree.
"""
def __init__(self, name=None):
super(NoSuchProjectError, self).__init__()
self.name = name
@ -89,6 +108,7 @@ class NoSuchProjectError(Exception):
class InvalidProjectGroupsError(Exception):
"""A specified project is not suitable for the specified groups
"""
def __init__(self, name=None):
super(InvalidProjectGroupsError, self).__init__()
self.name = name
@ -98,15 +118,18 @@ class InvalidProjectGroupsError(Exception):
return 'in current directory'
return self.name
class RepoChangedException(Exception):
"""Thrown if 'repo sync' results in repo updating its internal
repo or manifest repositories. In this special case we must
use exec to re-execute repo with the new code and manifest.
"""
def __init__(self, extra_args=None):
super(RepoChangedException, self).__init__()
self.extra_args = extra_args or []
class HookError(Exception):
"""Thrown if a 'repo-hook' could not be run.

View File

@ -23,6 +23,7 @@ TASK_COMMAND = 'command'
TASK_SYNC_NETWORK = 'sync-network'
TASK_SYNC_LOCAL = 'sync-local'
class EventLog(object):
"""Event log that records events that occurred during a repo invocation.
@ -138,7 +139,7 @@ class EventLog(object):
Returns:
A dictionary of the event added to the log.
"""
event['status'] = self.GetStatusString(success)
event['status'] = self.GetStatusString(success)
event['finish_time'] = finish
return event
@ -165,6 +166,7 @@ class EventLog(object):
# An integer id that is unique across this invocation of the program.
_EVENT_ID = multiprocessing.Value('i', 1)
def _NextEventId():
"""Helper function for grabbing the next unique id.

View File

@ -28,7 +28,17 @@ from repo_trace import REPO_TRACE, IsTrace, Trace
from wrapper import Wrapper
GIT = 'git'
MIN_GIT_VERSION = (1, 5, 4)
# NB: These do not need to be kept in sync with the repo launcher script.
# These may be much newer as it allows the repo launcher to roll between
# different repo releases while source versions might require a newer git.
#
# The soft version is when we start warning users that the version is old and
# we'll be dropping support for it. We'll refuse to work with versions older
# than the hard version.
#
# git-1.7 is in (EOL) Ubuntu Precise. git-1.9 is in Ubuntu Trusty.
MIN_GIT_VERSION_SOFT = (1, 9, 1)
MIN_GIT_VERSION_HARD = (1, 7, 2)
GIT_DIR = 'GIT_DIR'
LAST_GITDIR = None
@ -38,6 +48,7 @@ _ssh_proxy_path = None
_ssh_sock_path = None
_ssh_clients = []
def ssh_sock(create=True):
global _ssh_sock_path
if _ssh_sock_path is None:
@ -47,27 +58,31 @@ def ssh_sock(create=True):
if not os.path.exists(tmp_dir):
tmp_dir = tempfile.gettempdir()
_ssh_sock_path = os.path.join(
tempfile.mkdtemp('', 'ssh-', tmp_dir),
'master-%r@%h:%p')
tempfile.mkdtemp('', 'ssh-', tmp_dir),
'master-%r@%h:%p')
return _ssh_sock_path
def _ssh_proxy():
global _ssh_proxy_path
if _ssh_proxy_path is None:
_ssh_proxy_path = os.path.join(
os.path.dirname(__file__),
'git_ssh')
os.path.dirname(__file__),
'git_ssh')
return _ssh_proxy_path
def _add_ssh_client(p):
_ssh_clients.append(p)
def _remove_ssh_client(p):
try:
_ssh_clients.remove(p)
except ValueError:
pass
def terminate_ssh_clients():
global _ssh_clients
for p in _ssh_clients:
@ -78,8 +93,10 @@ def terminate_ssh_clients():
pass
_ssh_clients = []
_git_version = None
class _GitCall(object):
def version_tuple(self):
global _git_version
@ -91,12 +108,15 @@ class _GitCall(object):
return _git_version
def __getattr__(self, name):
name = name.replace('_','-')
name = name.replace('_', '-')
def fun(*cmdv):
command = [name]
command.extend(cmdv)
return GitCommand(None, command).Wait() == 0
return fun
git = _GitCall()
@ -177,8 +197,10 @@ class UserAgent(object):
return self._git_ua
user_agent = UserAgent()
def git_require(min_version, fail=False, msg=''):
git_version = git.version_tuple()
if min_version <= git_version:
@ -191,21 +213,23 @@ def git_require(min_version, fail=False, msg=''):
sys.exit(1)
return False
def _setenv(env, name, value):
env[name] = value.encode()
class GitCommand(object):
def __init__(self,
project,
cmdv,
bare = False,
provide_stdin = False,
capture_stdout = False,
capture_stderr = False,
disable_editor = False,
ssh_proxy = False,
cwd = None,
gitdir = None):
bare=False,
provide_stdin=False,
capture_stdout=False,
capture_stderr=False,
disable_editor=False,
ssh_proxy=False,
cwd=None,
gitdir=None):
env = self._GetBasicEnv()
# If we are not capturing std* then need to print it.
@ -285,11 +309,11 @@ class GitCommand(object):
try:
p = subprocess.Popen(command,
cwd = cwd,
env = env,
stdin = stdin,
stdout = stdout,
stderr = stderr)
cwd=cwd,
env=env,
stdin=stdin,
stdout=stdout,
stderr=stderr)
except Exception as e:
raise GitError('%s: %s' % (command[1], e))

View File

@ -59,39 +59,45 @@ ID_RE = re.compile(r'^[0-9a-f]{40}$')
REVIEW_CACHE = dict()
def IsChange(rev):
return rev.startswith(R_CHANGES)
def IsId(rev):
return ID_RE.match(rev)
def IsTag(rev):
return rev.startswith(R_TAGS)
def IsImmutable(rev):
return IsChange(rev) or IsId(rev) or IsTag(rev)
def _key(name):
parts = name.split('.')
if len(parts) < 2:
return name.lower()
parts[ 0] = parts[ 0].lower()
parts[0] = parts[0].lower()
parts[-1] = parts[-1].lower()
return '.'.join(parts)
class GitConfig(object):
_ForUser = None
@classmethod
def ForUser(cls):
if cls._ForUser is None:
cls._ForUser = cls(configfile = os.path.expanduser('~/.gitconfig'))
cls._ForUser = cls(configfile=os.path.expanduser('~/.gitconfig'))
return cls._ForUser
@classmethod
def ForRepository(cls, gitdir, defaults=None):
return cls(configfile = os.path.join(gitdir, 'config'),
defaults = defaults)
return cls(configfile=os.path.join(gitdir, 'config'),
defaults=defaults)
def __init__(self, configfile, defaults=None, jsonFile=None):
self.file = configfile
@ -104,16 +110,16 @@ class GitConfig(object):
self._json = jsonFile
if self._json is None:
self._json = os.path.join(
os.path.dirname(self.file),
'.repo_' + os.path.basename(self.file) + '.json')
os.path.dirname(self.file),
'.repo_' + os.path.basename(self.file) + '.json')
def Has(self, name, include_defaults = True):
def Has(self, name, include_defaults=True):
"""Return true if this configuration file has the key.
"""
if _key(name) in self._cache:
return True
if include_defaults and self.defaults:
return self.defaults.Has(name, include_defaults = True)
return self.defaults.Has(name, include_defaults=True)
return False
def GetBoolean(self, name):
@ -142,7 +148,7 @@ class GitConfig(object):
v = self._cache[_key(name)]
except KeyError:
if self.defaults:
return self.defaults.GetString(name, all_keys = all_keys)
return self.defaults.GetString(name, all_keys=all_keys)
v = []
if not all_keys:
@ -153,7 +159,7 @@ class GitConfig(object):
r = []
r.extend(v)
if self.defaults:
r.extend(self.defaults.GetString(name, all_keys = True))
r.extend(self.defaults.GetString(name, all_keys=True))
return r
def SetString(self, name, value):
@ -217,7 +223,7 @@ class GitConfig(object):
"""
return self._sections.get(section, set())
def HasSection(self, section, subsection = ''):
def HasSection(self, section, subsection=''):
"""Does at least one key in section.subsection exist?
"""
try:
@ -268,8 +274,7 @@ class GitConfig(object):
def _ReadJson(self):
try:
if os.path.getmtime(self._json) \
<= os.path.getmtime(self.file):
if os.path.getmtime(self._json) <= os.path.getmtime(self.file):
platform_utils.remove(self._json)
return None
except OSError:
@ -323,8 +328,8 @@ class GitConfig(object):
p = GitCommand(None,
command,
capture_stdout = True,
capture_stderr = True)
capture_stdout=True,
capture_stderr=True)
if p.Wait() == 0:
return p.stdout
else:
@ -392,6 +397,7 @@ _master_keys = set()
_ssh_master = True
_master_keys_lock = None
def init_ssh():
"""Should be called once at the start of repo to init ssh master handling.
@ -401,6 +407,7 @@ def init_ssh():
assert _master_keys_lock is None, "Should only call init_ssh once"
_master_keys_lock = _threading.Lock()
def _open_ssh(host, port=None):
global _ssh_master
@ -421,17 +428,17 @@ def _open_ssh(host, port=None):
if key in _master_keys:
return True
if not _ssh_master \
or 'GIT_SSH' in os.environ \
or sys.platform in ('win32', 'cygwin'):
if (not _ssh_master
or 'GIT_SSH' in os.environ
or sys.platform in ('win32', 'cygwin')):
# failed earlier, or cygwin ssh can't do this
#
return False
# We will make two calls to ssh; this is the common part of both calls.
command_base = ['ssh',
'-o','ControlPath %s' % ssh_sock(),
host]
'-o', 'ControlPath %s' % ssh_sock(),
host]
if port is not None:
command_base[1:1] = ['-p', str(port)]
@ -439,13 +446,13 @@ def _open_ssh(host, port=None):
# ...but before actually starting a master, we'll double-check. This can
# be important because we can't tell that that 'git@myhost.com' is the same
# as 'myhost.com' where "User git" is setup in the user's ~/.ssh/config file.
check_command = command_base + ['-O','check']
check_command = command_base + ['-O', 'check']
try:
Trace(': %s', ' '.join(check_command))
check_process = subprocess.Popen(check_command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
check_process.communicate() # read output, but ignore it...
check_process.communicate() # read output, but ignore it...
isnt_running = check_process.wait()
if not isnt_running:
@ -458,16 +465,14 @@ def _open_ssh(host, port=None):
# to the log there.
pass
command = command_base[:1] + \
['-M', '-N'] + \
command_base[1:]
command = command_base[:1] + ['-M', '-N'] + command_base[1:]
try:
Trace(': %s', ' '.join(command))
p = subprocess.Popen(command)
except Exception as e:
_ssh_master = False
print('\nwarn: cannot enable ssh control master for %s:%s\n%s'
% (host,port, str(e)), file=sys.stderr)
% (host, port, str(e)), file=sys.stderr)
return False
time.sleep(1)
@ -481,6 +486,7 @@ def _open_ssh(host, port=None):
finally:
_master_keys_lock.release()
def close_ssh():
global _master_keys_lock
@ -505,15 +511,18 @@ def close_ssh():
# We're done with the lock, so we can delete it.
_master_keys_lock = None
URI_SCP = re.compile(r'^([^@:]*@?[^:/]{1,}):')
URI_ALL = re.compile(r'^([a-z][a-z+-]*)://([^@/]*@?[^/]*)/')
def GetSchemeFromUrl(url):
m = URI_ALL.match(url)
if m:
return m.group(1)
return None
@contextlib.contextmanager
def GetUrlCookieFile(url, quiet):
if url.startswith('persistent-'):
@ -528,7 +537,7 @@ def GetUrlCookieFile(url, quiet):
cookiefile = None
proxy = None
for line in p.stdout:
line = line.strip()
line = line.strip().decode('utf-8')
if line.startswith(cookieprefix):
cookiefile = os.path.expanduser(line[len(cookieprefix):])
if line.startswith(proxyprefix):
@ -540,7 +549,7 @@ def GetUrlCookieFile(url, quiet):
finally:
p.stdin.close()
if p.wait():
err_msg = p.stderr.read()
err_msg = p.stderr.read().decode('utf-8')
if ' -print_config' in err_msg:
pass # Persistent proxy doesn't support -print_config.
elif not quiet:
@ -554,6 +563,7 @@ def GetUrlCookieFile(url, quiet):
cookiefile = os.path.expanduser(cookiefile)
yield cookiefile, None
def _preconnect(url):
m = URI_ALL.match(url)
if m:
@ -574,9 +584,11 @@ def _preconnect(url):
return False
class Remote(object):
"""Configuration options related to a remote.
"""
def __init__(self, config, name):
self._config = config
self.name = name
@ -585,7 +597,7 @@ class Remote(object):
self.review = self._Get('review')
self.projectname = self._Get('projectname')
self.fetch = list(map(RefSpec.FromString,
self._Get('fetch', all_keys=True)))
self._Get('fetch', all_keys=True)))
self._review_url = None
def _InsteadOf(self):
@ -599,8 +611,8 @@ class Remote(object):
insteadOfList = globCfg.GetString(key, all_keys=True)
for insteadOf in insteadOfList:
if self.url.startswith(insteadOf) \
and len(insteadOf) > len(longest):
if (self.url.startswith(insteadOf)
and len(insteadOf) > len(longest)):
longest = insteadOf
longestUrl = url
@ -731,12 +743,13 @@ class Remote(object):
def _Get(self, key, all_keys=False):
key = 'remote.%s.%s' % (self.name, key)
return self._config.GetString(key, all_keys = all_keys)
return self._config.GetString(key, all_keys=all_keys)
class Branch(object):
"""Configuration options related to a single branch.
"""
def __init__(self, config, name):
self._config = config
self.name = name
@ -780,4 +793,4 @@ class Branch(object):
def _Get(self, key, all_keys=False):
key = 'branch.%s.%s' % (self.name, key)
return self._config.GetString(key, all_keys = all_keys)
return self._config.GetString(key, all_keys=all_keys)

View File

@ -18,12 +18,12 @@ import os
from repo_trace import Trace
import platform_utils
HEAD = 'HEAD'
HEAD = 'HEAD'
R_CHANGES = 'refs/changes/'
R_HEADS = 'refs/heads/'
R_TAGS = 'refs/tags/'
R_PUB = 'refs/published/'
R_M = 'refs/remotes/m/'
R_HEADS = 'refs/heads/'
R_TAGS = 'refs/tags/'
R_PUB = 'refs/published/'
R_M = 'refs/remotes/m/'
class GitRefs(object):

View File

@ -29,12 +29,15 @@ from error import ManifestParseError
NUM_BATCH_RETRIEVE_REVISIONID = 32
def get_gitc_manifest_dir():
return wrapper.Wrapper().get_gitc_manifest_dir()
def parse_clientdir(gitc_fs_path):
return wrapper.Wrapper().gitc_parse_clientdir(gitc_fs_path)
def _set_project_revisions(projects):
"""Sets the revisionExpr for a list of projects.
@ -52,7 +55,7 @@ def _set_project_revisions(projects):
project.remote.url,
project.revisionExpr],
capture_stdout=True, cwd='/tmp'))
for project in projects if not git_config.IsId(project.revisionExpr)]
for project in projects if not git_config.IsId(project.revisionExpr)]
for proj, gitcmd in project_gitcmds:
if gitcmd.Wait():
print('FATAL: Failed to retrieve revisionExpr for %s' % proj)
@ -63,6 +66,7 @@ def _set_project_revisions(projects):
(proj.remote.url, proj.revisionExpr))
proj.revisionExpr = revisionExpr
def _manifest_groups(manifest):
"""Returns the manifest group string that should be synced
@ -77,6 +81,7 @@ def _manifest_groups(manifest):
groups = 'default,platform-' + platform.system().lower()
return groups
def generate_gitc_manifest(gitc_manifest, manifest, paths=None):
"""Generate a manifest for shafsd to use for this GITC client.
@ -104,11 +109,11 @@ def generate_gitc_manifest(gitc_manifest, manifest, paths=None):
if not proj.upstream and not git_config.IsId(proj.revisionExpr):
proj.upstream = proj.revisionExpr
if not path in gitc_manifest.paths:
if path not in gitc_manifest.paths:
# Any new projects need their first revision, even if we weren't asked
# for them.
projects.append(proj)
elif not path in paths:
elif path not in paths:
# And copy revisions from the previous manifest if we're not updating
# them now.
gitc_proj = gitc_manifest.paths[path]
@ -121,7 +126,7 @@ def generate_gitc_manifest(gitc_manifest, manifest, paths=None):
index = 0
while index < len(projects):
_set_project_revisions(
projects[index:(index+NUM_BATCH_RETRIEVE_REVISIONID)])
projects[index:(index + NUM_BATCH_RETRIEVE_REVISIONID)])
index += NUM_BATCH_RETRIEVE_REVISIONID
if gitc_manifest is not None:
@ -140,6 +145,7 @@ def generate_gitc_manifest(gitc_manifest, manifest, paths=None):
# Save the manifest.
save_manifest(manifest)
def save_manifest(manifest, client_dir=None):
"""Save the manifest file in the client_dir.

71
main.py
View File

@ -47,7 +47,7 @@ except ImportError:
from color import SetDefaultColoring
import event_log
from repo_trace import SetTrace
from git_command import git, GitCommand, user_agent
from git_command import user_agent
from git_config import init_ssh, close_ssh
from command import InteractiveCommand
from command import MirrorSafeCommand
@ -101,6 +101,7 @@ global_options.add_option('--event-log',
dest='event_log', action='store',
help='filename of event log to append timeline to')
class _Repo(object):
def __init__(self, repodir):
self.repodir = repodir
@ -188,7 +189,7 @@ class _Repo(object):
copts = cmd.ReadEnvironmentOptions(copts)
except NoManifestException as e:
print('error: in `%s`: %s' % (' '.join([name] + argv), str(e)),
file=sys.stderr)
file=sys.stderr)
print('error: manifest missing or unreadable -- please run init',
file=sys.stderr)
return 1
@ -211,9 +212,9 @@ class _Repo(object):
cmd.ValidateOptions(copts, cargs)
result = cmd.Execute(copts, cargs)
except (DownloadError, ManifestInvalidRevisionError,
NoManifestException) as e:
NoManifestException) as e:
print('error: in `%s`: %s' % (' '.join([name] + argv), str(e)),
file=sys.stderr)
file=sys.stderr)
if isinstance(e, NoManifestException):
print('error: manifest missing or unreadable -- please run init',
file=sys.stderr)
@ -255,27 +256,41 @@ class _Repo(object):
return result
def _CheckWrapperVersion(ver, repo_path):
def _CheckWrapperVersion(ver_str, repo_path):
"""Verify the repo launcher is new enough for this checkout.
Args:
ver_str: The version string passed from the repo launcher when it ran us.
repo_path: The path to the repo launcher that loaded us.
"""
# Refuse to work with really old wrapper versions. We don't test these,
# so might as well require a somewhat recent sane version.
# v1.15 of the repo launcher was released in ~Mar 2012.
MIN_REPO_VERSION = (1, 15)
min_str = '.'.join(str(x) for x in MIN_REPO_VERSION)
if not repo_path:
repo_path = '~/bin/repo'
if not ver:
if not ver_str:
print('no --wrapper-version argument', file=sys.stderr)
sys.exit(1)
# Pull out the version of the repo launcher we know about to compare.
exp = Wrapper().VERSION
ver = tuple(map(int, ver.split('.')))
if len(ver) == 1:
ver = (0, ver[0])
ver = tuple(map(int, ver_str.split('.')))
exp_str = '.'.join(map(str, exp))
if exp[0] > ver[0] or ver < (0, 4):
if ver < MIN_REPO_VERSION:
print("""
!!! A new repo command (%5s) is available. !!!
!!! You must upgrade before you can continue: !!!
repo: error:
!!! Your version of repo %s is too old.
!!! We need at least version %s.
!!! A new repo command (%s) is available.
!!! You must upgrade before you can continue:
cp %s %s
""" % (exp_str, WrapperPath(), repo_path), file=sys.stderr)
""" % (ver_str, min_str, exp_str, WrapperPath(), repo_path), file=sys.stderr)
sys.exit(1)
if exp > ver:
@ -286,11 +301,13 @@ def _CheckWrapperVersion(ver, repo_path):
cp %s %s
""" % (exp_str, WrapperPath(), repo_path), file=sys.stderr)
def _CheckRepoDir(repo_dir):
if not repo_dir:
print('no --repo-dir argument', file=sys.stderr)
sys.exit(1)
def _PruneOptions(argv, opt):
i = 0
while i < len(argv):
@ -306,6 +323,7 @@ def _PruneOptions(argv, opt):
continue
i += 1
class _UserAgentHandler(urllib.request.BaseHandler):
def http_request(self, req):
req.add_header('User-Agent', user_agent.repo)
@ -315,6 +333,7 @@ class _UserAgentHandler(urllib.request.BaseHandler):
req.add_header('User-Agent', user_agent.repo)
return req
def _AddPasswordFromUserInput(handler, msg, req):
# If repo could not find auth info from netrc, try to get it from user input
url = req.get_full_url()
@ -328,22 +347,24 @@ def _AddPasswordFromUserInput(handler, msg, req):
return
handler.passwd.add_password(None, url, user, password)
class _BasicAuthHandler(urllib.request.HTTPBasicAuthHandler):
def http_error_401(self, req, fp, code, msg, headers):
_AddPasswordFromUserInput(self, msg, req)
return urllib.request.HTTPBasicAuthHandler.http_error_401(
self, req, fp, code, msg, headers)
self, req, fp, code, msg, headers)
def http_error_auth_reqed(self, authreq, host, req, headers):
try:
old_add_header = req.add_header
def _add_header(name, val):
val = val.replace('\n', '')
old_add_header(name, val)
req.add_header = _add_header
return urllib.request.AbstractBasicAuthHandler.http_error_auth_reqed(
self, authreq, host, req, headers)
except:
self, authreq, host, req, headers)
except Exception:
reset = getattr(self, 'reset_retry_count', None)
if reset is not None:
reset()
@ -351,22 +372,24 @@ class _BasicAuthHandler(urllib.request.HTTPBasicAuthHandler):
self.retried = 0
raise
class _DigestAuthHandler(urllib.request.HTTPDigestAuthHandler):
def http_error_401(self, req, fp, code, msg, headers):
_AddPasswordFromUserInput(self, msg, req)
return urllib.request.HTTPDigestAuthHandler.http_error_401(
self, req, fp, code, msg, headers)
self, req, fp, code, msg, headers)
def http_error_auth_reqed(self, auth_header, host, req, headers):
try:
old_add_header = req.add_header
def _add_header(name, val):
val = val.replace('\n', '')
old_add_header(name, val)
req.add_header = _add_header
return urllib.request.AbstractDigestAuthHandler.http_error_auth_reqed(
self, auth_header, host, req, headers)
except:
self, auth_header, host, req, headers)
except Exception:
reset = getattr(self, 'reset_retry_count', None)
if reset is not None:
reset()
@ -374,6 +397,7 @@ class _DigestAuthHandler(urllib.request.HTTPDigestAuthHandler):
self.retried = 0
raise
class _KerberosAuthHandler(urllib.request.BaseHandler):
def __init__(self):
self.retried = 0
@ -392,7 +416,7 @@ class _KerberosAuthHandler(urllib.request.BaseHandler):
if self.retried > 3:
raise urllib.request.HTTPError(req.get_full_url(), 401,
"Negotiate auth failed", headers, None)
"Negotiate auth failed", headers, None)
else:
self.retried += 1
@ -408,7 +432,7 @@ class _KerberosAuthHandler(urllib.request.BaseHandler):
return response
except kerberos.GSSError:
return None
except:
except Exception:
self.reset_retry_count()
raise
finally:
@ -454,6 +478,7 @@ class _KerberosAuthHandler(urllib.request.BaseHandler):
kerberos.authGSSClientClean(self.context)
self.context = None
def init_http():
handlers = [_UserAgentHandler()]
@ -462,7 +487,7 @@ def init_http():
n = netrc.netrc()
for host in n.hosts:
p = n.hosts[host]
mgr.add_password(p[1], 'http://%s/' % host, p[0], p[2])
mgr.add_password(p[1], 'http://%s/' % host, p[0], p[2])
mgr.add_password(p[1], 'https://%s/' % host, p[0], p[2])
except netrc.NetrcParseError:
pass
@ -481,6 +506,7 @@ def init_http():
handlers.append(urllib.request.HTTPSHandler(debuglevel=1))
urllib.request.install_opener(urllib.request.build_opener(*handlers))
def _Main(argv):
result = 0
@ -537,5 +563,6 @@ def _Main(argv):
TerminatePager()
sys.exit(result)
if __name__ == '__main__':
_Main(sys.argv[1:])

View File

@ -35,7 +35,8 @@ from git_config import GitConfig
from git_refs import R_HEADS, HEAD
import platform_utils
from project import RemoteSpec, Project, MetaProject
from error import ManifestParseError, ManifestInvalidRevisionError
from error import (ManifestParseError, ManifestInvalidPathError,
ManifestInvalidRevisionError)
MANIFEST_FILE_NAME = 'manifest.xml'
LOCAL_MANIFEST_NAME = 'local_manifest.xml'
@ -55,6 +56,7 @@ urllib.parse.uses_netloc.extend([
'sso',
'rpc'])
class _Default(object):
"""Project defaults within the manifest."""
@ -73,6 +75,7 @@ class _Default(object):
def __ne__(self, other):
return self.__dict__ != other.__dict__
class _XmlRemote(object):
def __init__(self,
name,
@ -126,6 +129,7 @@ class _XmlRemote(object):
orig_name=self.name,
fetchUrl=self.fetchUrl)
class XmlManifest(object):
"""manages the repo configuration file"""
@ -139,12 +143,12 @@ class XmlManifest(object):
self._load_local_manifests = True
self.repoProject = MetaProject(self, 'repo',
gitdir = os.path.join(repodir, 'repo/.git'),
worktree = os.path.join(repodir, 'repo'))
gitdir=os.path.join(repodir, 'repo/.git'),
worktree=os.path.join(repodir, 'repo'))
self.manifestProject = MetaProject(self, 'manifests',
gitdir = os.path.join(repodir, 'manifests.git'),
worktree = os.path.join(repodir, 'manifests'))
gitdir=os.path.join(repodir, 'manifests.git'),
worktree=os.path.join(repodir, 'manifests'))
self._Unload()
@ -223,7 +227,7 @@ class XmlManifest(object):
if self.notice:
notice_element = root.appendChild(doc.createElement('notice'))
notice_lines = self.notice.splitlines()
indented_notice = ('\n'.join(" "*4 + line for line in notice_lines))[4:]
indented_notice = ('\n'.join(" " * 4 + line for line in notice_lines))[4:]
notice_element.appendChild(doc.createTextNode(indented_notice))
d = self.default
@ -461,12 +465,12 @@ class XmlManifest(object):
self.localManifestWarning = True
print('warning: %s is deprecated; put local manifests '
'in `%s` instead' % (LOCAL_MANIFEST_NAME,
os.path.join(self.repodir, LOCAL_MANIFESTS_DIR_NAME)),
os.path.join(self.repodir, LOCAL_MANIFESTS_DIR_NAME)),
file=sys.stderr)
nodes.append(self._ParseManifestXml(local, self.repodir))
local_dir = os.path.abspath(os.path.join(self.repodir,
LOCAL_MANIFESTS_DIR_NAME))
LOCAL_MANIFESTS_DIR_NAME))
try:
for local_file in sorted(platform_utils.listdir(local_dir)):
if local_file.endswith('.xml'):
@ -511,7 +515,7 @@ class XmlManifest(object):
fp = os.path.join(include_root, name)
if not os.path.isfile(fp):
raise ManifestParseError("include %s doesn't exist or isn't a file"
% (name,))
% (name,))
try:
nodes.extend(self._ParseManifestXml(fp, include_root))
# should isolate this to the exact exception, but that's
@ -598,6 +602,9 @@ class XmlManifest(object):
if groups:
groups = self._ParseGroups(groups)
revision = node.getAttribute('revision')
remote = node.getAttribute('remote')
if remote:
remote = self._get_remote(node)
for p in self._projects[name]:
if path and p.relpath != path:
@ -606,6 +613,8 @@ class XmlManifest(object):
p.groups.extend(groups)
if revision:
p.revisionExpr = revision
if remote:
p.remote = remote.ToRemoteSpec(name)
if node.nodeName == 'repo-hooks':
# Get the name of the project and the (space-separated) list of enabled.
repo_hooks_project = self._reqatt(node, 'in-project')
@ -649,7 +658,6 @@ class XmlManifest(object):
if self._repo_hooks_project and (self._repo_hooks_project.name == name):
self._repo_hooks_project = None
def _AddMetaProjectMirror(self, m):
name = None
m_url = m.GetRemote(m.remote.name).url
@ -676,15 +684,15 @@ class XmlManifest(object):
if name not in self._projects:
m.PreSync()
gitdir = os.path.join(self.topdir, '%s.git' % name)
project = Project(manifest = self,
name = name,
remote = remote.ToRemoteSpec(name),
gitdir = gitdir,
objdir = gitdir,
worktree = None,
relpath = name or None,
revisionExpr = m.revisionExpr,
revisionId = None)
project = Project(manifest=self,
name=name,
remote=remote.ToRemoteSpec(name),
gitdir=gitdir,
objdir=gitdir,
worktree=None,
relpath=name or None,
revisionExpr=m.revisionExpr,
revisionId=None)
self._projects[project.name] = [project]
self._paths[project.relpath] = project
@ -792,7 +800,7 @@ class XmlManifest(object):
def _UnjoinName(self, parent_name, name):
return os.path.relpath(name, parent_name)
def _ParseProject(self, node, parent = None, **extra_proj_attrs):
def _ParseProject(self, node, parent=None, **extra_proj_attrs):
"""
reads a <project> element from the manifest file
"""
@ -805,21 +813,21 @@ class XmlManifest(object):
remote = self._default.remote
if remote is None:
raise ManifestParseError("no remote for project %s within %s" %
(name, self.manifestFile))
(name, self.manifestFile))
revisionExpr = node.getAttribute('revision') or remote.revision
if not revisionExpr:
revisionExpr = self._default.revisionExpr
if not revisionExpr:
raise ManifestParseError("no revision for project %s within %s" %
(name, self.manifestFile))
(name, self.manifestFile))
path = node.getAttribute('path')
if not path:
path = name
if path.startswith('/'):
raise ManifestParseError("project %s path cannot be absolute in %s" %
(name, self.manifestFile))
(name, self.manifestFile))
rebase = node.getAttribute('rebase')
if not rebase:
@ -849,7 +857,7 @@ class XmlManifest(object):
if clone_depth:
try:
clone_depth = int(clone_depth)
if clone_depth <= 0:
if clone_depth <= 0:
raise ValueError()
except ValueError:
raise ManifestParseError('invalid clone-depth %s in %s' %
@ -877,24 +885,24 @@ class XmlManifest(object):
if node.getAttribute('force-path').lower() in ("yes", "true", "1"):
gitdir = os.path.join(self.topdir, '%s.git' % path)
project = Project(manifest = self,
name = name,
remote = remote.ToRemoteSpec(name),
gitdir = gitdir,
objdir = objdir,
worktree = worktree,
relpath = relpath,
revisionExpr = revisionExpr,
revisionId = None,
rebase = rebase,
groups = groups,
sync_c = sync_c,
sync_s = sync_s,
sync_tags = sync_tags,
clone_depth = clone_depth,
upstream = upstream,
parent = parent,
dest_branch = dest_branch,
project = Project(manifest=self,
name=name,
remote=remote.ToRemoteSpec(name),
gitdir=gitdir,
objdir=objdir,
worktree=worktree,
relpath=relpath,
revisionExpr=revisionExpr,
revisionId=None,
rebase=rebase,
groups=groups,
sync_c=sync_c,
sync_s=sync_s,
sync_tags=sync_tags,
clone_depth=clone_depth,
upstream=upstream,
parent=parent,
dest_branch=dest_branch,
**extra_proj_attrs)
for n in node.childNodes:
@ -905,7 +913,7 @@ class XmlManifest(object):
if n.nodeName == 'annotation':
self._ParseAnnotation(project, n)
if n.nodeName == 'project':
project.subprojects.append(self._ParseProject(n, parent = project))
project.subprojects.append(self._ParseProject(n, parent=project))
return project
@ -943,21 +951,101 @@ class XmlManifest(object):
worktree = os.path.join(parent.worktree, path).replace('\\', '/')
return relpath, worktree, gitdir, objdir
@staticmethod
def _CheckLocalPath(path, symlink=False):
"""Verify |path| is reasonable for use in <copyfile> & <linkfile>."""
if '~' in path:
return '~ not allowed (due to 8.3 filenames on Windows filesystems)'
# Some filesystems (like Apple's HFS+) try to normalize Unicode codepoints
# which means there are alternative names for ".git". Reject paths with
# these in it as there shouldn't be any reasonable need for them here.
# The set of codepoints here was cribbed from jgit's implementation:
# https://eclipse.googlesource.com/jgit/jgit/+/9110037e3e9461ff4dac22fee84ef3694ed57648/org.eclipse.jgit/src/org/eclipse/jgit/lib/ObjectChecker.java#884
BAD_CODEPOINTS = {
u'\u200C', # ZERO WIDTH NON-JOINER
u'\u200D', # ZERO WIDTH JOINER
u'\u200E', # LEFT-TO-RIGHT MARK
u'\u200F', # RIGHT-TO-LEFT MARK
u'\u202A', # LEFT-TO-RIGHT EMBEDDING
u'\u202B', # RIGHT-TO-LEFT EMBEDDING
u'\u202C', # POP DIRECTIONAL FORMATTING
u'\u202D', # LEFT-TO-RIGHT OVERRIDE
u'\u202E', # RIGHT-TO-LEFT OVERRIDE
u'\u206A', # INHIBIT SYMMETRIC SWAPPING
u'\u206B', # ACTIVATE SYMMETRIC SWAPPING
u'\u206C', # INHIBIT ARABIC FORM SHAPING
u'\u206D', # ACTIVATE ARABIC FORM SHAPING
u'\u206E', # NATIONAL DIGIT SHAPES
u'\u206F', # NOMINAL DIGIT SHAPES
u'\uFEFF', # ZERO WIDTH NO-BREAK SPACE
}
if BAD_CODEPOINTS & set(path):
# This message is more expansive than reality, but should be fine.
return 'Unicode combining characters not allowed'
# Assume paths might be used on case-insensitive filesystems.
path = path.lower()
# Some people use src="." to create stable links to projects. Lets allow
# that but reject all other uses of "." to keep things simple.
parts = path.split(os.path.sep)
if parts != ['.']:
for part in set(parts):
if part in {'.', '..', '.git'} or part.startswith('.repo'):
return 'bad component: %s' % (part,)
if not symlink and path.endswith(os.path.sep):
return 'dirs not allowed'
norm = os.path.normpath(path)
if norm == '..' or norm.startswith('../') or norm.startswith(os.path.sep):
return 'path cannot be outside'
@classmethod
def _ValidateFilePaths(cls, element, src, dest):
"""Verify |src| & |dest| are reasonable for <copyfile> & <linkfile>.
We verify the path independent of any filesystem state as we won't have a
checkout available to compare to. i.e. This is for parsing validation
purposes only.
We'll do full/live sanity checking before we do the actual filesystem
modifications in _CopyFile/_LinkFile/etc...
"""
# |dest| is the file we write to or symlink we create.
# It is relative to the top of the repo client checkout.
msg = cls._CheckLocalPath(dest)
if msg:
raise ManifestInvalidPathError(
'<%s> invalid "dest": %s: %s' % (element, dest, msg))
# |src| is the file we read from or path we point to for symlinks.
# It is relative to the top of the git project checkout.
msg = cls._CheckLocalPath(src, symlink=element == 'linkfile')
if msg:
raise ManifestInvalidPathError(
'<%s> invalid "src": %s: %s' % (element, src, msg))
def _ParseCopyFile(self, project, node):
src = self._reqatt(node, 'src')
dest = self._reqatt(node, 'dest')
if not self.IsMirror:
# src is project relative;
# dest is relative to the top of the tree
project.AddCopyFile(src, dest, os.path.join(self.topdir, dest))
# dest is relative to the top of the tree.
# We only validate paths if we actually plan to process them.
self._ValidateFilePaths('copyfile', src, dest)
project.AddCopyFile(src, dest, self.topdir)
def _ParseLinkFile(self, project, node):
src = self._reqatt(node, 'src')
dest = self._reqatt(node, 'dest')
if not self.IsMirror:
# src is project relative;
# dest is relative to the top of the tree
project.AddLinkFile(src, dest, os.path.join(self.topdir, dest))
# dest is relative to the top of the tree.
# We only validate paths if we actually plan to process them.
self._ValidateFilePaths('linkfile', src, dest)
project.AddLinkFile(src, dest, self.topdir)
def _ParseAnnotation(self, project, node):
name = self._reqatt(node, 'name')
@ -968,7 +1056,7 @@ class XmlManifest(object):
keep = "true"
if keep != "true" and keep != "false":
raise ManifestParseError('optional "keep" attribute must be '
'"true" or "false"')
'"true" or "false"')
project.AddAnnotation(name, value, keep)
def _get_remote(self, node):
@ -979,7 +1067,7 @@ class XmlManifest(object):
v = self._remotes.get(name)
if not v:
raise ManifestParseError("remote %s not defined in %s" %
(name, self.manifestFile))
(name, self.manifestFile))
return v
def _reqatt(self, node, attname):
@ -989,7 +1077,7 @@ class XmlManifest(object):
v = node.getAttribute(attname)
if not v:
raise ManifestParseError("no %s in <%s> within %s" %
(attname, node.nodeName, self.manifestFile))
(attname, node.nodeName, self.manifestFile))
return v
def projectsDiff(self, manifest):
@ -1007,7 +1095,7 @@ class XmlManifest(object):
diff = {'added': [], 'removed': [], 'changed': [], 'unreachable': []}
for proj in fromKeys:
if not proj in toKeys:
if proj not in toKeys:
diff['removed'].append(fromProjects[proj])
else:
fromProj = fromProjects[proj]
@ -1039,7 +1127,7 @@ class GitcManifest(XmlManifest):
gitc_client_name)
self.manifestFile = os.path.join(self.gitc_client_dir, '.manifest')
def _ParseProject(self, node, parent = None):
def _ParseProject(self, node, parent=None):
"""Override _ParseProject and add support for GITC specific attributes."""
return super(GitcManifest, self)._ParseProject(
node, parent=parent, old_revision=node.getAttribute('old-revision'))
@ -1048,4 +1136,3 @@ class GitcManifest(XmlManifest):
"""Output GITC Specific Project attributes"""
if p.old_revision:
e.setAttribute('old-revision', str(p.old_revision))

10
pager.py Executable file → Normal file
View File

@ -27,6 +27,7 @@ pager_process = None
old_stdout = None
old_stderr = None
def RunPager(globalConfig):
if not os.isatty(0) or not os.isatty(1):
return
@ -35,23 +36,25 @@ def RunPager(globalConfig):
return
if platform_utils.isWindows():
_PipePager(pager);
_PipePager(pager)
else:
_ForkPager(pager)
def TerminatePager():
global pager_process, old_stdout, old_stderr
if pager_process:
sys.stdout.flush()
sys.stderr.flush()
pager_process.stdin.close()
pager_process.wait();
pager_process.wait()
pager_process = None
# Restore initial stdout/err in case there is more output in this process
# after shutting down the pager process
sys.stdout = old_stdout
sys.stderr = old_stderr
def _PipePager(pager):
global pager_process, old_stdout, old_stderr
assert pager_process is None, "Only one active pager process at a time"
@ -62,6 +65,7 @@ def _PipePager(pager):
sys.stdout = pager_process.stdin
sys.stderr = pager_process.stdin
def _ForkPager(pager):
global active
# This process turns into the pager; a child it forks will
@ -88,6 +92,7 @@ def _ForkPager(pager):
print("fatal: cannot start pager '%s'" % pager, file=sys.stderr)
sys.exit(255)
def _SelectPager(globalConfig):
try:
return os.environ['GIT_PAGER']
@ -105,6 +110,7 @@ def _SelectPager(globalConfig):
return 'less'
def _BecomePager(pager):
# Delaying execution of the pager until we have output
# ready works around a long-standing bug in popularly

View File

@ -92,6 +92,7 @@ class _FileDescriptorStreamsNonBlocking(FileDescriptorStreams):
"""
class Stream(object):
""" Encapsulates a file descriptor """
def __init__(self, fd, dest, std_name):
self.fd = fd
self.dest = dest
@ -125,6 +126,7 @@ class _FileDescriptorStreamsThreads(FileDescriptorStreams):
non blocking I/O. This implementation requires creating threads issuing
blocking read operations on file descriptors.
"""
def __init__(self):
super(_FileDescriptorStreamsThreads, self).__init__()
# The queue is shared accross all threads so we can simulate the
@ -144,12 +146,14 @@ class _FileDescriptorStreamsThreads(FileDescriptorStreams):
class QueueItem(object):
""" Item put in the shared queue """
def __init__(self, stream, data):
self.stream = stream
self.data = data
class Stream(object):
""" Encapsulates a file descriptor """
def __init__(self, fd, dest, std_name, queue):
self.fd = fd
self.dest = dest

View File

@ -19,13 +19,18 @@ import errno
from pyversion import is_python3
from ctypes import WinDLL, get_last_error, FormatError, WinError, addressof
from ctypes import c_buffer
from ctypes.wintypes import BOOL, BOOLEAN, LPCWSTR, DWORD, HANDLE, POINTER, c_ubyte
from ctypes.wintypes import WCHAR, USHORT, LPVOID, Structure, Union, ULONG
from ctypes.wintypes import byref
from ctypes.wintypes import BOOL, BOOLEAN, LPCWSTR, DWORD, HANDLE
from ctypes.wintypes import WCHAR, USHORT, LPVOID, ULONG
if is_python3():
from ctypes import c_ubyte, Structure, Union, byref
from ctypes.wintypes import LPDWORD
else:
# For legacy Python2 different imports are needed.
from ctypes.wintypes import POINTER, c_ubyte, Structure, Union, byref
LPDWORD = POINTER(DWORD)
kernel32 = WinDLL('kernel32', use_last_error=True)
LPDWORD = POINTER(DWORD)
UCHAR = c_ubyte
# Win32 error codes

View File

@ -26,6 +26,7 @@ _NOT_TTY = not os.isatty(2)
# column 0.
CSI_ERASE_LINE = '\x1b[2K'
class Progress(object):
def __init__(self, title, total=0, units='', print_newline=False,
always_print_percentage=False):
@ -53,9 +54,9 @@ class Progress(object):
if self._total <= 0:
sys.stderr.write('%s\r%s: %d,' % (
CSI_ERASE_LINE,
self._title,
self._done))
CSI_ERASE_LINE,
self._title,
self._done))
sys.stderr.flush()
else:
p = (100 * self._done) / self._total
@ -63,13 +64,13 @@ class Progress(object):
if self._lastp != p or self._always_print_percentage:
self._lastp = p
sys.stderr.write('%s\r%s: %3d%% (%d%s/%d%s)%s%s%s' % (
CSI_ERASE_LINE,
self._title,
p,
self._done, self._units,
self._total, self._units,
' ' if msg else '', msg,
"\n" if self._print_newline else ""))
CSI_ERASE_LINE,
self._title,
p,
self._done, self._units,
self._total, self._units,
' ' if msg else '', msg,
"\n" if self._print_newline else ""))
sys.stderr.flush()
def end(self):
@ -78,16 +79,16 @@ class Progress(object):
if self._total <= 0:
sys.stderr.write('%s\r%s: %d, done.\n' % (
CSI_ERASE_LINE,
self._title,
self._done))
CSI_ERASE_LINE,
self._title,
self._done))
sys.stderr.flush()
else:
p = (100 * self._done) / self._total
sys.stderr.write('%s\r%s: %3d%% (%d%s/%d%s), done.\n' % (
CSI_ERASE_LINE,
self._title,
p,
self._done, self._units,
self._total, self._units))
CSI_ERASE_LINE,
self._title,
p,
self._done, self._units,
self._total, self._units))
sys.stderr.flush()

226
project.py Executable file → Normal file
View File

@ -36,7 +36,7 @@ from git_command import GitCommand, git_require
from git_config import GitConfig, IsId, GetSchemeFromUrl, GetUrlCookieFile, \
ID_RE
from error import GitError, HookError, UploadError, DownloadError
from error import ManifestInvalidRevisionError
from error import ManifestInvalidRevisionError, ManifestInvalidPathError
from error import NoManifestException
import platform_utils
import progress
@ -85,6 +85,7 @@ def not_rev(r):
def sq(r):
return "'" + r.replace("'", "'\''") + "'"
_project_hook_list = None
@ -261,17 +262,70 @@ class _Annotation(object):
self.keep = keep
class _CopyFile(object):
def _SafeExpandPath(base, subpath, skipfinal=False):
"""Make sure |subpath| is completely safe under |base|.
def __init__(self, src, dest, abssrc, absdest):
We make sure no intermediate symlinks are traversed, and that the final path
is not a special file (e.g. not a socket or fifo).
NB: We rely on a number of paths already being filtered out while parsing the
manifest. See the validation logic in manifest_xml.py for more details.
"""
components = subpath.split(os.path.sep)
if skipfinal:
# Whether the caller handles the final component itself.
finalpart = components.pop()
path = base
for part in components:
if part in {'.', '..'}:
raise ManifestInvalidPathError(
'%s: "%s" not allowed in paths' % (subpath, part))
path = os.path.join(path, part)
if platform_utils.islink(path):
raise ManifestInvalidPathError(
'%s: traversing symlinks not allow' % (path,))
if os.path.exists(path):
if not os.path.isfile(path) and not platform_utils.isdir(path):
raise ManifestInvalidPathError(
'%s: only regular files & directories allowed' % (path,))
if skipfinal:
path = os.path.join(path, finalpart)
return path
class _CopyFile(object):
"""Container for <copyfile> manifest element."""
def __init__(self, git_worktree, src, topdir, dest):
"""Register a <copyfile> request.
Args:
git_worktree: Absolute path to the git project checkout.
src: Relative path under |git_worktree| of file to read.
topdir: Absolute path to the top of the repo client checkout.
dest: Relative path under |topdir| of file to write.
"""
self.git_worktree = git_worktree
self.topdir = topdir
self.src = src
self.dest = dest
self.abs_src = abssrc
self.abs_dest = absdest
def _Copy(self):
src = self.abs_src
dest = self.abs_dest
src = _SafeExpandPath(self.git_worktree, self.src)
dest = _SafeExpandPath(self.topdir, self.dest)
if platform_utils.isdir(src):
raise ManifestInvalidPathError(
'%s: copying from directory not supported' % (self.src,))
if platform_utils.isdir(dest):
raise ManifestInvalidPathError(
'%s: copying to directory not allowed' % (self.dest,))
# copy file if it does not exist or is out of date
if not os.path.exists(dest) or not filecmp.cmp(src, dest):
try:
@ -292,13 +346,21 @@ class _CopyFile(object):
class _LinkFile(object):
"""Container for <linkfile> manifest element."""
def __init__(self, git_worktree, src, dest, relsrc, absdest):
def __init__(self, git_worktree, src, topdir, dest):
"""Register a <linkfile> request.
Args:
git_worktree: Absolute path to the git project checkout.
src: Target of symlink relative to path under |git_worktree|.
topdir: Absolute path to the top of the repo client checkout.
dest: Relative path under |topdir| of symlink to create.
"""
self.git_worktree = git_worktree
self.topdir = topdir
self.src = src
self.dest = dest
self.src_rel_to_dest = relsrc
self.abs_dest = absdest
def __linkIt(self, relSrc, absDest):
# link file if it does not exist or is out of date
@ -316,35 +378,42 @@ class _LinkFile(object):
_error('Cannot link file %s to %s', relSrc, absDest)
def _Link(self):
"""Link the self.rel_src_to_dest and self.abs_dest. Handles wild cards
on the src linking all of the files in the source in to the destination
directory.
"""Link the self.src & self.dest paths.
Handles wild cards on the src linking all of the files in the source in to
the destination directory.
"""
# We use the absSrc to handle the situation where the current directory
# is not the root of the repo
absSrc = os.path.join(self.git_worktree, self.src)
if os.path.exists(absSrc):
# Entity exists so just a simple one to one link operation
self.__linkIt(self.src_rel_to_dest, self.abs_dest)
# Some people use src="." to create stable links to projects. Lets allow
# that but reject all other uses of "." to keep things simple.
if self.src == '.':
src = self.git_worktree
else:
src = _SafeExpandPath(self.git_worktree, self.src)
if os.path.exists(src):
# Entity exists so just a simple one to one link operation.
dest = _SafeExpandPath(self.topdir, self.dest, skipfinal=True)
# dest & src are absolute paths at this point. Make sure the target of
# the symlink is relative in the context of the repo client checkout.
relpath = os.path.relpath(src, os.path.dirname(dest))
self.__linkIt(relpath, dest)
else:
dest = _SafeExpandPath(self.topdir, self.dest)
# Entity doesn't exist assume there is a wild card
absDestDir = self.abs_dest
if os.path.exists(absDestDir) and not platform_utils.isdir(absDestDir):
_error('Link error: src with wildcard, %s must be a directory',
absDestDir)
if os.path.exists(dest) and not platform_utils.isdir(dest):
_error('Link error: src with wildcard, %s must be a directory', dest)
else:
absSrcFiles = glob.glob(absSrc)
for absSrcFile in absSrcFiles:
for absSrcFile in glob.glob(src):
# Create a releative path from source dir to destination dir
absSrcDir = os.path.dirname(absSrcFile)
relSrcDir = os.path.relpath(absSrcDir, absDestDir)
relSrcDir = os.path.relpath(absSrcDir, dest)
# Get the source file name
srcFile = os.path.basename(absSrcFile)
# Now form the final full paths to srcFile. They will be
# absolute for the desintaiton and relative for the srouce.
absDest = os.path.join(absDestDir, srcFile)
absDest = os.path.join(dest, srcFile)
relSrc = os.path.join(relSrcDir, srcFile)
self.__linkIt(relSrc, absDest)
@ -1188,9 +1257,7 @@ class Project(object):
print(line[:-1])
return p.Wait() == 0
# Publish / Upload ##
def WasPublished(self, branch, all_refs=None):
"""Was the branch published (uploaded) for code review?
If so, returns the SHA-1 hash of the last published
@ -1342,9 +1409,7 @@ class Project(object):
R_HEADS + branch.name,
message=msg)
# Sync ##
def _ExtractArchive(self, tarpath, path=None):
"""Extract the given tar on its current location
@ -1712,18 +1777,25 @@ class Project(object):
if submodules:
syncbuf.later1(self, _dosubmodules)
def AddCopyFile(self, src, dest, absdest):
# dest should already be an absolute path, but src is project relative
# make src an absolute path
abssrc = os.path.join(self.worktree, src)
self.copyfiles.append(_CopyFile(src, dest, abssrc, absdest))
def AddCopyFile(self, src, dest, topdir):
"""Mark |src| for copying to |dest| (relative to |topdir|).
def AddLinkFile(self, src, dest, absdest):
# dest should already be an absolute path, but src is project relative
# make src relative path to dest
absdestdir = os.path.dirname(absdest)
relsrc = os.path.relpath(os.path.join(self.worktree, src), absdestdir)
self.linkfiles.append(_LinkFile(self.worktree, src, dest, relsrc, absdest))
No filesystem changes occur here. Actual copying happens later on.
Paths should have basic validation run on them before being queued.
Further checking will be handled when the actual copy happens.
"""
self.copyfiles.append(_CopyFile(self.worktree, src, topdir, dest))
def AddLinkFile(self, src, dest, topdir):
"""Mark |dest| to create a symlink (relative to |topdir|) pointing to |src|.
No filesystem changes occur here. Actual linking happens later on.
Paths should have basic validation run on them before being queued.
Further checking will be handled when the actual link happens.
"""
self.linkfiles.append(_LinkFile(self.worktree, src, topdir, dest))
def AddAnnotation(self, name, value, keep):
self.annotations.append(_Annotation(name, value, keep))
@ -1744,8 +1816,18 @@ class Project(object):
patch_id,
self.bare_git.rev_parse('FETCH_HEAD'))
# Branch Management ##
def GetHeadPath(self):
"""Return the full path to the HEAD ref."""
dotgit = os.path.join(self.worktree, '.git')
if os.path.isfile(dotgit):
# Git worktrees use a "gitdir:" syntax to point to the scratch space.
with open(dotgit) as fp:
setting = fp.read()
assert setting.startswith('gitdir:')
gitdir = setting.split(':', 1)[1].strip()
dotgit = os.path.join(self.worktree, gitdir)
return os.path.join(dotgit, HEAD)
def StartBranch(self, name, branch_merge='', revision=None):
"""Create a new branch off the manifest's revision.
@ -1786,8 +1868,7 @@ class Project(object):
except OSError:
pass
_lwrite(ref, '%s\n' % revid)
_lwrite(os.path.join(self.worktree, '.git', HEAD),
'ref: %s%s\n' % (R_HEADS, name))
_lwrite(self.GetHeadPath(), 'ref: %s%s\n' % (R_HEADS, name))
branch.Save()
return True
@ -1834,8 +1915,7 @@ class Project(object):
# Same revision; just update HEAD to point to the new
# target branch, but otherwise take no other action.
#
_lwrite(os.path.join(self.worktree, '.git', HEAD),
'ref: %s%s\n' % (R_HEADS, name))
_lwrite(self.GetHeadPath(), 'ref: %s%s\n' % (R_HEADS, name))
return True
return GitCommand(self,
@ -1868,8 +1948,7 @@ class Project(object):
revid = self.GetRevisionId(all_refs)
if head == revid:
_lwrite(os.path.join(self.worktree, '.git', HEAD),
'%s\n' % revid)
_lwrite(self.GetHeadPath(), '%s\n' % revid)
else:
self._Checkout(revid, quiet=True)
@ -1935,9 +2014,7 @@ class Project(object):
kept.append(ReviewableBranch(self, branch, base))
return kept
# Submodule Management ##
def GetRegisteredSubprojects(self):
result = []
@ -1989,7 +2066,7 @@ class Project(object):
gitmodules_lines = []
fd, temp_gitmodules_path = tempfile.mkstemp()
try:
os.write(fd, p.stdout)
os.write(fd, p.stdout.encode('utf-8'))
os.close(fd)
cmd = ['config', '--file', temp_gitmodules_path, '--list']
p = GitCommand(None, cmd, capture_stdout=True, capture_stderr=True,
@ -2088,7 +2165,6 @@ class Project(object):
result.extend(subproject.GetDerivedSubprojects())
return result
# Direct Git Commands ##
def _CheckForImmutableRevision(self):
try:
@ -2252,9 +2328,12 @@ class Project(object):
spec.append('tag')
spec.append(tag_name)
branch = self.revisionExpr
if self.manifest.IsMirror and not current_branch_only:
branch = None
else:
branch = self.revisionExpr
if (not self.manifest.IsMirror and is_sha1 and depth
and git_require((1, 8, 3))):
and git_require((1, 8, 3))):
# Shallow checkout of a specific commit, fetch from that commit and not
# the heads only as the commit might be deeper in the history.
spec.append(branch)
@ -2395,7 +2474,7 @@ class Project(object):
platform_utils.remove(tmpPath)
with GetUrlCookieFile(srcUrl, quiet) as (cookiefile, proxy):
if cookiefile:
cmd += ['--cookie', cookiefile, '--cookie-jar', cookiefile]
cmd += ['--cookie', cookiefile]
if proxy:
cmd += ['--proxy', proxy]
elif 'http_proxy' in os.environ and 'darwin' == sys.platform:
@ -2540,7 +2619,7 @@ class Project(object):
(self.worktree)):
platform_utils.rmtree(platform_utils.realpath(self.worktree))
return self._InitGitDir(mirror_git=mirror_git, force_sync=False)
except:
except Exception:
raise e
raise e
@ -2673,9 +2752,31 @@ class Project(object):
symlink_dirs += self.working_tree_dirs
to_symlink = symlink_files + symlink_dirs
for name in set(to_symlink):
dst = platform_utils.realpath(os.path.join(destdir, name))
# Try to self-heal a bit in simple cases.
dst_path = os.path.join(destdir, name)
src_path = os.path.join(srcdir, name)
if name in self.working_tree_dirs:
# If the dir is missing under .repo/projects/, create it.
if not os.path.exists(src_path):
os.makedirs(src_path)
elif name in self.working_tree_files:
# If it's a file under the checkout .git/ and the .repo/projects/ has
# nothing, move the file under the .repo/projects/ tree.
if not os.path.exists(src_path) and os.path.isfile(dst_path):
platform_utils.rename(dst_path, src_path)
# If the path exists under the .repo/projects/ and there's no symlink
# under the checkout .git/, recreate the symlink.
if name in self.working_tree_dirs or name in self.working_tree_files:
if os.path.exists(src_path) and not os.path.exists(dst_path):
platform_utils.symlink(
os.path.relpath(src_path, os.path.dirname(dst_path)), dst_path)
dst = platform_utils.realpath(dst_path)
if os.path.lexists(dst):
src = platform_utils.realpath(os.path.join(srcdir, name))
src = platform_utils.realpath(src_path)
# Fail if the links are pointing to the wrong place
if src != dst:
_error('%s is different in %s vs %s', name, destdir, srcdir)
@ -2763,7 +2864,7 @@ class Project(object):
try:
platform_utils.rmtree(dotgit)
return self._InitWorkTree(force_sync=False, submodules=submodules)
except:
except Exception:
raise e
raise e
@ -2929,7 +3030,7 @@ class Project(object):
if self._bare:
path = os.path.join(self._project.gitdir, HEAD)
else:
path = os.path.join(self._project.worktree, '.git', HEAD)
path = self._project.GetHeadPath()
try:
with open(path) as fd:
line = fd.readline()
@ -3024,9 +3125,6 @@ class Project(object):
raise TypeError('%s() got an unexpected keyword argument %r'
% (name, k))
if config is not None:
if not git_require((1, 7, 2)):
raise ValueError('cannot set config on command line for %s()'
% name)
for k, v in config.items():
cmdv.append('-c')
cmdv.append('%s=%s' % (k, v))

View File

@ -16,5 +16,6 @@
import sys
def is_python3():
return sys.version_info[0] == 3

357
repo
View File

@ -10,13 +10,94 @@ copy of repo in the checkout.
from __future__ import print_function
import datetime
import os
import platform
import subprocess
import sys
def exec_command(cmd):
"""Execute |cmd| or return None on failure."""
try:
if platform.system() == 'Windows':
ret = subprocess.call(cmd)
sys.exit(ret)
else:
os.execvp(cmd[0], cmd)
except Exception:
pass
def check_python_version():
"""Make sure the active Python version is recent enough."""
def reexec(prog):
exec_command([prog] + sys.argv)
MIN_PYTHON_VERSION = (3, 6)
ver = sys.version_info
major = ver.major
minor = ver.minor
# Abort on very old Python 2 versions.
if (major, minor) < (2, 7):
print('repo: error: Your Python version is too old. '
'Please use Python {}.{} or newer instead.'.format(
*MIN_PYTHON_VERSION), file=sys.stderr)
sys.exit(1)
# Try to re-exec the version specific Python 3 if needed.
if (major, minor) < MIN_PYTHON_VERSION:
# Python makes releases ~once a year, so try our min version +10 to help
# bridge the gap. This is the fallback anyways so perf isn't critical.
min_major, min_minor = MIN_PYTHON_VERSION
for inc in range(0, 10):
reexec('python{}.{}'.format(min_major, min_minor + inc))
# Try the generic Python 3 wrapper, but only if it's new enough. We don't
# want to go from (still supported) Python 2.7 to (unsupported) Python 3.5.
try:
proc = subprocess.Popen(
['python3', '-c', 'import sys; '
'print(sys.version_info.major, sys.version_info.minor)'],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(output, _) = proc.communicate()
python3_ver = tuple(int(x) for x in output.decode('utf-8').split())
except (OSError, subprocess.CalledProcessError):
python3_ver = None
# The python3 version looks like it's new enough, so give it a try.
if python3_ver and python3_ver >= MIN_PYTHON_VERSION:
reexec('python3')
# We're still here, so diagnose things for the user.
if major < 3:
print('repo: warning: Python 2 is no longer supported; '
'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION),
file=sys.stderr)
else:
print('repo: error: Python 3 version is too old; '
'Please use Python {}.{} or newer.'.format(*MIN_PYTHON_VERSION),
file=sys.stderr)
sys.exit(1)
if __name__ == '__main__':
# TODO(vapier): Enable this on Windows once we have Python 3 issues fixed.
if platform.system() != 'Windows':
check_python_version()
# repo default configuration
#
import os
REPO_URL = os.environ.get('REPO_URL', None)
if not REPO_URL:
REPO_URL = 'https://gerrit.googlesource.com/git-repo'
REPO_REV = 'stable'
REPO_REV = os.environ.get('REPO_REV')
if not REPO_REV:
REPO_REV = 'stable'
# Copyright (C) 2008 Google Inc.
#
@ -33,10 +114,10 @@ REPO_REV = 'stable'
# limitations under the License.
# increment this whenever we make important changes to this script
VERSION = (1, 26)
VERSION = (2, 3)
# increment this if the MAINTAINER_KEYS block is modified
KEYRING_VERSION = (1, 2)
KEYRING_VERSION = (2, 0)
# Each individual key entry is created by using:
# gpg --armor --export keyid
@ -82,48 +163,20 @@ HTHs37+/QLMomGEGKZMWi0dShU2J5mNRQu3Hhxl3hHDVbt5CeJBb26aQcQrFz69W
zE3GNvmJosh6leayjtI9P2A6iEkEGBECAAkFAkj3uiACGwwACgkQFlMNXpIPXGWp
TACbBS+Up3RpfYVfd63c1cDdlru13pQAn3NQy/SN858MkxN+zym86UBgOad2
=CMiZ
-----END PGP PUBLIC KEY BLOCK-----
Conley Owens <cco3@android.com>
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.11 (GNU/Linux)
mQENBFHRvc8BCADFg45Xx/y6QDC+T7Y/gGc7vx0ww7qfOwIKlAZ9xG3qKunMxo+S
hPCnzEl3cq+6I1Ww/ndop/HB3N3toPXRCoN8Vs4/Hc7by+SnaLFnacrm+tV5/OgT
V37Lzt8lhay1Kl+YfpFwHYYpIEBLFV9knyfRXS/428W2qhdzYfvB15/AasRmwmor
py4NIzSs8UD/SPr1ihqNCdZM76+MQyN5HMYXW/ALZXUFG0pwluHFA7hrfPG74i8C
zMiP7qvMWIl/r/jtzHioH1dRKgbod+LZsrDJ8mBaqsZaDmNJMhss9g76XvfMyLra
9DI9/iFuBpGzeqBv0hwOGQspLRrEoyTeR6n1ABEBAAG0H0NvbmxleSBPd2VucyA8
Y2NvM0BhbmRyb2lkLmNvbT6JATgEEwECACIFAlHRvc8CGwMGCwkIBwMCBhUIAgkK
CwQWAgMBAh4BAheAAAoJEGe35EhpKzgsP6AIAJKJmNtn4l7hkYHKHFSo3egb6RjQ
zEIP3MFTcu8HFX1kF1ZFbrp7xqurLaE53kEkKuAAvjJDAgI8mcZHP1JyplubqjQA
xvv84gK+OGP3Xk+QK1ZjUQSbjOpjEiSZpRhWcHci3dgOUH4blJfByHw25hlgHowd
a/2PrNKZVcJ92YienaxxGjcXEUcd0uYEG2+rwllQigFcnMFDhr9B71MfalRHjFKE
fmdoypqLrri61YBc59P88Rw2/WUpTQjgNubSqa3A2+CKdaRyaRw+2fdF4TdR0h8W
zbg+lbaPtJHsV+3mJC7fq26MiJDRJa5ZztpMn8su20gbLgi2ShBOaHAYDDi5AQ0E
UdG9zwEIAMoOBq+QLNozAhxOOl5GL3StTStGRgPRXINfmViTsihrqGCWBBUfXlUE
OytC0mYcrDUQev/8ToVoyqw+iGSwDkcSXkrEUCKFtHV/GECWtk1keyHgR10YKI1R
mquSXoubWGqPeG1PAI74XWaRx8UrL8uCXUtmD8Q5J7mDjKR5NpxaXrwlA0bKsf2E
Gp9tu1kKauuToZhWHMRMqYSOGikQJwWSFYKT1KdNcOXLQF6+bfoJ6sjVYdwfmNQL
Ixn8QVhoTDedcqClSWB17VDEFDFa7MmqXZz2qtM3X1R/MUMHqPtegQzBGNhRdnI2
V45+1Nnx/uuCxDbeI4RbHzujnxDiq70AEQEAAYkBHwQYAQIACQUCUdG9zwIbDAAK
CRBnt+RIaSs4LNVeB/0Y2pZ8I7gAAcEM0Xw8drr4omg2fUoK1J33ozlA/RxeA/lJ
I3KnyCDTpXuIeBKPGkdL8uMATC9Z8DnBBajRlftNDVZS3Hz4G09G9QpMojvJkFJV
By+01Flw/X+eeN8NpqSuLV4W+AjEO8at/VvgKr1AFvBRdZ7GkpI1o6DgPe7ZqX+1
dzQZt3e13W0rVBb/bUgx9iSLoeWP3aq/k+/GRGOR+S6F6BBSl0SQ2EF2+dIywb1x
JuinEP+AwLAUZ1Bsx9ISC0Agpk2VeHXPL3FGhroEmoMvBzO0kTFGyoeT7PR/BfKv
+H/g3HsL2LOB9uoIm8/5p2TTU5ttYCXMHhQZ81AY
=AUp4
-----END PGP PUBLIC KEY BLOCK-----
"""
GIT = 'git' # our git command
# NB: The version of git that the repo launcher requires may be much older than
# the version of git that the main repo source tree requires. Keeping this at
# an older version also makes it easier for users to upgrade/rollback as needed.
#
# git-1.7 is in (EOL) Ubuntu Precise.
MIN_GIT_VERSION = (1, 7, 2) # minimum supported git version
repodir = '.repo' # name of repo's private directory
S_repo = 'repo' # special repo repository
S_manifests = 'manifests' # special manifest repository
REPO_MAIN = S_repo + '/main.py' # main script
MIN_PYTHON_VERSION = (2, 7) # minimum supported python version
GITC_CONFIG_FILE = '/gitc/.config'
GITC_FS_ROOT_DIR = '/gitc/manifest-rw/'
@ -131,12 +184,9 @@ GITC_FS_ROOT_DIR = '/gitc/manifest-rw/'
import collections
import errno
import optparse
import platform
import re
import shutil
import stat
import subprocess
import sys
if sys.version_info[0] == 3:
import urllib.request
@ -149,105 +199,79 @@ else:
urllib.error = urllib2
# Python version check
ver = sys.version_info
if (ver[0], ver[1]) < MIN_PYTHON_VERSION:
print('error: Python version {} unsupported.\n'
'Please use Python {}.{} instead.'.format(
sys.version.split(' ')[0],
MIN_PYTHON_VERSION[0],
MIN_PYTHON_VERSION[1],
), file=sys.stderr)
sys.exit(1)
home_dot_repo = os.path.expanduser('~/.repoconfig')
gpg_dir = os.path.join(home_dot_repo, 'gnupg')
extra_args = []
init_optparse = optparse.OptionParser(usage="repo init -u url [options]")
# Logging
group = init_optparse.add_option_group('Logging options')
group.add_option('-q', '--quiet',
dest="quiet", action="store_true", default=False,
help="be quiet")
def _InitParser():
"""Setup the init subcommand parser."""
# Logging.
group = init_optparse.add_option_group('Logging options')
group.add_option('-q', '--quiet',
action='store_true', default=False,
help='be quiet')
# Manifest
group = init_optparse.add_option_group('Manifest options')
group.add_option('-u', '--manifest-url',
dest='manifest_url',
help='manifest repository location', metavar='URL')
group.add_option('-b', '--manifest-branch',
dest='manifest_branch',
help='manifest branch or revision', metavar='REVISION')
group.add_option('-m', '--manifest-name',
dest='manifest_name',
help='initial manifest file', metavar='NAME.xml')
group.add_option('--current-branch',
dest='current_branch_only', action='store_true',
help='fetch only current manifest branch from server')
group.add_option('--mirror',
dest='mirror', action='store_true',
help='create a replica of the remote repositories '
'rather than a client working directory')
group.add_option('--reference',
dest='reference',
help='location of mirror directory', metavar='DIR')
group.add_option('--dissociate',
dest='dissociate', action='store_true',
help='dissociate from reference mirrors after clone')
group.add_option('--depth', type='int', default=None,
dest='depth',
help='create a shallow clone with given depth; see git clone')
group.add_option('--partial-clone', action='store_true',
dest='partial_clone',
help='perform partial clone (https://git-scm.com/'
'docs/gitrepository-layout#_code_partialclone_code)')
group.add_option('--clone-filter', action='store', default='blob:none',
dest='clone_filter',
help='filter for use with --partial-clone [default: %default]')
group.add_option('--archive',
dest='archive', action='store_true',
help='checkout an archive instead of a git repository for '
'each project. See git archive.')
group.add_option('--submodules',
dest='submodules', action='store_true',
help='sync any submodules associated with the manifest repo')
group.add_option('-g', '--groups',
dest='groups', default='default',
help='restrict manifest projects to ones with specified '
'group(s) [default|all|G1,G2,G3|G4,-G5,-G6]',
metavar='GROUP')
group.add_option('-p', '--platform',
dest='platform', default="auto",
help='restrict manifest projects to ones with a specified '
'platform group [auto|all|none|linux|darwin|...]',
metavar='PLATFORM')
group.add_option('--no-clone-bundle',
dest='no_clone_bundle', action='store_true',
help='disable use of /clone.bundle on HTTP/HTTPS')
group.add_option('--no-tags',
dest='no_tags', action='store_true',
help="don't fetch tags in the manifest")
# Manifest.
group = init_optparse.add_option_group('Manifest options')
group.add_option('-u', '--manifest-url',
help='manifest repository location', metavar='URL')
group.add_option('-b', '--manifest-branch',
help='manifest branch or revision', metavar='REVISION')
group.add_option('-m', '--manifest-name',
help='initial manifest file', metavar='NAME.xml')
group.add_option('--current-branch',
dest='current_branch_only', action='store_true',
help='fetch only current manifest branch from server')
group.add_option('--mirror', action='store_true',
help='create a replica of the remote repositories '
'rather than a client working directory')
group.add_option('--reference',
help='location of mirror directory', metavar='DIR')
group.add_option('--dissociate', action='store_true',
help='dissociate from reference mirrors after clone')
group.add_option('--depth', type='int', default=None,
help='create a shallow clone with given depth; '
'see git clone')
group.add_option('--partial-clone', action='store_true',
help='perform partial clone (https://git-scm.com/'
'docs/gitrepository-layout#_code_partialclone_code)')
group.add_option('--clone-filter', action='store', default='blob:none',
help='filter for use with --partial-clone '
'[default: %default]')
group.add_option('--archive', action='store_true',
help='checkout an archive instead of a git repository for '
'each project. See git archive.')
group.add_option('--submodules', action='store_true',
help='sync any submodules associated with the manifest repo')
group.add_option('-g', '--groups', default='default',
help='restrict manifest projects to ones with specified '
'group(s) [default|all|G1,G2,G3|G4,-G5,-G6]',
metavar='GROUP')
group.add_option('-p', '--platform', default='auto',
help='restrict manifest projects to ones with a specified '
'platform group [auto|all|none|linux|darwin|...]',
metavar='PLATFORM')
group.add_option('--no-clone-bundle', action='store_true',
help='disable use of /clone.bundle on HTTP/HTTPS')
group.add_option('--no-tags', action='store_true',
help="don't fetch tags in the manifest")
# Tool.
group = init_optparse.add_option_group('repo Version options')
group.add_option('--repo-url', metavar='URL',
help='repo repository location ($REPO_URL)')
group.add_option('--repo-branch', metavar='REVISION',
help='repo branch or revision ($REPO_REV)')
group.add_option('--no-repo-verify', action='store_true',
help='do not verify repo source code')
# Tool
group = init_optparse.add_option_group('repo Version options')
group.add_option('--repo-url',
dest='repo_url',
help='repo repository location', metavar='URL')
group.add_option('--repo-branch',
dest='repo_branch',
help='repo branch or revision', metavar='REVISION')
group.add_option('--no-repo-verify',
dest='no_repo_verify', action='store_true',
help='do not verify repo source code')
# Other
group = init_optparse.add_option_group('Other options')
group.add_option('--config-name',
dest='config_name', action="store_true", default=False,
help='Always prompt for name/e-mail')
# Other.
group = init_optparse.add_option_group('Other options')
group.add_option('--config-name',
action='store_true', default=False,
help='Always prompt for name/e-mail')
def _GitcInitOptions(init_optparse_arg):
@ -455,6 +479,39 @@ def _CheckGitVersion():
raise CloneFailure()
def SetGitTrace2ParentSid(env=None):
"""Set up GIT_TRACE2_PARENT_SID for git tracing."""
# We roughly follow the format git itself uses in trace2/tr2_sid.c.
# (1) Be unique (2) be valid filename (3) be fixed length.
#
# Since we always export this variable, we try to avoid more expensive calls.
# e.g. We don't attempt hostname lookups or hashing the results.
if env is None:
env = os.environ
KEY = 'GIT_TRACE2_PARENT_SID'
now = datetime.datetime.utcnow()
value = 'repo-%s-P%08x' % (now.strftime('%Y%m%dT%H%M%SZ'), os.getpid())
# If it's already set, then append ourselves.
if KEY in env:
value = env[KEY] + '/' + value
_setenv(KEY, value, env=env)
def _setenv(key, value, env=None):
"""Set |key| in the OS environment |env| to |value|."""
if env is None:
env = os.environ
# Environment handling across systems is messy.
try:
env[key] = value
except UnicodeEncodeError:
env[key] = value.encode()
def NeedSetupGnuPG():
if not os.path.isdir(home_dot_repo):
return True
@ -491,10 +548,7 @@ def SetupGnuPG(quiet):
sys.exit(1)
env = os.environ.copy()
try:
env['GNUPGHOME'] = gpg_dir
except UnicodeEncodeError:
env['GNUPGHOME'] = gpg_dir.encode()
_setenv('GNUPGHOME', gpg_dir, env)
cmd = ['gpg', '--import']
try:
@ -700,10 +754,7 @@ def _Verify(cwd, branch, quiet):
print(file=sys.stderr)
env = os.environ.copy()
try:
env['GNUPGHOME'] = gpg_dir
except UnicodeEncodeError:
env['GNUPGHOME'] = gpg_dir.encode()
_setenv('GNUPGHOME', gpg_dir, env)
cmd = [GIT, 'tag', '-v', cur]
proc = subprocess.Popen(cmd,
@ -768,6 +819,7 @@ def _FindRepo():
class _Options(object):
help = False
version = False
def _ParseArguments(args):
@ -779,7 +831,8 @@ def _ParseArguments(args):
a = args[i]
if a == '-h' or a == '--help':
opt.help = True
elif a == '--version':
opt.version = True
elif not a.startswith('-'):
cmd = a
arg = args[i + 1:]
@ -826,6 +879,16 @@ def _Help(args):
sys.exit(1)
def _Version():
"""Show version information."""
print('<repo not installed>')
print('repo launcher version %s' % ('.'.join(str(x) for x in VERSION),))
print(' (from %s)' % (__file__,))
print('git %s' % (ParseGitVersion().full,))
print('Python %s' % sys.version)
sys.exit(0)
def _NotInstalled():
print('error: repo is not installed. Use "repo init" to install it here.',
file=sys.stderr)
@ -878,6 +941,9 @@ def _SetDefaultsTo(gitdir):
def main(orig_args):
cmd, opt, args = _ParseArguments(orig_args)
# We run this early as we run some git commands ourselves.
SetGitTrace2ParentSid()
repo_main, rel_repo_dir = None, None
# Don't use the local repo copy, make sure to switch to the gitc client first.
if cmd != 'gitc-init':
@ -893,11 +959,14 @@ def main(orig_args):
'command from the corresponding client under /gitc/',
file=sys.stderr)
sys.exit(1)
_InitParser()
if not repo_main:
if opt.help:
_Usage()
if cmd == 'help':
_Help(args)
if opt.version or cmd == 'version':
_Version()
if not cmd:
_NotInstalled()
if cmd == 'init' or cmd == 'gitc-init':
@ -926,15 +995,9 @@ def main(orig_args):
'--']
me.extend(orig_args)
me.extend(extra_args)
try:
if platform.system() == "Windows":
sys.exit(subprocess.call(me))
else:
os.execv(sys.executable, me)
except OSError as e:
print("fatal: unable to start %s" % repo_main, file=sys.stderr)
print("fatal: %s" % e, file=sys.stderr)
sys.exit(148)
exec_command(me)
print("fatal: unable to start %s" % repo_main, file=sys.stderr)
sys.exit(148)
if __name__ == '__main__':

View File

@ -28,13 +28,16 @@ REPO_TRACE = 'REPO_TRACE'
_TRACE = os.environ.get(REPO_TRACE) == '1'
def IsTrace():
return _TRACE
def SetTrace():
global _TRACE
_TRACE = True
def Trace(fmt, *args):
if IsTrace():
print(fmt % args, file=sys.stderr)

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
# -*- coding:utf-8 -*-
# Copyright 2019 The Android Open Source Project
#

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
# -*- coding:utf-8 -*-
# Copyright 2019 The Android Open Source Project
#

View File

@ -40,7 +40,7 @@ for py in os.listdir(my_dir):
cmd = getattr(mod, clsn)()
except AttributeError:
raise SyntaxError('%s/%s does not define class %s' % (
__name__, py, clsn))
__name__, py, clsn))
name = name.replace('_', '-')
cmd.NAME = name

View File

@ -21,6 +21,7 @@ from collections import defaultdict
from git_command import git
from progress import Progress
class Abandon(Command):
common = True
helpSummary = "Permanently abandon a development branch"
@ -32,6 +33,7 @@ deleting it (and all its history) from your local repository.
It is equivalent to "git branch -D <branchname>".
"""
def _Options(self, p):
p.add_option('--all',
dest='all', action='store_true',
@ -79,10 +81,10 @@ It is equivalent to "git branch -D <branchname>".
if err:
for br in err.keys():
err_msg = "error: cannot abandon %s" %br
err_msg = "error: cannot abandon %s" % br
print(err_msg, file=sys.stderr)
for proj in err[br]:
print(' '*len(err_msg) + " | %s" % proj.relpath, file=sys.stderr)
print(' ' * len(err_msg) + " | %s" % proj.relpath, file=sys.stderr)
sys.exit(1)
elif not success:
print('error: no project has local branch(es) : %s' % nb,
@ -95,5 +97,5 @@ It is equivalent to "git branch -D <branchname>".
result = "all project"
else:
result = "%s" % (
('\n'+' '*width + '| ').join(p.relpath for p in success[br]))
print("%s%s| %s\n" % (br,' '*(width-len(br)), result),file=sys.stderr)
('\n' + ' ' * width + '| ').join(p.relpath for p in success[br]))
print("%s%s| %s\n" % (br, ' ' * (width - len(br)), result), file=sys.stderr)

View File

@ -19,13 +19,15 @@ import sys
from color import Coloring
from command import Command
class BranchColoring(Coloring):
def __init__(self, config):
Coloring.__init__(self, config, 'branch')
self.current = self.printer('current', fg='green')
self.local = self.printer('local')
self.local = self.printer('local')
self.notinproject = self.printer('notinproject', fg='red')
class BranchInfo(object):
def __init__(self, name):
self.name = name
@ -158,7 +160,7 @@ is shown, then the branch appears in all projects.
for b in i.projects:
have.add(b.project)
for p in projects:
if not p in have:
if p not in have:
paths.append(p.relpath)
s = ' %s %s' % (in_type, ', '.join(paths))
@ -170,11 +172,11 @@ is shown, then the branch appears in all projects.
fmt = out.current if i.IsCurrent else out.write
for p in paths:
out.nl()
fmt(width*' ' + ' %s' % p)
fmt(width * ' ' + ' %s' % p)
fmt = out.write
for p in non_cur_paths:
out.nl()
fmt(width*' ' + ' %s' % p)
fmt(width * ' ' + ' %s' % p)
else:
out.write(' in all projects')
out.nl()

View File

@ -19,6 +19,7 @@ import sys
from command import Command
from progress import Progress
class Checkout(Command):
common = True
helpSummary = "Checkout a branch for development"

View File

@ -22,6 +22,7 @@ from git_command import GitCommand
CHANGE_ID_RE = re.compile(r'^\s*Change-Id: I([0-9a-f]{40})\s*$')
class CherryPick(Command):
common = True
helpSummary = "Cherry-pick a change."
@ -46,8 +47,8 @@ change id will be added.
p = GitCommand(None,
['rev-parse', '--verify', reference],
capture_stdout = True,
capture_stderr = True)
capture_stdout=True,
capture_stderr=True)
if p.Wait() != 0:
print(p.stderr, file=sys.stderr)
sys.exit(1)
@ -61,8 +62,8 @@ change id will be added.
p = GitCommand(None,
['cherry-pick', sha1],
capture_stdout = True,
capture_stderr = True)
capture_stdout=True,
capture_stderr=True)
status = p.Wait()
print(p.stdout, file=sys.stdout)
@ -74,9 +75,9 @@ change id will be added.
new_msg = self._Reformat(old_msg, sha1)
p = GitCommand(None, ['commit', '--amend', '-F', '-'],
provide_stdin = True,
capture_stdout = True,
capture_stderr = True)
provide_stdin=True,
capture_stdout=True,
capture_stderr=True)
p.stdin.write(new_msg)
p.stdin.close()
if p.Wait() != 0:
@ -97,7 +98,7 @@ change id will be added.
def _StripHeader(self, commit_msg):
lines = commit_msg.splitlines()
return "\n".join(lines[lines.index("")+1:])
return "\n".join(lines[lines.index("") + 1:])
def _Reformat(self, old_msg, sha1):
new_msg = []

View File

@ -16,6 +16,7 @@
from command import PagedCommand
class Diff(PagedCommand):
common = True
helpSummary = "Show changes between commit and working tree"

View File

@ -18,10 +18,12 @@ from color import Coloring
from command import PagedCommand
from manifest_xml import XmlManifest
class _Coloring(Coloring):
def __init__(self, config):
Coloring.__init__(self, config, "status")
class Diffmanifests(PagedCommand):
""" A command to see logs in projects represented by manifests
@ -184,10 +186,10 @@ synced and their revisions won't be found.
self.out = _Coloring(self.manifest.globalConfig)
self.printText = self.out.nofmt_printer('text')
if opt.color:
self.printProject = self.out.nofmt_printer('project', attr = 'bold')
self.printAdded = self.out.nofmt_printer('green', fg = 'green', attr = 'bold')
self.printRemoved = self.out.nofmt_printer('red', fg = 'red', attr = 'bold')
self.printRevision = self.out.nofmt_printer('revision', fg = 'yellow')
self.printProject = self.out.nofmt_printer('project', attr='bold')
self.printAdded = self.out.nofmt_printer('green', fg='green', attr='bold')
self.printRemoved = self.out.nofmt_printer('red', fg='red', attr='bold')
self.printRevision = self.out.nofmt_printer('revision', fg='yellow')
else:
self.printProject = self.printAdded = self.printRemoved = self.printRevision = self.printText

5
subcmds/download.py Executable file → Normal file
View File

@ -23,6 +23,7 @@ from error import GitError
CHANGE_RE = re.compile(r'^([1-9][0-9]*)(?:[/\.-]([1-9][0-9]*))?$')
class Download(Command):
common = True
helpSummary = "Download and checkout a change"
@ -93,7 +94,7 @@ If no project is specified try to use current directory as a project.
continue
if len(dl.commits) > 1:
print('[%s] %d/%d depends on %d unmerged changes:' \
print('[%s] %d/%d depends on %d unmerged changes:'
% (project.name, change_id, ps_id, len(dl.commits)),
file=sys.stderr)
for c in dl.commits:
@ -102,7 +103,7 @@ If no project is specified try to use current directory as a project.
try:
project._CherryPick(dl.commit)
except GitError:
print('[%s] Could not complete the cherry-pick of %s' \
print('[%s] Could not complete the cherry-pick of %s'
% (project.name, dl.commit), file=sys.stderr)
sys.exit(1)

View File

@ -28,10 +28,10 @@ from command import Command, MirrorSafeCommand
import platform_utils
_CAN_COLOR = [
'branch',
'diff',
'grep',
'log',
'branch',
'diff',
'grep',
'log',
]
@ -170,14 +170,14 @@ without iterating through the remaining projects.
else:
lrev = None
return {
'name': project.name,
'relpath': project.relpath,
'remote_name': project.remote.name,
'lrev': lrev,
'rrev': project.revisionExpr,
'annotations': dict((a.name, a.value) for a in project.annotations),
'gitdir': project.gitdir,
'worktree': project.worktree,
'name': project.name,
'relpath': project.relpath,
'remote_name': project.remote.name,
'lrev': lrev,
'rrev': project.revisionExpr,
'annotations': dict((a.name, a.value) for a in project.annotations),
'gitdir': project.gitdir,
'worktree': project.worktree,
}
def ValidateOptions(self, opt, args):
@ -195,9 +195,9 @@ without iterating through the remaining projects.
cmd.append(cmd[0])
cmd.extend(opt.command[1:])
if opt.project_header \
and not shell \
and cmd[0] == 'git':
if opt.project_header \
and not shell \
and cmd[0] == 'git':
# If this is a direct git command that can enable colorized
# output and the user prefers coloring, add --color into the
# command line because we are going to wrap the command into
@ -220,7 +220,7 @@ without iterating through the remaining projects.
smart_sync_manifest_name = "smart_sync_override.xml"
smart_sync_manifest_path = os.path.join(
self.manifest.manifestProject.worktree, smart_sync_manifest_name)
self.manifest.manifestProject.worktree, smart_sync_manifest_name)
if os.path.isfile(smart_sync_manifest_path):
self.manifest.Override(smart_sync_manifest_path)
@ -238,8 +238,8 @@ without iterating through the remaining projects.
try:
config = self.manifest.manifestProject.config
results_it = pool.imap(
DoWorkWrapper,
self.ProjectArgs(projects, mirror, opt, cmd, shell, config))
DoWorkWrapper,
self.ProjectArgs(projects, mirror, opt, cmd, shell, config))
pool.close()
for r in results_it:
rc = rc or r
@ -253,7 +253,7 @@ without iterating through the remaining projects.
except Exception as e:
# Catch any other exceptions raised
print('Got an error, terminating the pool: %s: %s' %
(type(e).__name__, e),
(type(e).__name__, e),
file=sys.stderr)
pool.terminate()
rc = rc or getattr(e, 'errno', 1)
@ -268,7 +268,7 @@ without iterating through the remaining projects.
project = self._SerializeProject(p)
except Exception as e:
print('Project list error on project %s: %s: %s' %
(p.name, type(e).__name__, e),
(p.name, type(e).__name__, e),
file=sys.stderr)
return
except KeyboardInterrupt:
@ -277,6 +277,7 @@ without iterating through the remaining projects.
return
yield [mirror, opt, cmd, shell, cnt, config, project]
class WorkerKeyboardInterrupt(Exception):
""" Keyboard interrupt exception for worker processes. """
pass
@ -285,6 +286,7 @@ class WorkerKeyboardInterrupt(Exception):
def InitWorker():
signal.signal(signal.SIGINT, signal.SIG_IGN)
def DoWorkWrapper(args):
""" A wrapper around the DoWork() method.
@ -303,6 +305,7 @@ def DoWorkWrapper(args):
def DoWork(project, mirror, opt, cmd, shell, cnt, config):
env = os.environ.copy()
def setenv(name, val):
if val is None:
val = ''
@ -331,7 +334,7 @@ def DoWork(project, mirror, opt, cmd, shell, cnt, config):
if opt.ignore_missing:
return 0
if ((opt.project_header and opt.verbose)
or not opt.project_header):
or not opt.project_header):
print('skipping %s/' % project['relpath'], file=sys.stderr)
return 1
@ -366,7 +369,7 @@ def DoWork(project, mirror, opt, cmd, shell, cnt, config):
while not s_in.is_done:
in_ready = s_in.select()
for s in in_ready:
buf = s.read()
buf = s.read().decode()
if not buf:
s.close()
s_in.remove(s)

View File

@ -24,6 +24,7 @@ from pyversion import is_python3
if not is_python3():
input = raw_input
class GitcDelete(Command, GitcClientCommand):
common = True
visible_everywhere = False

View File

@ -50,7 +50,7 @@ use for this GITC client.
"""
def _Options(self, p):
super(GitcInit, self)._Options(p)
super(GitcInit, self)._Options(p, gitc_init=True)
g = p.add_option_group('GITC options')
g.add_option('-f', '--manifest-file',
dest='manifest_file',

View File

@ -21,7 +21,8 @@ import sys
from color import Coloring
from command import PagedCommand
from error import GitError
from git_command import git_require, GitCommand
from git_command import GitCommand
class GrepColoring(Coloring):
def __init__(self, config):
@ -29,6 +30,7 @@ class GrepColoring(Coloring):
self.project = self.printer('project', attr='bold')
self.fail = self.printer('fail', fg='red')
class Grep(PagedCommand):
common = True
helpSummary = "Print lines matching a pattern"
@ -156,12 +158,11 @@ contain a line that matches both expressions:
action='callback', callback=carry,
help='Show only file names not containing matching lines')
def Execute(self, opt, args):
out = GrepColoring(self.manifest.manifestProject.config)
cmd_argv = ['grep']
if out.is_on and git_require((1, 6, 3)):
if out.is_on:
cmd_argv.append('--color')
cmd_argv.extend(getattr(opt, 'cmd_argv', []))

View File

@ -23,6 +23,7 @@ from color import Coloring
from command import PagedCommand, MirrorSafeCommand, GitcAvailableCommand, GitcClientCommand
import gitc_utils
class Help(PagedCommand, MirrorSafeCommand):
common = False
helpSummary = "Display detailed help on a command"
@ -72,13 +73,13 @@ Displays detailed usage information about a command.
return False
commandNames = list(sorted([name
for name, command in self.commands.items()
if command.common and gitc_supported(command)]))
for name, command in self.commands.items()
if command.common and gitc_supported(command)]))
self._PrintCommands(commandNames)
print(
"See 'repo help <command>' for more information on a specific command.\n"
"See 'repo help --all' for a complete list of recognized commands.")
"See 'repo help <command>' for more information on a specific command.\n"
"See 'repo help --all' for a complete list of recognized commands.")
def _PrintCommandHelp(self, cmd, header_prefix=''):
class _Out(Coloring):

View File

@ -18,10 +18,12 @@ from command import PagedCommand
from color import Coloring
from git_refs import R_M
class _Coloring(Coloring):
def __init__(self, config):
Coloring.__init__(self, config, "status")
class Info(PagedCommand):
common = True
helpSummary = "Get info on the manifest branch, current branch or unmerged branches"
@ -41,15 +43,14 @@ class Info(PagedCommand):
dest="local", action="store_true",
help="Disable all remote operations")
def Execute(self, opt, args):
self.out = _Coloring(self.manifest.globalConfig)
self.heading = self.out.printer('heading', attr = 'bold')
self.headtext = self.out.nofmt_printer('headtext', fg = 'yellow')
self.redtext = self.out.printer('redtext', fg = 'red')
self.sha = self.out.printer("sha", fg = 'yellow')
self.heading = self.out.printer('heading', attr='bold')
self.headtext = self.out.nofmt_printer('headtext', fg='yellow')
self.redtext = self.out.printer('redtext', fg='red')
self.sha = self.out.printer("sha", fg='yellow')
self.text = self.out.nofmt_printer('text')
self.dimtext = self.out.printer('dimtext', attr = 'dim')
self.dimtext = self.out.printer('dimtext', attr='dim')
self.opt = opt
@ -122,7 +123,7 @@ class Info(PagedCommand):
self.printSeparator()
def findRemoteLocalDiff(self, project):
#Fetch all the latest commits
# Fetch all the latest commits.
if not self.opt.local:
project.Sync_NetworkHalf(quiet=True, current_branch_only=True)
@ -195,16 +196,16 @@ class Info(PagedCommand):
commits = branch.commits
date = branch.date
self.text('%s %-33s (%2d commit%s, %s)' % (
branch.name == project.CurrentBranch and '*' or ' ',
branch.name,
len(commits),
len(commits) != 1 and 's' or '',
date))
branch.name == project.CurrentBranch and '*' or ' ',
branch.name,
len(commits),
len(commits) != 1 and 's' or '',
date))
self.out.nl()
for commit in commits:
split = commit.split()
self.text('{0:38}{1} '.format('','-'))
self.text('{0:38}{1} '.format('', '-'))
self.sha(split[0] + " ")
self.text(" ".join(split[1:]))
self.out.nl()

View File

@ -34,9 +34,10 @@ from command import InteractiveCommand, MirrorSafeCommand
from error import ManifestParseError
from project import SyncBuffer
from git_config import GitConfig
from git_command import git_require, MIN_GIT_VERSION
from git_command import git_require, MIN_GIT_VERSION_SOFT, MIN_GIT_VERSION_HARD
import platform_utils
class Init(InteractiveCommand, MirrorSafeCommand):
common = True
helpSummary = "Initialize repo in the current directory"
@ -81,7 +82,7 @@ manifest, a subsequent `repo sync` (or `repo sync -d`) is necessary
to update the working directory files.
"""
def _Options(self, p):
def _Options(self, p, gitc_init=False):
# Logging
g = p.add_option_group('Logging options')
g.add_option('-q', '--quiet',
@ -96,7 +97,12 @@ to update the working directory files.
g.add_option('-b', '--manifest-branch',
dest='manifest_branch',
help='manifest branch or revision', metavar='REVISION')
g.add_option('-c', '--current-branch',
cbr_opts = ['--current-branch']
# The gitc-init subcommand allocates -c itself, but a lot of init users
# want -c, so try to satisfy both as best we can.
if not gitc_init:
cbr_opts += ['-c']
g.add_option(*cbr_opts,
dest='current_branch_only', action='store_true',
help='fetch only current manifest branch from server')
g.add_option('-m', '--manifest-name',
@ -218,7 +224,7 @@ to update the working directory files.
platformize = lambda x: 'platform-' + x
if opt.platform == 'auto':
if (not opt.mirror and
not m.config.GetString('repo.mirror') == 'true'):
not m.config.GetString('repo.mirror') == 'true'):
groups.append(platformize(platform.system().lower()))
elif opt.platform == 'all':
groups.extend(map(platformize, all_platforms))
@ -275,10 +281,10 @@ to update the working directory files.
m.config.SetString('repo.submodules', 'true')
if not m.Sync_NetworkHalf(is_new=is_new, quiet=opt.quiet,
clone_bundle=not opt.no_clone_bundle,
current_branch_only=opt.current_branch_only,
no_tags=opt.no_tags, submodules=opt.submodules,
clone_filter=opt.clone_filter):
clone_bundle=not opt.no_clone_bundle,
current_branch_only=opt.current_branch_only,
no_tags=opt.no_tags, submodules=opt.submodules,
clone_filter=opt.clone_filter):
r = m.GetRemote(m.remote.name)
print('fatal: cannot obtain manifest %s' % r.url, file=sys.stderr)
@ -344,7 +350,7 @@ to update the working directory files.
while True:
print()
name = self._Prompt('Your Name', mp.UserName)
name = self._Prompt('Your Name', mp.UserName)
email = self._Prompt('Your Email', mp.UserEmail)
print()
@ -446,7 +452,12 @@ to update the working directory files.
self.OptionParser.error('--mirror and --archive cannot be used together.')
def Execute(self, opt, args):
git_require(MIN_GIT_VERSION, fail=True)
git_require(MIN_GIT_VERSION_HARD, fail=True)
if not git_require(MIN_GIT_VERSION_SOFT):
print('repo: warning: git-%s+ will soon be required; please upgrade your '
'version of git to maintain support.'
% ('.'.join(str(x) for x in MIN_GIT_VERSION_SOFT),),
file=sys.stderr)
self._SyncManifest(opt)
self._LinkManifest(opt.manifest_name)

View File

@ -15,10 +15,10 @@
# limitations under the License.
from __future__ import print_function
import sys
from command import Command, MirrorSafeCommand
class List(Command, MirrorSafeCommand):
common = True
helpSummary = "List projects and their associated directories"
@ -77,7 +77,7 @@ This is similar to running: repo forall -c 'echo "$REPO_PATH : $REPO_PROJECT"'.
lines = []
for project in projects:
if opt.name_only and not opt.path_only:
lines.append("%s" % ( project.name))
lines.append("%s" % (project.name))
elif opt.path_only and not opt.name_only:
lines.append("%s" % (_getpath(project)))
else:

View File

@ -20,6 +20,7 @@ import sys
from command import PagedCommand
class Manifest(PagedCommand):
common = False
helpSummary = "Manifest inspection utility"
@ -66,8 +67,8 @@ in a Git repository for use during future 'repo init' invocations.
else:
fd = open(opt.output_file, 'w')
self.manifest.Save(fd,
peg_rev = opt.peg_rev,
peg_rev_upstream = opt.peg_rev_upstream)
peg_rev=opt.peg_rev,
peg_rev_upstream=opt.peg_rev_upstream)
fd.close()
if opt.output_file != '-':
print('Saved manifest to %s' % opt.output_file, file=sys.stderr)

View File

@ -18,6 +18,7 @@ from __future__ import print_function
from color import Coloring
from command import PagedCommand
class Prune(PagedCommand):
common = True
helpSummary = "Prune (delete) already merged topics"

View File

@ -43,8 +43,8 @@ branch but need to incorporate new upstream changes "underneath" them.
def _Options(self, p):
p.add_option('-i', '--interactive',
dest="interactive", action="store_true",
help="interactive rebase (single project only)")
dest="interactive", action="store_true",
help="interactive rebase (single project only)")
p.add_option('--fail-fast',
dest='fail_fast', action='store_true',
@ -82,7 +82,7 @@ branch but need to incorporate new upstream changes "underneath" them.
file=sys.stderr)
if len(args) == 1:
print('note: project %s is mapped to more than one path' % (args[0],),
file=sys.stderr)
file=sys.stderr)
return 1
# Setup the common git rebase args that we use for all projects.

View File

@ -22,6 +22,7 @@ from command import Command, MirrorSafeCommand
from subcmds.sync import _PostRepoUpgrade
from subcmds.sync import _PostRepoFetch
class Selfupdate(Command, MirrorSafeCommand):
common = False
helpSummary = "Update repo to the latest version"
@ -59,5 +60,5 @@ need to be performed by an end-user.
rp.bare_git.gc('--auto')
_PostRepoFetch(rp,
no_repo_verify = opt.no_repo_verify,
verbose = True)
no_repo_verify=opt.no_repo_verify,
verbose=True)

View File

@ -16,6 +16,7 @@
from subcmds.sync import Sync
class Smartsync(Sync):
common = True
helpSummary = "Update working tree to the latest known good revision"

View File

@ -21,6 +21,7 @@ from color import Coloring
from command import InteractiveCommand
from git_command import GitCommand
class _ProjectList(Coloring):
def __init__(self, gc):
Coloring.__init__(self, gc, 'interactive')
@ -28,6 +29,7 @@ class _ProjectList(Coloring):
self.header = self.printer('header', attr='bold')
self.help = self.printer('help', fg='red', attr='bold')
class Stage(InteractiveCommand):
common = True
helpSummary = "Stage file(s) for commit"
@ -105,6 +107,7 @@ The '%prog' command stages files to prepare the next commit.
continue
print('Bye.')
def _AddI(project):
p = GitCommand(project, ['add', '--interactive'], bare=False)
p.Wait()

View File

@ -25,6 +25,7 @@ import gitc_utils
from progress import Progress
from project import SyncBuffer
class Start(Command):
common = True
helpSummary = "Start a new branch for development"
@ -60,7 +61,7 @@ revision specified in the manifest.
if not opt.all:
projects = args[1:]
if len(projects) < 1:
projects = ['.',] # start it in the local project by default
projects = ['.'] # start it in the local project by default
all_projects = self.GetProjects(projects,
missing_ok=bool(self.gitc_manifest))
@ -113,7 +114,7 @@ revision specified in the manifest.
branch_merge = self.manifest.default.revisionExpr
if not project.StartBranch(
nb, branch_merge=branch_merge, revision=opt.revision):
nb, branch_merge=branch_merge, revision=opt.revision):
err.append(project)
pm.end()

View File

@ -31,6 +31,7 @@ import os
from color import Coloring
import platform_utils
class Status(PagedCommand):
common = True
helpSummary = "Show the working tree status"
@ -126,8 +127,8 @@ the following meanings:
continue
if item in proj_dirs_parents:
self._FindOrphans(glob.glob('%s/.*' % item) +
glob.glob('%s/*' % item),
proj_dirs, proj_dirs_parents, outstring)
glob.glob('%s/*' % item),
proj_dirs, proj_dirs_parents, outstring)
continue
outstring.append(''.join([status_header, item, '/']))
@ -170,8 +171,8 @@ the following meanings:
class StatusColoring(Coloring):
def __init__(self, config):
Coloring.__init__(self, config, 'status')
self.project = self.printer('header', attr = 'bold')
self.untracked = self.printer('untracked', fg = 'red')
self.project = self.printer('header', attr='bold')
self.untracked = self.printer('untracked', fg='red')
orig_path = os.getcwd()
try:
@ -179,8 +180,8 @@ the following meanings:
outstring = []
self._FindOrphans(glob.glob('.*') +
glob.glob('*'),
proj_dirs, proj_dirs_parents, outstring)
glob.glob('*'),
proj_dirs, proj_dirs_parents, outstring)
if outstring:
output = StatusColoring(self.manifest.globalConfig)

View File

@ -53,6 +53,7 @@ except ImportError:
try:
import resource
def _rlimit_nofile():
return resource.getrlimit(resource.RLIMIT_NOFILE)
except ImportError:
@ -81,13 +82,16 @@ from manifest_xml import GitcManifest
_ONE_DAY_S = 24 * 60 * 60
class _FetchError(Exception):
"""Internal error thrown in _FetchHelper() when we don't want stack trace."""
pass
class _CheckoutError(Exception):
"""Internal error thrown in _CheckoutOne() when we don't want stack trace."""
class Sync(Command, MirrorSafeCommand):
jobs = 1
common = True
@ -217,6 +221,10 @@ later is required to fix a server side protocol bug.
p.add_option('-l', '--local-only',
dest='local_only', action='store_true',
help="only update working tree, don't fetch")
p.add_option('--no-manifest-update', '--nmu',
dest='mp_update', action='store_false', default='true',
help='use the existing manifest checkout as-is. '
'(do not update to the latest revision)')
p.add_option('-n', '--network-only',
dest='network_only', action='store_true',
help="fetch only, don't update working tree")
@ -323,14 +331,14 @@ later is required to fix a server side protocol bug.
try:
try:
success = project.Sync_NetworkHalf(
quiet=opt.quiet,
current_branch_only=opt.current_branch_only,
force_sync=opt.force_sync,
clone_bundle=not opt.no_clone_bundle,
no_tags=opt.no_tags, archive=self.manifest.IsArchive,
optimized_fetch=opt.optimized_fetch,
prune=opt.prune,
clone_filter=clone_filter)
quiet=opt.quiet,
current_branch_only=opt.current_branch_only,
force_sync=opt.force_sync,
clone_bundle=not opt.no_clone_bundle,
no_tags=opt.no_tags, archive=self.manifest.IsArchive,
optimized_fetch=opt.optimized_fetch,
prune=opt.prune,
clone_filter=clone_filter)
self._fetch_times.Set(project, time.time() - start)
# Lock around all the rest of the code, since printing, updating a set
@ -351,8 +359,8 @@ later is required to fix a server side protocol bug.
except _FetchError:
pass
except Exception as e:
print('error: Cannot fetch %s (%s: %s)' \
% (project.name, type(e).__name__, str(e)), file=sys.stderr)
print('error: Cannot fetch %s (%s: %s)'
% (project.name, type(e).__name__, str(e)), file=sys.stderr)
err_event.set()
raise
finally:
@ -364,7 +372,7 @@ later is required to fix a server side protocol bug.
return success
def _Fetch(self, projects, opt):
def _Fetch(self, projects, opt, err_event):
fetched = set()
lock = _threading.Lock()
pm = Progress('Fetching projects', len(projects),
@ -376,7 +384,6 @@ later is required to fix a server side protocol bug.
threads = set()
sem = _threading.Semaphore(self.jobs)
err_event = _threading.Event()
for project_list in objdir_project_map.values():
# Check for any errors before running any more tasks.
# ...we'll let existing threads finish, though.
@ -393,8 +400,8 @@ later is required to fix a server side protocol bug.
err_event=err_event,
clone_filter=self.manifest.CloneFilter)
if self.jobs > 1:
t = _threading.Thread(target = self._FetchProjectList,
kwargs = kwargs)
t = _threading.Thread(target=self._FetchProjectList,
kwargs=kwargs)
# Ensure that Ctrl-C will not freeze the repo process.
t.daemon = True
threads.add(t)
@ -405,16 +412,11 @@ later is required to fix a server side protocol bug.
for t in threads:
t.join()
# If we saw an error, exit with code 1 so that other scripts can check.
if err_event.isSet() and opt.fail_fast:
print('\nerror: Exited sync due to fetch errors', file=sys.stderr)
sys.exit(1)
pm.end()
self._fetch_times.Save()
if not self.manifest.IsArchive:
self._GCProjects(projects)
self._GCProjects(projects, opt, err_event)
return fetched
@ -500,12 +502,16 @@ later is required to fix a server side protocol bug.
return success
def _Checkout(self, all_projects, opt):
def _Checkout(self, all_projects, opt, err_event, err_results):
"""Checkout projects listed in all_projects
Args:
all_projects: List of all projects that should be checked out.
opt: Program options returned from optparse. See _Options().
err_event: We'll set this event in the case of an error (after printing
out info about the error).
err_results: A list of strings, paths to git repos where checkout
failed.
"""
# Perform checkouts in multiple threads when we are using partial clone.
@ -524,8 +530,6 @@ later is required to fix a server side protocol bug.
threads = set()
sem = _threading.Semaphore(syncjobs)
err_event = _threading.Event()
err_results = []
for project in all_projects:
# Check for any errors before running any more tasks.
@ -556,24 +560,27 @@ later is required to fix a server side protocol bug.
t.join()
pm.end()
# If we saw an error, exit with code 1 so that other scripts can check.
if err_event.isSet():
print('\nerror: Exited sync due to checkout errors', file=sys.stderr)
if err_results:
print('Failing repos:\n%s' % '\n'.join(err_results),
file=sys.stderr)
sys.exit(1)
def _GCProjects(self, projects):
def _GCProjects(self, projects, opt, err_event):
gc_gitdirs = {}
for project in projects:
# Make sure pruning never kicks in with shared projects.
if len(project.manifest.GetProjectsWithName(project.name)) > 1:
print('Shared project %s found, disabling pruning.' % project.name)
project.bare_git.config('--replace-all', 'gc.pruneExpire', 'never')
print('%s: Shared project %s found, disabling pruning.' %
(project.relpath, project.name))
if git_require((2, 7, 0)):
project.config.SetString('core.repositoryFormatVersion', '1')
project.config.SetString('extensions.preciousObjects', 'true')
else:
# This isn't perfect, but it's the best we can do with old git.
print('%s: WARNING: shared projects are unreliable when using old '
'versions of git; please upgrade to git-2.7.0+.'
% (project.relpath,),
file=sys.stderr)
project.config.SetString('gc.pruneExpire', 'never')
gc_gitdirs[project.gitdir] = project.bare_git
has_dash_c = git_require((1, 7, 2))
if multiprocessing and has_dash_c:
if multiprocessing:
cpu_count = multiprocessing.cpu_count()
else:
cpu_count = 1
@ -588,7 +595,6 @@ later is required to fix a server side protocol bug.
threads = set()
sem = _threading.Semaphore(jobs)
err_event = _threading.Event()
def GC(bare_git):
try:
@ -596,14 +602,14 @@ later is required to fix a server side protocol bug.
bare_git.gc('--auto', config=config)
except GitError:
err_event.set()
except:
except Exception:
err_event.set()
raise
finally:
sem.release()
for bare_git in gc_gitdirs.values():
if err_event.isSet():
if err_event.isSet() and opt.fail_fast:
break
sem.acquire()
t = _threading.Thread(target=GC, args=(bare_git,))
@ -614,10 +620,6 @@ later is required to fix a server side protocol bug.
for t in threads:
t.join()
if err_event.isSet():
print('\nerror: Exited sync due to gc errors', file=sys.stderr)
sys.exit(1)
def _ReloadManifest(self, manifest_name=None):
if manifest_name:
# Override calls _Unload already
@ -705,16 +707,16 @@ later is required to fix a server side protocol bug.
gitdir = os.path.join(self.manifest.topdir, path, '.git')
if os.path.exists(gitdir):
project = Project(
manifest = self.manifest,
name = path,
remote = RemoteSpec('origin'),
gitdir = gitdir,
objdir = gitdir,
worktree = os.path.join(self.manifest.topdir, path),
relpath = path,
revisionExpr = 'HEAD',
revisionId = None,
groups = None)
manifest=self.manifest,
name=path,
remote=RemoteSpec('origin'),
gitdir=gitdir,
objdir=gitdir,
worktree=os.path.join(self.manifest.topdir, path),
relpath=path,
revisionExpr='HEAD',
revisionId=None,
groups=None)
if project.IsDirty() and opt.force_remove_dirty:
print('WARNING: Removing dirty project "%s": uncommitted changes '
@ -745,7 +747,7 @@ later is required to fix a server side protocol bug.
if not opt.quiet:
print('Using manifest server %s' % manifest_server)
if not '@' in manifest_server:
if '@' not in manifest_server:
username = None
password = None
if opt.manifest_server_username and opt.manifest_server_password:
@ -886,7 +888,7 @@ later is required to fix a server side protocol bug.
manifest_name = opt.manifest_name
smart_sync_manifest_path = os.path.join(
self.manifest.manifestProject.worktree, 'smart_sync_override.xml')
self.manifest.manifestProject.worktree, 'smart_sync_override.xml')
if opt.smart_sync or opt.smart_tag:
manifest_name = self._SmartSyncSetup(opt, smart_sync_manifest_path)
@ -898,6 +900,8 @@ later is required to fix a server side protocol bug.
print('error: failed to remove existing smart sync override manifest: %s' %
e, file=sys.stderr)
err_event = _threading.Event()
rp = self.manifest.repoProject
rp.PreSync()
@ -907,7 +911,10 @@ later is required to fix a server side protocol bug.
if opt.repo_upgraded:
_PostRepoUpgrade(self.manifest, quiet=opt.quiet)
self._UpdateManifestProject(opt, mp, manifest_name)
if not opt.mp_update:
print('Skipping update of local manifest project.')
else:
self._UpdateManifestProject(opt, mp, manifest_name)
if self.gitc_manifest:
gitc_manifest_projects = self.GetProjects(args,
@ -948,6 +955,10 @@ later is required to fix a server side protocol bug.
missing_ok=True,
submodules_ok=opt.fetch_submodules)
err_network_sync = False
err_update_projects = False
err_checkout = False
self._fetch_times = _FetchTimes(self.manifest)
if not opt.local_only:
to_fetch = []
@ -957,10 +968,14 @@ later is required to fix a server side protocol bug.
to_fetch.extend(all_projects)
to_fetch.sort(key=self._fetch_times.Get, reverse=True)
fetched = self._Fetch(to_fetch, opt)
fetched = self._Fetch(to_fetch, opt, err_event)
_PostRepoFetch(rp, opt.no_repo_verify)
if opt.network_only:
# bail out now; the rest touches the working tree
if err_event.isSet():
print('\nerror: Exited sync due to fetch errors.\n', file=sys.stderr)
sys.exit(1)
return
# Iteratively fetch missing and/or nested unregistered submodules
@ -982,22 +997,60 @@ later is required to fix a server side protocol bug.
if previously_missing_set == missing_set:
break
previously_missing_set = missing_set
fetched.update(self._Fetch(missing, opt))
fetched.update(self._Fetch(missing, opt, err_event))
# If we saw an error, exit with code 1 so that other scripts can check.
if err_event.isSet():
err_network_sync = True
if opt.fail_fast:
print('\nerror: Exited sync due to fetch errors.\n'
'Local checkouts *not* updated. Resolve network issues & '
'retry.\n'
'`repo sync -l` will update some local checkouts.',
file=sys.stderr)
sys.exit(1)
if self.manifest.IsMirror or self.manifest.IsArchive:
# bail out now, we have no working tree
return
if self.UpdateProjectList(opt):
sys.exit(1)
err_event.set()
err_update_projects = True
if opt.fail_fast:
print('\nerror: Local checkouts *not* updated.', file=sys.stderr)
sys.exit(1)
self._Checkout(all_projects, opt)
err_results = []
self._Checkout(all_projects, opt, err_event, err_results)
if err_event.isSet():
err_checkout = True
# NB: We don't exit here because this is the last step.
# If there's a notice that's supposed to print at the end of the sync, print
# it now...
if self.manifest.notice:
print(self.manifest.notice)
# If we saw an error, exit with code 1 so that other scripts can check.
if err_event.isSet():
print('\nerror: Unable to fully sync the tree.', file=sys.stderr)
if err_network_sync:
print('error: Downloading network changes failed.', file=sys.stderr)
if err_update_projects:
print('error: Updating local project lists failed.', file=sys.stderr)
if err_checkout:
print('error: Checking out local projects failed.', file=sys.stderr)
if err_results:
print('Failing repos:\n%s' % '\n'.join(err_results), file=sys.stderr)
print('Try re-running with "-j1 --fail-fast" to exit at the first error.',
file=sys.stderr)
sys.exit(1)
if not opt.quiet:
print('repo sync has finished successfully.')
def _PostRepoUpgrade(manifest, quiet=False):
wrapper = Wrapper()
if wrapper.NeedSetupGnuPG():
@ -1006,6 +1059,7 @@ def _PostRepoUpgrade(manifest, quiet=False):
if project.Exists:
project.PostRepoUpgrade()
def _PostRepoFetch(rp, no_repo_verify=False, verbose=False):
if rp.HasChanges:
print('info: A new version of repo is available', file=sys.stderr)
@ -1024,6 +1078,7 @@ def _PostRepoFetch(rp, no_repo_verify=False, verbose=False):
print('repo version %s is current' % rp.work_git.describe(HEAD),
file=sys.stderr)
def _VerifyTag(project):
gpg_dir = os.path.expanduser('~/.repoconfig/gnupg')
if not os.path.exists(gpg_dir):
@ -1054,9 +1109,9 @@ def _VerifyTag(project):
cmd = [GIT, 'tag', '-v', cur]
proc = subprocess.Popen(cmd,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE,
env = env)
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env=env)
out = proc.stdout.read()
proc.stdout.close()
@ -1090,7 +1145,7 @@ class _FetchTimes(object):
old = self._times.get(name, t)
self._seen.add(name)
a = self._ALPHA
self._times[name] = (a*t) + ((1-a) * old)
self._times[name] = (a * t) + ((1 - a) * old)
def _Load(self):
if self._times is None:
@ -1128,6 +1183,8 @@ class _FetchTimes(object):
# and supporting persistent-http[s]. It cannot change hosts from
# request to request like the normal transport, the real url
# is passed during initialization.
class PersistentTransport(xmlrpc.client.Transport):
def __init__(self, orig_host):
self.orig_host = orig_host
@ -1162,7 +1219,7 @@ class PersistentTransport(xmlrpc.client.Transport):
if proxy:
proxyhandler = urllib.request.ProxyHandler({
"http": proxy,
"https": proxy })
"https": proxy})
opener = urllib.request.build_opener(
urllib.request.HTTPCookieProcessor(cookiejar),
@ -1219,4 +1276,3 @@ class PersistentTransport(xmlrpc.client.Transport):
def close(self):
pass

View File

@ -33,6 +33,7 @@ else:
UNUSUAL_COMMIT_THRESHOLD = 5
def _ConfirmManyUploads(multiple_branches=False):
if multiple_branches:
print('ATTENTION: One or more branches has an unusually high number '
@ -44,17 +45,20 @@ def _ConfirmManyUploads(multiple_branches=False):
answer = input("If you are sure you intend to do this, type 'yes': ").strip()
return answer == "yes"
def _die(fmt, *args):
msg = fmt % args
print('error: %s' % msg, file=sys.stderr)
sys.exit(1)
def _SplitEmails(values):
result = []
for value in values:
result.extend([s.strip() for s in value.split(',')])
return result
class Upload(InteractiveCommand):
common = True
helpSummary = "Upload changes for code review"
@ -137,13 +141,13 @@ Gerrit Code Review: https://www.gerritcodereview.com/
dest='auto_topic', action='store_true',
help='Send local branch name to Gerrit Code Review')
p.add_option('--re', '--reviewers',
type='string', action='append', dest='reviewers',
type='string', action='append', dest='reviewers',
help='Request reviews from these people.')
p.add_option('--cc',
type='string', action='append', dest='cc',
type='string', action='append', dest='cc',
help='Also send email to these email addresses.')
p.add_option('--br',
type='string', action='store', dest='branch',
type='string', action='store', dest='branch',
help='Branch to upload.')
p.add_option('--cbr', '--current-branch',
dest='current_branch', action='store_true',
@ -168,6 +172,9 @@ Gerrit Code Review: https://www.gerritcodereview.com/
type='string', action='store', dest='dest_branch',
metavar='BRANCH',
help='Submit for review on this target branch.')
p.add_option('--no-cert-checks',
dest='validate_certs', action='store_false', default=True,
help='Disable verifying ssl certs (unsafe).')
# Options relating to upload hook. Note that verify and no-verify are NOT
# opposites of each other, which is why they store to different locations.
@ -185,15 +192,16 @@ Gerrit Code Review: https://www.gerritcodereview.com/
# Never run upload hooks, but upload anyway (AKA bypass hooks).
# - no-verify=True, verify=True:
# Invalid
p.add_option('--no-cert-checks',
dest='validate_certs', action='store_false', default=True,
help='Disable verifying ssl certs (unsafe).')
p.add_option('--no-verify',
g = p.add_option_group('Upload hooks')
g.add_option('--no-verify',
dest='bypass_hooks', action='store_true',
help='Do not run the upload hook.')
p.add_option('--verify',
g.add_option('--verify',
dest='allow_all_hooks', action='store_true',
help='Run the upload hook without prompting.')
g.add_option('--ignore-hooks',
dest='ignore_hooks', action='store_true',
help='Do not abort uploading if upload hooks fail.')
def _SingleBranch(self, opt, branch, people):
project = branch.project
@ -214,10 +222,10 @@ Gerrit Code Review: https://www.gerritcodereview.com/
print('Upload project %s/ to remote branch %s%s:' %
(project.relpath, destination, ' (draft)' if opt.draft else ''))
print(' branch %s (%2d commit%s, %s):' % (
name,
len(commit_list),
len(commit_list) != 1 and 's' or '',
date))
name,
len(commit_list),
len(commit_list) != 1 and 's' or '',
date))
for commit in commit_list:
print(' %s' % commit)
@ -322,12 +330,12 @@ Gerrit Code Review: https://www.gerritcodereview.com/
key = 'review.%s.autoreviewer' % project.GetBranch(name).remote.review
raw_list = project.config.GetString(key)
if not raw_list is None:
if raw_list is not None:
people[0].extend([entry.strip() for entry in raw_list.split(',')])
key = 'review.%s.autocopy' % project.GetBranch(name).remote.review
raw_list = project.config.GetString(key)
if not raw_list is None and len(people[0]) > 0:
if raw_list is not None and len(people[0]) > 0:
people[1].extend([entry.strip() for entry in raw_list.split(',')])
def _FindGerritChange(self, branch):
@ -418,18 +426,18 @@ Gerrit Code Review: https://www.gerritcodereview.com/
else:
fmt = '\n (%s)'
print(('[FAILED] %-15s %-15s' + fmt) % (
branch.project.relpath + '/', \
branch.name, \
str(branch.error)),
file=sys.stderr)
branch.project.relpath + '/',
branch.name,
str(branch.error)),
file=sys.stderr)
print()
for branch in todo:
if branch.uploaded:
print('[OK ] %-15s %s' % (
branch.project.relpath + '/',
branch.name),
file=sys.stderr)
branch.project.relpath + '/',
branch.name),
file=sys.stderr)
if have_errors:
sys.exit(1)
@ -437,14 +445,14 @@ Gerrit Code Review: https://www.gerritcodereview.com/
def _GetMergeBranch(self, project):
p = GitCommand(project,
['rev-parse', '--abbrev-ref', 'HEAD'],
capture_stdout = True,
capture_stderr = True)
capture_stdout=True,
capture_stderr=True)
p.Wait()
local_branch = p.stdout.strip()
p = GitCommand(project,
['config', '--get', 'branch.%s.merge' % local_branch],
capture_stdout = True,
capture_stderr = True)
capture_stdout=True,
capture_stderr=True)
p.Wait()
merge_branch = p.stdout.strip()
return merge_branch
@ -488,12 +496,24 @@ Gerrit Code Review: https://www.gerritcodereview.com/
abort_if_user_denies=True)
pending_proj_names = [project.name for (project, available) in pending]
pending_worktrees = [project.worktree for (project, available) in pending]
passed = True
try:
hook.Run(opt.allow_all_hooks, project_list=pending_proj_names,
worktree_list=pending_worktrees)
except SystemExit:
passed = False
if not opt.ignore_hooks:
raise
except HookError as e:
passed = False
print("ERROR: %s" % str(e), file=sys.stderr)
return
if not passed:
if opt.ignore_hooks:
print('\nWARNING: pre-upload hooks failed, but uploading anyways.',
file=sys.stderr)
else:
return
if opt.reviewers:
reviewers = _SplitEmails(opt.reviewers)

View File

@ -20,6 +20,7 @@ from command import Command, MirrorSafeCommand
from git_command import git, RepoSourceVersion, user_agent
from git_refs import HEAD
class Version(Command, MirrorSafeCommand):
wrapper_version = None
wrapper_path = None

View File

@ -35,7 +35,7 @@ class GitCallUnitTest(unittest.TestCase):
# We don't dive too deep into the values here to avoid having to update
# whenever git versions change. We do check relative to this min version
# as this is what `repo` itself requires via MIN_GIT_VERSION.
MIN_GIT_VERSION = (1, 7, 2)
MIN_GIT_VERSION = (2, 10, 2)
self.assertTrue(isinstance(ver.major, int))
self.assertTrue(isinstance(ver.minor, int))
self.assertTrue(isinstance(ver.micro, int))

View File

@ -23,14 +23,17 @@ import unittest
import git_config
def fixture(*paths):
"""Return a path relative to test/fixtures.
"""
return os.path.join(os.path.dirname(__file__), 'fixtures', *paths)
class GitConfigUnitTest(unittest.TestCase):
"""Tests the GitConfig class.
"""
def setUp(self):
"""Create a GitConfig object using the test.gitconfig fixture.
"""
@ -68,5 +71,6 @@ class GitConfigUnitTest(unittest.TestCase):
val = config.GetString('empty')
self.assertEqual(val, None)
if __name__ == '__main__':
unittest.main()

View File

@ -0,0 +1,85 @@
# -*- coding:utf-8 -*-
#
# Copyright (C) 2019 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unittests for the manifest_xml.py module."""
from __future__ import print_function
import unittest
import error
import manifest_xml
class ManifestValidateFilePaths(unittest.TestCase):
"""Check _ValidateFilePaths helper.
This doesn't access a real filesystem.
"""
def check_both(self, *args):
manifest_xml.XmlManifest._ValidateFilePaths('copyfile', *args)
manifest_xml.XmlManifest._ValidateFilePaths('linkfile', *args)
def test_normal_path(self):
"""Make sure good paths are accepted."""
self.check_both('foo', 'bar')
self.check_both('foo/bar', 'bar')
self.check_both('foo', 'bar/bar')
self.check_both('foo/bar', 'bar/bar')
def test_symlink_targets(self):
"""Some extra checks for symlinks."""
def check(*args):
manifest_xml.XmlManifest._ValidateFilePaths('linkfile', *args)
# We allow symlinks to end in a slash since we allow them to point to dirs
# in general. Technically the slash isn't necessary.
check('foo/', 'bar')
# We allow a single '.' to get a reference to the project itself.
check('.', 'bar')
def test_bad_paths(self):
"""Make sure bad paths (src & dest) are rejected."""
PATHS = (
'..',
'../',
'./',
'foo/',
'./foo',
'../foo',
'foo/./bar',
'foo/../../bar',
'/foo',
'./../foo',
'.git/foo',
# Check case folding.
'.GIT/foo',
'blah/.git/foo',
'.repo/foo',
'.repoconfig',
# Block ~ due to 8.3 filenames on Windows filesystems.
'~',
'foo~',
'blah/foo~',
# Block Unicode characters that get normalized out by filesystems.
u'foo\u200Cbar',
)
for path in PATHS:
self.assertRaises(
error.ManifestInvalidPathError, self.check_both, path, 'a')
self.assertRaises(
error.ManifestInvalidPathError, self.check_both, 'a', path)

View File

@ -25,6 +25,7 @@ import subprocess
import tempfile
import unittest
import error
import git_config
import project
@ -134,3 +135,229 @@ class ReviewableBranchTests(unittest.TestCase):
self.assertFalse(rb.base_exists)
# Hard to assert anything useful about this.
self.assertTrue(rb.date)
class CopyLinkTestCase(unittest.TestCase):
"""TestCase for stub repo client checkouts.
It'll have a layout like:
tempdir/ # self.tempdir
checkout/ # self.topdir
git-project/ # self.worktree
Attributes:
tempdir: A dedicated temporary directory.
worktree: The top of the repo client checkout.
topdir: The top of a project checkout.
"""
def setUp(self):
self.tempdir = tempfile.mkdtemp(prefix='repo_tests')
self.topdir = os.path.join(self.tempdir, 'checkout')
self.worktree = os.path.join(self.topdir, 'git-project')
os.makedirs(self.topdir)
os.makedirs(self.worktree)
def tearDown(self):
shutil.rmtree(self.tempdir, ignore_errors=True)
@staticmethod
def touch(path):
with open(path, 'w'):
pass
def assertExists(self, path, msg=None):
"""Make sure |path| exists."""
if os.path.exists(path):
return
if msg is None:
msg = ['path is missing: %s' % path]
while path != '/':
path = os.path.dirname(path)
if not path:
# If we're given something like "foo", abort once we get to "".
break
result = os.path.exists(path)
msg.append('\tos.path.exists(%s): %s' % (path, result))
if result:
msg.append('\tcontents: %r' % os.listdir(path))
break
msg = '\n'.join(msg)
raise self.failureException(msg)
class CopyFile(CopyLinkTestCase):
"""Check _CopyFile handling."""
def CopyFile(self, src, dest):
return project._CopyFile(self.worktree, src, self.topdir, dest)
def test_basic(self):
"""Basic test of copying a file from a project to the toplevel."""
src = os.path.join(self.worktree, 'foo.txt')
self.touch(src)
cf = self.CopyFile('foo.txt', 'foo')
cf._Copy()
self.assertExists(os.path.join(self.topdir, 'foo'))
def test_src_subdir(self):
"""Copy a file from a subdir of a project."""
src = os.path.join(self.worktree, 'bar', 'foo.txt')
os.makedirs(os.path.dirname(src))
self.touch(src)
cf = self.CopyFile('bar/foo.txt', 'new.txt')
cf._Copy()
self.assertExists(os.path.join(self.topdir, 'new.txt'))
def test_dest_subdir(self):
"""Copy a file to a subdir of a checkout."""
src = os.path.join(self.worktree, 'foo.txt')
self.touch(src)
cf = self.CopyFile('foo.txt', 'sub/dir/new.txt')
self.assertFalse(os.path.exists(os.path.join(self.topdir, 'sub')))
cf._Copy()
self.assertExists(os.path.join(self.topdir, 'sub', 'dir', 'new.txt'))
def test_update(self):
"""Make sure changed files get copied again."""
src = os.path.join(self.worktree, 'foo.txt')
dest = os.path.join(self.topdir, 'bar')
with open(src, 'w') as f:
f.write('1st')
cf = self.CopyFile('foo.txt', 'bar')
cf._Copy()
self.assertExists(dest)
with open(dest) as f:
self.assertEqual(f.read(), '1st')
with open(src, 'w') as f:
f.write('2nd!')
cf._Copy()
with open(dest) as f:
self.assertEqual(f.read(), '2nd!')
def test_src_block_symlink(self):
"""Do not allow reading from a symlinked path."""
src = os.path.join(self.worktree, 'foo.txt')
sym = os.path.join(self.worktree, 'sym')
self.touch(src)
os.symlink('foo.txt', sym)
self.assertExists(sym)
cf = self.CopyFile('sym', 'foo')
self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
def test_src_block_symlink_traversal(self):
"""Do not allow reading through a symlink dir."""
src = os.path.join(self.worktree, 'bar', 'passwd')
os.symlink('/etc', os.path.join(self.worktree, 'bar'))
self.assertExists(src)
cf = self.CopyFile('bar/foo.txt', 'foo')
self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
def test_src_block_copy_from_dir(self):
"""Do not allow copying from a directory."""
src = os.path.join(self.worktree, 'dir')
os.makedirs(src)
cf = self.CopyFile('dir', 'foo')
self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
def test_dest_block_symlink(self):
"""Do not allow writing to a symlink."""
src = os.path.join(self.worktree, 'foo.txt')
self.touch(src)
os.symlink('dest', os.path.join(self.topdir, 'sym'))
cf = self.CopyFile('foo.txt', 'sym')
self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
def test_dest_block_symlink_traversal(self):
"""Do not allow writing through a symlink dir."""
src = os.path.join(self.worktree, 'foo.txt')
self.touch(src)
os.symlink('/tmp', os.path.join(self.topdir, 'sym'))
cf = self.CopyFile('foo.txt', 'sym/foo.txt')
self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
def test_src_block_copy_to_dir(self):
"""Do not allow copying to a directory."""
src = os.path.join(self.worktree, 'foo.txt')
self.touch(src)
os.makedirs(os.path.join(self.topdir, 'dir'))
cf = self.CopyFile('foo.txt', 'dir')
self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
class LinkFile(CopyLinkTestCase):
"""Check _LinkFile handling."""
def LinkFile(self, src, dest):
return project._LinkFile(self.worktree, src, self.topdir, dest)
def test_basic(self):
"""Basic test of linking a file from a project into the toplevel."""
src = os.path.join(self.worktree, 'foo.txt')
self.touch(src)
lf = self.LinkFile('foo.txt', 'foo')
lf._Link()
dest = os.path.join(self.topdir, 'foo')
self.assertExists(dest)
self.assertTrue(os.path.islink(dest))
self.assertEqual('git-project/foo.txt', os.readlink(dest))
def test_src_subdir(self):
"""Link to a file in a subdir of a project."""
src = os.path.join(self.worktree, 'bar', 'foo.txt')
os.makedirs(os.path.dirname(src))
self.touch(src)
lf = self.LinkFile('bar/foo.txt', 'foo')
lf._Link()
self.assertExists(os.path.join(self.topdir, 'foo'))
def test_src_self(self):
"""Link to the project itself."""
dest = os.path.join(self.topdir, 'foo', 'bar')
lf = self.LinkFile('.', 'foo/bar')
lf._Link()
self.assertExists(dest)
self.assertEqual('../git-project', os.readlink(dest))
def test_dest_subdir(self):
"""Link a file to a subdir of a checkout."""
src = os.path.join(self.worktree, 'foo.txt')
self.touch(src)
lf = self.LinkFile('foo.txt', 'sub/dir/foo/bar')
self.assertFalse(os.path.exists(os.path.join(self.topdir, 'sub')))
lf._Link()
self.assertExists(os.path.join(self.topdir, 'sub', 'dir', 'foo', 'bar'))
def test_src_block_relative(self):
"""Do not allow relative symlinks."""
BAD_SOURCES = (
'./',
'..',
'../',
'foo/.',
'foo/./bar',
'foo/..',
'foo/../foo',
)
for src in BAD_SOURCES:
lf = self.LinkFile(src, 'foo')
self.assertRaises(error.ManifestInvalidPathError, lf._Link)
def test_update(self):
"""Make sure changed targets get updated."""
dest = os.path.join(self.topdir, 'sym')
src = os.path.join(self.worktree, 'foo.txt')
self.touch(src)
lf = self.LinkFile('foo.txt', 'sym')
lf._Link()
self.assertEqual('git-project/foo.txt', os.readlink(dest))
# Point the symlink somewhere else.
os.unlink(dest)
os.symlink('/', dest)
lf._Link()
self.assertEqual('git-project/foo.txt', os.readlink(dest))

View File

@ -19,24 +19,53 @@
from __future__ import print_function
import os
import re
import unittest
from pyversion import is_python3
import wrapper
if is_python3():
from unittest import mock
from io import StringIO
else:
import mock
from StringIO import StringIO
def fixture(*paths):
"""Return a path relative to tests/fixtures.
"""
return os.path.join(os.path.dirname(__file__), 'fixtures', *paths)
class RepoWrapperUnitTest(unittest.TestCase):
"""Tests helper functions in the repo wrapper
"""
class RepoWrapperTestCase(unittest.TestCase):
"""TestCase for the wrapper module."""
def setUp(self):
"""Load the wrapper module every time
"""
"""Load the wrapper module every time."""
wrapper._wrapper_module = None
self.wrapper = wrapper.Wrapper()
if not is_python3():
self.assertRegex = self.assertRegexpMatches
class RepoWrapperUnitTest(RepoWrapperTestCase):
"""Tests helper functions in the repo wrapper
"""
def test_version(self):
"""Make sure _Version works."""
with self.assertRaises(SystemExit) as e:
with mock.patch('sys.stdout', new_callable=StringIO) as stdout:
with mock.patch('sys.stderr', new_callable=StringIO) as stderr:
self.wrapper._Version()
self.assertEqual(0, e.exception.code)
self.assertEqual('', stderr.getvalue())
self.assertIn('repo launcher version', stdout.getvalue())
def test_get_gitc_manifest_dir_no_gitc(self):
"""
Test reading a missing gitc config file
@ -76,5 +105,38 @@ class RepoWrapperUnitTest(unittest.TestCase):
self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/'), None)
self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/'), None)
class SetGitTrace2ParentSid(RepoWrapperTestCase):
"""Check SetGitTrace2ParentSid behavior."""
KEY = 'GIT_TRACE2_PARENT_SID'
VALID_FORMAT = re.compile(r'^repo-[0-9]{8}T[0-9]{6}Z-P[0-9a-f]{8}$')
def test_first_set(self):
"""Test env var not yet set."""
env = {}
self.wrapper.SetGitTrace2ParentSid(env)
self.assertIn(self.KEY, env)
value = env[self.KEY]
self.assertRegex(value, self.VALID_FORMAT)
def test_append(self):
"""Test env var is appended."""
env = {self.KEY: 'pfx'}
self.wrapper.SetGitTrace2ParentSid(env)
self.assertIn(self.KEY, env)
value = env[self.KEY]
self.assertTrue(value.startswith('pfx/'))
self.assertRegex(value[4:], self.VALID_FORMAT)
def test_global_context(self):
"""Check os.environ gets updated by default."""
os.environ.pop(self.KEY, None)
self.wrapper.SetGitTrace2ParentSid()
self.assertIn(self.KEY, os.environ)
value = os.environ[self.KEY]
self.assertRegex(value, self.VALID_FORMAT)
if __name__ == '__main__':
unittest.main()

View File

@ -20,3 +20,8 @@ envlist = py27, py36, py37, py38
[testenv]
deps = pytest
commands = {toxinidir}/run_tests
[testenv:py27]
deps =
mock
pytest

View File

@ -27,7 +27,10 @@ import os
def WrapperPath():
return os.path.join(os.path.dirname(__file__), 'repo')
_wrapper_module = None
def Wrapper():
global _wrapper_module
if not _wrapper_module: