launcher: bump version for accumulated fixes

Change-Id: I45da9facb525355c4963735e087d87024dea2017
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260232
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>

.flake8

@@ -7,6 +7,8 @@ ignore=
     E114,
     # E402: Module level import not at top of file
     E402,
+    # E731: do not assign a lambda expression, use a def
+    E731,
     # W503: Line break before binary operator
     W503,
     # W504: Line break after binary operator

.github/workflows/test-ci.yml

@@ -0,0 +1,34 @@
+# GitHub actions workflow.
+# https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions
+
+name: Test CI
+
+on:
+  push:
+    branches: [master, repo-1, stable, maint]
+    tags: [v*]
+
+jobs:
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python-version: [2.7, 3.6, 3.7, 3.8]
+        exclude:
+          - os: windows-latest
+            python-version: 2.7
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v1
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install tox tox-gh-actions
+    - name: Test with tox
+      run: tox

.gitignore

@@ -1,3 +1,4 @@
+*.asc
 *.egg-info/
 *.log
 *.pyc

.mailmap

@@ -4,6 +4,7 @@ Hu Xiuyun <xiuyun.hu@hisilicon.com>           Hu xiuyun <xiuyun.hu@hisilicon.com
 Hu Xiuyun <xiuyun.hu@hisilicon.com>           Hu Xiuyun <clouds08@qq.com>
 Jelly Chen <chenguodong@huawei.com>           chenguodong <chenguodong@huawei.com>
 Jia Bi <bijia@xiaomi.com>                     bijia <bijia@xiaomi.com>
+Jiri Tyr <jiri.tyr@gmail.com>                 Jiri tyr <jiri.tyr@gmail.com>
 JoonCheol Park <jooncheol@gmail.com>          Jooncheol Park <jooncheol@gmail.com>
 Sergii Pylypenko <x.pelya.x@gmail.com>        pelya <x.pelya.x@gmail.com>
 Shawn Pearce <sop@google.com>                 Shawn O. Pearce <sop@google.com>

README.md

@@ -6,15 +6,29 @@ development workflow.  Repo is not meant to replace Git, only to make it
 easier to work with Git.  The repo command is an executable Python script
 that you can put anywhere in your path.
 
-* Homepage: https://gerrit.googlesource.com/git-repo/
-* Bug reports: https://bugs.chromium.org/p/gerrit/issues/list?q=component:repo
-* Source: https://gerrit.googlesource.com/git-repo/
-* Overview: https://source.android.com/source/developing.html
-* Docs: https://source.android.com/source/using-repo.html
+* Homepage: <https://gerrit.googlesource.com/git-repo/>
+* Mailing list: [repo-discuss on Google Groups][repo-discuss]
+* Bug reports: <https://bugs.chromium.org/p/gerrit/issues/list?q=component:repo>
+* Source: <https://gerrit.googlesource.com/git-repo/>
+* Overview: <https://source.android.com/source/developing.html>
+* Docs: <https://source.android.com/source/using-repo.html>
 * [repo Manifest Format](./docs/manifest-format.md)
 * [repo Hooks](./docs/repo-hooks.md)
 * [Submitting patches](./SUBMITTING_PATCHES.md)
 * Running Repo in [Microsoft Windows](./docs/windows.md)
+* GitHub mirror: <https://github.com/GerritCodeReview/git-repo>
+* Postsubmit tests: <https://github.com/GerritCodeReview/git-repo/actions>
+
+## Contact
+
+Please use the [repo-discuss] mailing list or [issue tracker] for questions.
+
+You can [file a new bug report][new-bug] under the "repo" component.
+
+Please do not e-mail individual developers for support.
+They do not have the bandwidth for it, and often times questions have already
+been asked on [repo-discuss] or bugs posted to the [issue tracker].
+So please search those sites first.
 
 ## Install
 
@@ -34,3 +48,8 @@ $ PATH="${HOME}/.bin:${PATH}"
 $ curl https://storage.googleapis.com/git-repo-downloads/repo > ~/.bin/repo
 $ chmod a+rx ~/.bin/repo
 ```
+
+
+[new-bug]: https://bugs.chromium.org/p/gerrit/issues/entry?template=Repo+tool+issue
+[issue tracker]: https://bugs.chromium.org/p/gerrit/issues/list?q=component:repo
+[repo-discuss]: https://groups.google.com/forum/#!forum/repo-discuss

SUBMITTING_PATCHES.md

@@ -4,7 +4,7 @@
 
  - Make small logical changes.
  - Provide a meaningful commit message.
- - Check for coding errors and style nits with pyflakes and flake8
+ - Check for coding errors and style nits with flake8.
  - Make sure all code is under the Apache License, 2.0.
  - Publish your changes for review.
  - Make corrections if requested.
@@ -38,34 +38,30 @@ If your description starts to get too long, that's a sign that you
 probably need to split up your commit to finer grained pieces.
 
 
-## Check for coding errors and style nits with pyflakes and flake8
+## Check for coding errors and style violations with flake8
 
-### Coding errors
-
-Run `pyflakes` on changed modules:
-
-    pyflakes file.py
-
-Ideally there should be no new errors or warnings introduced.
-
-### Style violations
-
-Run `flake8` on changes modules:
+Run `flake8` on changed modules:
 
     flake8 file.py
 
-Note that repo generally follows [Google's python style guide] rather than
-[PEP 8], so it's possible that the output of `flake8` will be quite noisy.
-It's not mandatory to avoid all warnings, but at least the maximum line
-length should be followed.
+Note that repo generally follows [Google's Python Style Guide] rather than
+[PEP 8], with a couple of notable exceptions:
 
-If there are many occurrences of the same warning that cannot be
-avoided without going against the Google style guide, these may be
-suppressed in the included `.flake8` file.
+* Indentation is at 2 columns rather than 4
+* The maximum line length is 100 columns rather than 80
 
-[Google's python style guide]: https://google.github.io/styleguide/pyguide.html
+There should be no new errors or warnings introduced.
+
+Warnings that cannot be avoided without going against the Google Style Guide
+may be suppressed inline individally using a `# noqa` comment as described
+in the [flake8 documentation].
+
+If there are many occurrences of the same warning, these may be suppressed for
+the entire project in the included `.flake8` file.
+
+[Google's Python Style Guide]: https://google.github.io/styleguide/pyguide.html
 [PEP 8]: https://www.python.org/dev/peps/pep-0008/
-
+[flake8 documentation]: https://flake8.pycqa.org/en/3.1.1/user/ignoring-errors.html#in-line-ignoring-errors
 
 ## Running tests
 

command.py

@@ -66,7 +66,8 @@ class Command(object):
         usage = self.helpUsage.strip().replace('%prog', me)
       except AttributeError:
         usage = 'repo %s' % self.NAME
-      self._optparse = optparse.OptionParser(usage=usage)
+      epilog = 'Run `repo help %s` to view the detailed manual.' % self.NAME
+      self._optparse = optparse.OptionParser(usage=usage, epilog=epilog)
       self._Options(self._optparse)
     return self._optparse
 

docs/internal-fs-layout.md

@@ -23,6 +23,10 @@ It is always safe to re-run `repo init` in existing repo client checkouts.
 For example, if you want to change the manifest branch, you can simply run
 `repo init --manifest-branch=<new name>` and repo will take care of the rest.
 
+*   `config`: Per-repo client checkout settings using [git-config] file format.
+*   `.repo_config.json`: JSON cache of the `config` file for repo to
+    read/process quickly.
+
 ### repo/ state
 
 *   `repo/`: A git checkout of the repo project.  This is how `repo` re-execs
@@ -30,7 +34,7 @@ For example, if you want to change the manifest branch, you can simply run
 
     It tracks the git repository at `REPO_URL` using the `REPO_REV` branch.
     Those are specified at `repo init` time using the `--repo-url=<REPO_URL>`
-    and `--repo-branch=<REPO_REV>` options.
+    and `--repo-rev=<REPO_REV>` options.
 
     Any changes made to this directory will usually be automatically discarded
     by repo itself when it checks for updates.  If you want to update to the
@@ -64,13 +68,20 @@ support, see the [manifest-format.md] file.
     If you want to switch the tracking settings, re-run `repo init` with the
     new settings.
 
+*   `manifest.xml`: The manifest that repo uses.  It is generated at `repo init`
+    and uses the `--manifest-name` to determine what manifest file to load next
+    out of `manifests/`.
+
+    Do not try to modify this to load other manifests as it will confuse repo.
+    If you want to switch manifest files, re-run `repo init` with the new
+    setting.
+
+    Older versions of repo managed this with symlinks.
+
 *   `manifest.xml -> manifests/<manifest-name>.xml`: A symlink to the manifest
     that the user wishes to sync.  It is specified at `repo init` time via
     `--manifest-name`.
 
-    Do not try to repoint this symlink to other files as it will confuse repo.
-    If you want to switch manifest files, re-run `repo init` with the new
-    setting.
 
 *   `manifests.git/.repo_config.json`: JSON cache of the `manifests.git/config`
     file for repo to read/process quickly.
@@ -92,18 +103,27 @@ support, see the [manifest-format.md] file.
     Some git state is further split out under `project-objects/`.
 *   `project-objects/`: Git objects that are safe to share across multiple
     git checkouts.  The filesystem layout matches the `<project name=...`
-    setting in the manifest (i.e. the path on the remote server).  This allows
-    for multiple checkouts of the same remote git repo to share their objects.
-    For example, you could have different branches of `foo/bar.git` checked
-    out to `foo/bar-master`, `foo/bar-release`, etc...  There will be multiple
-    trees under `projects/` for each one, but only one under `project-objects/`.
+    setting in the manifest (i.e. the path on the remote server) with a `.git`
+    suffix.  This allows for multiple checkouts of the same remote git repo to
+    share their objects.  For example, you could have different branches of
+    `foo/bar.git` checked out to `foo/bar-master`, `foo/bar-release`, etc...
+    There will be multiple trees under `projects/` for each one, but only one
+    under `project-objects/`.
 
-    This can run into problems if different remotes use the same path on their
-    respective servers ...
+    This layout is designed to allow people to sync against different remotes
+    (e.g. a local mirror & a public review server) while avoiding duplicating
+    the content.  However, this can run into problems if different remotes use
+    the same path on their respective servers.  Best to avoid that.
 *   `subprojects/`: Like `projects/`, but for git submodules.
 *   `subproject-objects/`: Like `project-objects/`, but for git submodules.
+*   `worktrees/`: Bare checkouts of every project synced by the manifest.  The
+    filesystem layout matches the `<project name=...` setting in the manifest
+    (i.e. the path on the remote server) with a `.git` suffix.  This has the
+    same advantages as the `project-objects/` layout above.
 
-### Settings
+    This is used when git worktrees are enabled.
+
+### Global settings
 
 The `.repo/manifests.git/config` file is used to track settings for the entire
 repo client checkout.
@@ -121,25 +141,91 @@ User controlled settings are initialized when running `repo init`.
 | repo.partialclone | `--partial-clone`         | Create [partial git clones] |
 | repo.reference    | `--reference`             | Reference repo client checkout |
 | repo.submodules   | `--submodules`            | Sync git submodules |
+| repo.worktree     | `--worktree`              | Use `git worktree` for checkouts |
 | user.email        | `--config-name`           | User's e-mail address; Copied into `.git/config` when checking out a new project |
 | user.name         | `--config-name`           | User's name; Copied into `.git/config` when checking out a new project |
 
 [partial git clones]: https://git-scm.com/docs/gitrepository-layout#_code_partialclone_code
 
+### Repo hooks settings
+
+For more details on this feature, see the [repo-hooks docs](./repo-hooks.md).
+We'll just discuss the internal configuration settings.
+These are stored in the registered `<repo-hooks>` project itself, so if the
+manifest switches to a different project, the settings will not be copied.
+
+| Setting                              | Use/Meaning |
+|--------------------------------------|-------------|
+| repo.hooks.\<hook\>.approvedmanifest | User approval for secure manifest sources (e.g. https://) |
+| repo.hooks.\<hook\>.approvedhash     | User approval for insecure manifest sources (e.g. http://) |
+
+
+For example, if our manifest had the following entries, we would store settings
+under `.repo/projects/src/repohooks.git/config` (which would be reachable via
+`git --git-dir=src/repohooks/.git config`).
+```xml
+  <project path="src/repohooks" name="chromiumos/repohooks" ... />
+  <repo-hooks in-project="chromiumos/repohooks" ... />
+```
+
+If `<hook>` is `pre-upload`, the `.git/config` setting might be:
+```ini
+[repo "hooks.pre-upload"]
+	approvedmanifest = https://chromium.googlesource.com/chromiumos/manifest
+```
+
+## Per-project settings
+
+These settings are somewhat meant to be tweaked by the user on a per-project
+basis (e.g. `git config` in a checked out source repo).
+
+Where possible, we re-use standard git settings to avoid confusion, and we
+refrain from documenting those, so see [git-config] documentation instead.
+
+See `repo help upload` for documentation on `[review]` settings.
+
+The `[remote]` settings are automatically populated/updated from the manifest.
+
+The `[branch]` settings are updated by `repo start` and `git branch`.
+
+| Setting                       | Subcommands   | Use/Meaning |
+|-------------------------------|---------------|-------------|
+| review.\<url\>.autocopy       | upload        | Automatically add to `--cc=<value>` |
+| review.\<url\>.autoreviewer   | upload        | Automatically add to `--reviewers=<value>` |
+| review.\<url\>.autoupload     | upload        | Automatically answer "yes" or "no" to all prompts |
+| review.\<url\>.uploadhashtags | upload        | Automatically add to `--hashtag=<value>` |
+| review.\<url\>.uploadlabels   | upload        | Automatically add to `--label=<value>` |
+| review.\<url\>.uploadnotify   | upload        | [Notify setting][upload-notify] to use |
+| review.\<url\>.uploadtopic    | upload        | Default [topic] to use |
+| review.\<url\>.username       | upload        | Override username with `ssh://` review URIs |
+| remote.\<remote\>.fetch       | sync          | Set of refs to fetch |
+| remote.\<remote\>.projectname | \<network\>   | The name of the project as it exists in Gerrit review |
+| remote.\<remote\>.pushurl     | upload        | The base URI for pushing CLs |
+| remote.\<remote\>.review      | upload        | The URI of the Gerrit review server |
+| remote.\<remote\>.url         | sync & upload | The URI of the git project to fetch |
+| branch.\<branch\>.merge       | sync & upload | The branch to merge & upload & track |
+| branch.\<branch\>.remote      | sync & upload | The remote to track |
+
 ## ~/ dotconfig layout
 
 Repo will create & maintain a few files in the user's home directory.
 
 *   `.repoconfig/`: Repo's per-user directory for all random config files/state.
+*   `.repoconfig/config`: Per-user settings using [git-config] file format.
 *   `.repoconfig/keyring-version`: Cache file for checking if the gnupg subdir
     has all the same keys as the repo launcher.  Used to avoid running gpg
     constantly as that can be quite slow.
 *   `.repoconfig/gnupg/`: GnuPG's internal state directory used when repo needs
     to run `gpg`.  This provides isolation from the user's normal `~/.gnupg/`.
 
+*   `.repoconfig/.repo_config.json`: JSON cache of the `.repoconfig/config`
+    file for repo to read/process quickly.
 *   `.repo_.gitconfig.json`: JSON cache of the `.gitconfig` file for repo to
     read/process quickly.
 
 
+[git-config]: https://git-scm.com/docs/git-config
 [manifest-format.md]: ./manifest-format.md
 [local manifests]: ./manifest-format.md#Local-Manifests
+[topic]: https://gerrit-review.googlesource.com/Documentation/intro-user.html#topics
+[upload-notify]: https://gerrit-review.googlesource.com/Documentation/user-upload.html#notify

docs/release-process.md

@@ -49,11 +49,11 @@ control how repo finds updates:
 
 *   `--repo-url`: This tells repo where to clone the full repo project itself.
     It defaults to the official project (`REPO_URL` in the launcher script).
-*   `--repo-branch`: This tells repo which branch to use for the full project.
+*   `--repo-rev`: This tells repo which branch to use for the full project.
     It defaults to the `stable` branch (`REPO_REV` in the launcher script).
 
 Whenever `repo sync` is run, repo will check to see if an update is available.
-It fetches the latest repo-branch from the repo-url.
+It fetches the latest repo-rev from the repo-url.
 Then it verifies that the latest commit in the branch has a valid signed tag
 using `git tag -v` (which uses gpg).
 If the tag is valid, then repo will update its internal checkout to it.
@@ -91,7 +91,7 @@ When you want to create a new release, you'll need to select a good version and
 create a signed tag using a key registered in repo itself.
 Typically we just tag the latest version of the `master` branch.
 The tag could be pushed now, but it won't be used by clients normally (since the
-default `repo-branch` setting is `stable`).
+default `repo-rev` setting is `stable`).
 This would allow some early testing on systems who explicitly select `master`.
 
 ### Creating a signed tag

docs/windows.md

@@ -19,7 +19,33 @@ also due to most developers not using Windows.
 We will never add code specific to older versions of Windows.
 It might work, but it most likely won't, so please don't bother asking.
 
-## Symlinks
+## Git worktrees
+
+*** note
+**Warning**: Repo's support for Git worktrees is new & experimental.
+Please report any bugs and be sure to maintain backups!
+***
+
+The Repo 2.4 release introduced support for [Git worktrees][git-worktree].
+You don't have to worry about or understand this particular feature, so don't
+worry if this section of the Git manual is particularly impenetrable.
+
+The salient point is that Git worktrees allow Repo to create repo client
+checkouts that do not require symlinks at all under Windows.
+This means users no longer need Administrator access to sync code.
+
+Simply use `--worktree` when running `repo init` to opt in.
+
+This does not effect specific Git repositories that use symlinks themselves.
+
+[git-worktree]: https://git-scm.com/docs/git-worktree
+
+## Symlinks by default
+
+*** note
+**NB**: This section applies to the default Repo behavior which does not use
+Git worktrees (see the previous section for more info).
+***
 
 Repo will use symlinks heavily internally.
 On *NIX platforms, this isn't an issue, but Windows makes it a bit difficult.
@@ -62,9 +88,8 @@ This also helps `tar` unpack symlinks, so that's nice.
 
 ## Python
 
-You should make sure to be running Python 3.6 or newer under Windows.
-Python 2 might work, but due to already limited platform testing, you should
-only run newer Python versions.
+Python 3.6 or newer is required.
+Python 2 is known to be broken when running under Windows.
 See our [Python Support](./python-support.md) document for more details.
 
 You can grab the latest Windows installer here:<br>

git_command.py

@@ -214,10 +214,6 @@ def git_require(min_version, fail=False, msg=''):
   return False
 
 
-def _setenv(env, name, value):
-  env[name] = value.encode()
-
-
 class GitCommand(object):
   def __init__(self,
                project,
@@ -226,6 +222,7 @@ class GitCommand(object):
                provide_stdin=False,
                capture_stdout=False,
                capture_stderr=False,
+               merge_output=False,
                disable_editor=False,
                ssh_proxy=False,
                cwd=None,
@@ -236,21 +233,21 @@ class GitCommand(object):
     self.tee = {'stdout': not capture_stdout, 'stderr': not capture_stderr}
 
     if disable_editor:
-      _setenv(env, 'GIT_EDITOR', ':')
+      env['GIT_EDITOR'] = ':'
     if ssh_proxy:
-      _setenv(env, 'REPO_SSH_SOCK', ssh_sock())
-      _setenv(env, 'GIT_SSH', _ssh_proxy())
-      _setenv(env, 'GIT_SSH_VARIANT', 'ssh')
+      env['REPO_SSH_SOCK'] = ssh_sock()
+      env['GIT_SSH'] = _ssh_proxy()
+      env['GIT_SSH_VARIANT'] = 'ssh'
     if 'http_proxy' in env and 'darwin' == sys.platform:
       s = "'http.proxy=%s'" % (env['http_proxy'],)
       p = env.get('GIT_CONFIG_PARAMETERS')
       if p is not None:
         s = p + ' ' + s
-      _setenv(env, 'GIT_CONFIG_PARAMETERS', s)
+      env['GIT_CONFIG_PARAMETERS'] = s
     if 'GIT_ALLOW_PROTOCOL' not in env:
-      _setenv(env, 'GIT_ALLOW_PROTOCOL',
-              'file:git:http:https:ssh:persistent-http:persistent-https:sso:rpc')
-    _setenv(env, 'GIT_HTTP_USER_AGENT', user_agent.git)
+      env['GIT_ALLOW_PROTOCOL'] = (
+          'file:git:http:https:ssh:persistent-http:persistent-https:sso:rpc')
+    env['GIT_HTTP_USER_AGENT'] = user_agent.git
 
     if project:
       if not cwd:
@@ -261,7 +258,7 @@ class GitCommand(object):
     command = [GIT]
     if bare:
       if gitdir:
-        _setenv(env, GIT_DIR, gitdir)
+        env[GIT_DIR] = gitdir
       cwd = None
     command.append(cmdv[0])
     # Need to use the --progress flag for fetch/clone so output will be
@@ -277,7 +274,7 @@ class GitCommand(object):
       stdin = None
 
     stdout = subprocess.PIPE
-    stderr = subprocess.PIPE
+    stderr = subprocess.STDOUT if merge_output else subprocess.PIPE
 
     if IsTrace():
       global LAST_CWD
@@ -305,6 +302,8 @@ class GitCommand(object):
         dbg += ' 1>|'
       if stderr == subprocess.PIPE:
         dbg += ' 2>|'
+      elif stderr == subprocess.STDOUT:
+        dbg += ' 2>&1'
       Trace('%s', dbg)
 
     try:
@@ -352,7 +351,8 @@ class GitCommand(object):
     p = self.process
     s_in = platform_utils.FileDescriptorStreams.create()
     s_in.add(p.stdout, sys.stdout, 'stdout')
-    s_in.add(p.stderr, sys.stderr, 'stderr')
+    if p.stderr is not None:
+      s_in.add(p.stderr, sys.stderr, 'stderr')
     self.stdout = ''
     self.stderr = ''
 

git_config.py

@@ -21,6 +21,7 @@ import errno
 import json
 import os
 import re
+import signal
 import ssl
 import subprocess
 import sys
@@ -41,7 +42,6 @@ else:
   urllib.request = urllib2
   urllib.error = urllib2
 
-from signal import SIGTERM
 from error import GitError, UploadError
 import platform_utils
 from repo_trace import Trace
@@ -88,10 +88,12 @@ def _key(name):
 class GitConfig(object):
   _ForUser = None
 
+  _USER_CONFIG = '~/.gitconfig'
+
   @classmethod
   def ForUser(cls):
     if cls._ForUser is None:
-      cls._ForUser = cls(configfile=os.path.expanduser('~/.gitconfig'))
+      cls._ForUser = cls(configfile=os.path.expanduser(cls._USER_CONFIG))
     return cls._ForUser
 
   @classmethod
@@ -122,6 +124,43 @@ class GitConfig(object):
       return self.defaults.Has(name, include_defaults=True)
     return False
 
+  def GetInt(self, name):
+    """Returns an integer from the configuration file.
+
+    This follows the git config syntax.
+
+    Args:
+      name: The key to lookup.
+
+    Returns:
+      None if the value was not defined, or is not a boolean.
+      Otherwise, the number itself.
+    """
+    v = self.GetString(name)
+    if v is None:
+      return None
+    v = v.strip()
+
+    mult = 1
+    if v.endswith('k'):
+      v = v[:-1]
+      mult = 1024
+    elif v.endswith('m'):
+      v = v[:-1]
+      mult = 1024 * 1024
+    elif v.endswith('g'):
+      v = v[:-1]
+      mult = 1024 * 1024 * 1024
+
+    base = 10
+    if v.startswith('0x'):
+      base = 16
+
+    try:
+      return int(v, base=base) * mult
+    except ValueError:
+      return None
+
   def GetBoolean(self, name):
     """Returns a boolean from the configuration file.
        None : The value was not defined, or is not a boolean.
@@ -336,6 +375,12 @@ class GitConfig(object):
       GitError('git config %s: %s' % (str(args), p.stderr))
 
 
+class RepoConfig(GitConfig):
+  """User settings for repo itself."""
+
+  _USER_CONFIG = '~/.repoconfig/config'
+
+
 class RefSpec(object):
   """A Git refspec line, split into its components:
 
@@ -494,7 +539,7 @@ def close_ssh():
 
   for p in _master_processes:
     try:
-      os.kill(p.pid, SIGTERM)
+      os.kill(p.pid, signal.SIGTERM)
       p.wait()
     except OSError:
       pass

git_refs.py

@@ -23,6 +23,8 @@ R_CHANGES = 'refs/changes/'
 R_HEADS = 'refs/heads/'
 R_TAGS = 'refs/tags/'
 R_PUB = 'refs/published/'
+R_WORKTREE = 'refs/worktree/'
+R_WORKTREE_M = R_WORKTREE + 'm/'
 R_M = 'refs/remotes/m/'
 
 

hooks/commit-msg

@@ -1,5 +1,5 @@
 #!/bin/sh
-# From Gerrit Code Review 2.14.6
+# From Gerrit Code Review 3.1.3
 #
 # Part of Gerrit Code Review (https://www.gerritcodereview.com/)
 #
@@ -16,176 +16,48 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-#
 
-unset GREP_OPTIONS
+# avoid [[ which is not POSIX sh.
+if test "$#" != 1 ; then
+  echo "$0 requires an argument."
+  exit 1
+fi
 
-CHANGE_ID_AFTER="Bug|Depends-On|Issue|Test|Feature|Fixes|Fixed"
-MSG="$1"
+if test ! -f "$1" ; then
+  echo "file does not exist: $1"
+  exit 1
+fi
 
-# Check for, and add if missing, a unique Change-Id
-#
-add_ChangeId() {
-	clean_message=`sed -e '
-		/^diff --git .*/{
-			s///
-			q
-		}
-		/^Signed-off-by:/d
-		/^#/d
-	' "$MSG" | git stripspace`
-	if test -z "$clean_message"
-	then
-		return
-	fi
+# Do not create a change id if requested
+if test "false" = "`git config --bool --get gerrit.createChangeId`" ; then
+  exit 0
+fi
 
-	# Do not add Change-Id to temp commits
-	if echo "$clean_message" | head -1 | grep -q '^\(fixup\|squash\)!'
-	then
-		return
-	fi
+# $RANDOM will be undefined if not using bash, so don't use set -u
+random=$( (whoami ; hostname ; date; cat $1 ; echo $RANDOM) | git hash-object --stdin)
+dest="$1.tmp.${random}"
 
-	if test "false" = "`git config --bool --get gerrit.createChangeId`"
-	then
-		return
-	fi
+trap 'rm -f "${dest}"' EXIT
 
-	# Does Change-Id: already exist? if so, exit (no change).
-	if grep -i '^Change-Id:' "$MSG" >/dev/null
-	then
-		return
-	fi
+if ! git stripspace --strip-comments < "$1" > "${dest}" ; then
+   echo "cannot strip comments from $1"
+   exit 1
+fi
 
-	id=`_gen_ChangeId`
-	T="$MSG.tmp.$$"
-	AWK=awk
-	if [ -x /usr/xpg4/bin/awk ]; then
-		# Solaris AWK is just too broken
-		AWK=/usr/xpg4/bin/awk
-	fi
+if test ! -s "${dest}" ; then
+  echo "file is empty: $1"
+  exit 1
+fi
 
-	# Get core.commentChar from git config or use default symbol
-	commentChar=`git config --get core.commentChar`
-	commentChar=${commentChar:-#}
+# Avoid the --in-place option which only appeared in Git 2.8
+# Avoid the --if-exists option which only appeared in Git 2.15
+if ! git -c trailer.ifexists=doNothing interpret-trailers \
+      --trailer "Change-Id: I${random}" < "$1" > "${dest}" ; then
+  echo "cannot insert change-id line in $1"
+  exit 1
+fi
 
-	# How this works:
-	# - parse the commit message as (textLine+ blankLine*)*
-	# - assume textLine+ to be a footer until proven otherwise
-	# - exception: the first block is not footer (as it is the title)
-	# - read textLine+ into a variable
-	# - then count blankLines
-	# - once the next textLine appears, print textLine+ blankLine* as these
-	#   aren't footer
-	# - in END, the last textLine+ block is available for footer parsing
-	$AWK '
-	BEGIN {
-		# while we start with the assumption that textLine+
-		# is a footer, the first block is not.
-		isFooter = 0
-		footerComment = 0
-		blankLines = 0
-	}
-
-	# Skip lines starting with commentChar without any spaces before it.
-	/^'"$commentChar"'/ { next }
-
-	# Skip the line starting with the diff command and everything after it,
-	# up to the end of the file, assuming it is only patch data.
-	# If more than one line before the diff was empty, strip all but one.
-	/^diff --git / {
-		blankLines = 0
-		while (getline) { }
-		next
-	}
-
-	# Count blank lines outside footer comments
-	/^$/ && (footerComment == 0) {
-		blankLines++
-		next
-	}
-
-	# Catch footer comment
-	/^\[[a-zA-Z0-9-]+:/ && (isFooter == 1) {
-		footerComment = 1
-	}
-
-	/]$/ && (footerComment == 1) {
-		footerComment = 2
-	}
-
-	# We have a non-blank line after blank lines. Handle this.
-	(blankLines > 0) {
-		print lines
-		for (i = 0; i < blankLines; i++) {
-			print ""
-		}
-
-		lines = ""
-		blankLines = 0
-		isFooter = 1
-		footerComment = 0
-	}
-
-	# Detect that the current block is not the footer
-	(footerComment == 0) && (!/^\[?[a-zA-Z0-9-]+:/ || /^[a-zA-Z0-9-]+:\/\//) {
-		isFooter = 0
-	}
-
-	{
-		# We need this information about the current last comment line
-		if (footerComment == 2) {
-			footerComment = 0
-		}
-		if (lines != "") {
-			lines = lines "\n";
-		}
-		lines = lines $0
-	}
-
-	# Footer handling:
-	# If the last block is considered a footer, splice in the Change-Id at the
-	# right place.
-	# Look for the right place to inject Change-Id by considering
-	# CHANGE_ID_AFTER. Keys listed in it (case insensitive) come first,
-	# then Change-Id, then everything else (eg. Signed-off-by:).
-	#
-	# Otherwise just print the last block, a new line and the Change-Id as a
-	# block of its own.
-	END {
-		unprinted = 1
-		if (isFooter == 0) {
-			print lines "\n"
-			lines = ""
-		}
-		changeIdAfter = "^(" tolower("'"$CHANGE_ID_AFTER"'") "):"
-		numlines = split(lines, footer, "\n")
-		for (line = 1; line <= numlines; line++) {
-			if (unprinted && match(tolower(footer[line]), changeIdAfter) != 1) {
-				unprinted = 0
-				print "Change-Id: I'"$id"'"
-			}
-			print footer[line]
-		}
-		if (unprinted) {
-			print "Change-Id: I'"$id"'"
-		}
-	}' "$MSG" > "$T" && mv "$T" "$MSG" || rm -f "$T"
-}
-_gen_ChangeIdInput() {
-	echo "tree `git write-tree`"
-	if parent=`git rev-parse "HEAD^0" 2>/dev/null`
-	then
-		echo "parent $parent"
-	fi
-	echo "author `git var GIT_AUTHOR_IDENT`"
-	echo "committer `git var GIT_COMMITTER_IDENT`"
-	echo
-	printf '%s' "$clean_message"
-}
-_gen_ChangeId() {
-	_gen_ChangeIdInput |
-	git hash-object -t commit --stdin
-}
-
-
-add_ChangeId
+if ! mv "${dest}" "$1" ; then
+  echo "cannot mv ${dest} to $1"
+  exit 1
+fi

main.py

@@ -26,6 +26,7 @@ import getpass
 import netrc
 import optparse
 import os
+import shlex
 import sys
 import textwrap
 import time
@@ -48,7 +49,7 @@ from color import SetDefaultColoring
 import event_log
 from repo_trace import SetTrace
 from git_command import user_agent
-from git_config import init_ssh, close_ssh
+from git_config import init_ssh, close_ssh, RepoConfig
 from command import InteractiveCommand
 from command import MirrorSafeCommand
 from command import GitcAvailableCommand, GitcClientCommand
@@ -69,7 +70,35 @@ from wrapper import WrapperPath, Wrapper
 from subcmds import all_commands
 
 if not is_python3():
-  input = raw_input
+  input = raw_input  # noqa: F821
+
+# NB: These do not need to be kept in sync with the repo launcher script.
+# These may be much newer as it allows the repo launcher to roll between
+# different repo releases while source versions might require a newer python.
+#
+# The soft version is when we start warning users that the version is old and
+# we'll be dropping support for it.  We'll refuse to work with versions older
+# than the hard version.
+#
+# python-3.6 is in Ubuntu Bionic.
+MIN_PYTHON_VERSION_SOFT = (3, 6)
+MIN_PYTHON_VERSION_HARD = (3, 4)
+
+if sys.version_info.major < 3:
+  print('repo: warning: Python 2 is no longer supported; '
+        'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION_SOFT),
+        file=sys.stderr)
+else:
+  if sys.version_info < MIN_PYTHON_VERSION_HARD:
+    print('repo: error: Python 3 version is too old; '
+          'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION_SOFT),
+          file=sys.stderr)
+    sys.exit(1)
+  elif sys.version_info < MIN_PYTHON_VERSION_SOFT:
+    print('repo: warning: your Python 3 version is no longer supported; '
+          'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION_SOFT),
+          file=sys.stderr)
+
 
 global_options = optparse.OptionParser(
     usage='repo [-p|--paginate|--no-pager] COMMAND [ARGS]',
@@ -80,7 +109,7 @@ global_options.add_option('-p', '--paginate',
                           dest='pager', action='store_true',
                           help='display command output in the pager')
 global_options.add_option('--no-pager',
-                          dest='no_pager', action='store_true',
+                          dest='pager', action='store_false',
                           help='disable the pager')
 global_options.add_option('--color',
                           choices=('auto', 'always', 'never'), default=None,
@@ -106,8 +135,6 @@ class _Repo(object):
   def __init__(self, repodir):
     self.repodir = repodir
     self.commands = all_commands
-    # add 'branch' as an alias for 'branches'
-    all_commands['branch'] = all_commands['branches']
 
   def _ParseArgs(self, argv):
     """Parse the main `repo` command line options."""
@@ -127,6 +154,9 @@ class _Repo(object):
       argv = []
     gopts, _gargs = global_options.parse_args(glob)
 
+    name, alias_args = self._ExpandAlias(name)
+    argv = alias_args + argv
+
     if gopts.help:
       global_options.print_help()
       commands = ' '.join(sorted(self.commands))
@@ -137,6 +167,27 @@ class _Repo(object):
 
     return (name, gopts, argv)
 
+  def _ExpandAlias(self, name):
+    """Look up user registered aliases."""
+    # We don't resolve aliases for existing subcommands.  This matches git.
+    if name in self.commands:
+      return name, []
+
+    key = 'alias.%s' % (name,)
+    alias = RepoConfig.ForRepository(self.repodir).GetString(key)
+    if alias is None:
+      alias = RepoConfig.ForUser().GetString(key)
+    if alias is None:
+      return name, []
+
+    args = alias.strip().split(' ', 1)
+    name = args[0]
+    if len(args) == 2:
+      args = shlex.split(args[1])
+    else:
+      args = []
+    return name, args
+
   def _Run(self, name, gopts, argv):
     """Execute the requested subcommand."""
     result = 0
@@ -153,7 +204,7 @@ class _Repo(object):
     SetDefaultColoring(gopts.color)
 
     try:
-      cmd = self.commands[name]
+      cmd = self.commands[name]()
     except KeyError:
       print("repo: '%s' is not a repo command.  See 'repo help'." % name,
             file=sys.stderr)
@@ -194,7 +245,7 @@ class _Repo(object):
             file=sys.stderr)
       return 1
 
-    if not gopts.no_pager and not isinstance(cmd, InteractiveCommand):
+    if gopts.pager is not False and not isinstance(cmd, InteractiveCommand):
       config = cmd.manifest.globalConfig
       if gopts.pager:
         use_pager = True
@@ -229,7 +280,8 @@ class _Repo(object):
       if e.name:
         print('error: project group must be enabled for project %s' % e.name, file=sys.stderr)
       else:
-        print('error: project group must be enabled for the project in the current directory', file=sys.stderr)
+        print('error: project group must be enabled for the project in the current directory',
+              file=sys.stderr)
       result = 1
     except SystemExit as e:
       if e.code:
@@ -286,7 +338,7 @@ def _CheckWrapperVersion(ver_str, repo_path):
 repo: error:
 !!! Your version of repo %s is too old.
 !!! We need at least version %s.
-!!! A new repo command (%s) is available.
+!!! A new version of repo (%s) is available.
 !!! You must upgrade before you can continue:
 
     cp %s %s
@@ -294,12 +346,20 @@ repo: error:
     sys.exit(1)
 
   if exp > ver:
-    print("""
-... A new repo command (%5s) is available.
+    print('\n... A new version of repo (%s) is available.' % (exp_str,),
+          file=sys.stderr)
+    if os.access(repo_path, os.W_OK):
+      print("""\
 ... You should upgrade soon:
-
     cp %s %s
-""" % (exp_str, WrapperPath(), repo_path), file=sys.stderr)
+""" % (WrapperPath(), repo_path), file=sys.stderr)
+    else:
+      print("""\
+... New version is available at: %s
+... The launcher is run from: %s
+!!! The launcher is not writable.  Please talk to your sysadmin or distro
+!!! to get an update installed.
+""" % (WrapperPath(), repo_path), file=sys.stderr)
 
 
 def _CheckRepoDir(repo_dir):

manifest_xml.py

@@ -57,6 +57,60 @@ urllib.parse.uses_netloc.extend([
     'rpc'])
 
 
+def XmlBool(node, attr, default=None):
+  """Determine boolean value of |node|'s |attr|.
+
+  Invalid values will issue a non-fatal warning.
+
+  Args:
+    node: XML node whose attributes we access.
+    attr: The attribute to access.
+    default: If the attribute is not set (value is empty), then use this.
+
+  Returns:
+    True if the attribute is a valid string representing true.
+    False if the attribute is a valid string representing false.
+    |default| otherwise.
+  """
+  value = node.getAttribute(attr)
+  s = value.lower()
+  if s == '':
+    return default
+  elif s in {'yes', 'true', '1'}:
+    return True
+  elif s in {'no', 'false', '0'}:
+    return False
+  else:
+    print('warning: manifest: %s="%s": ignoring invalid XML boolean' %
+          (attr, value), file=sys.stderr)
+    return default
+
+
+def XmlInt(node, attr, default=None):
+  """Determine integer value of |node|'s |attr|.
+
+  Args:
+    node: XML node whose attributes we access.
+    attr: The attribute to access.
+    default: If the attribute is not set (value is empty), then use this.
+
+  Returns:
+    The number if the attribute is a valid number.
+
+  Raises:
+    ManifestParseError: The number is invalid.
+  """
+  value = node.getAttribute(attr)
+  if not value:
+    return default
+
+  try:
+    return int(value)
+  except ValueError:
+    raise ManifestParseError('manifest: invalid %s="%s" integer' %
+                             (attr, value))
+
+
 class _Default(object):
   """Project defaults within the manifest."""
 
@@ -146,9 +200,17 @@ class XmlManifest(object):
                                    gitdir=os.path.join(repodir, 'repo/.git'),
                                    worktree=os.path.join(repodir, 'repo'))
 
-    self.manifestProject = MetaProject(self, 'manifests',
-                                       gitdir=os.path.join(repodir, 'manifests.git'),
-                                       worktree=os.path.join(repodir, 'manifests'))
+    mp = MetaProject(self, 'manifests',
+                     gitdir=os.path.join(repodir, 'manifests.git'),
+                     worktree=os.path.join(repodir, 'manifests'))
+    self.manifestProject = mp
+
+    # This is a bit hacky, but we're in a chicken & egg situation: all the
+    # normal repo settings live in the manifestProject which we just setup
+    # above, so we couldn't easily query before that.  We assume Project()
+    # init doesn't care if this changes afterwards.
+    if os.path.exists(mp.gitdir) and mp.config.GetBoolean('repo.worktree'):
+      mp.use_git_worktrees = True
 
     self._Unload()
 
@@ -183,12 +245,27 @@ class XmlManifest(object):
     """
     self.Override(name)
 
-    try:
-      if os.path.lexists(self.manifestFile):
-        platform_utils.remove(self.manifestFile)
-      platform_utils.symlink(os.path.join('manifests', name), self.manifestFile)
-    except OSError as e:
-      raise ManifestParseError('cannot link manifest %s: %s' % (name, str(e)))
+    # Old versions of repo would generate symlinks we need to clean up.
+    if os.path.lexists(self.manifestFile):
+      platform_utils.remove(self.manifestFile)
+    # This file is interpreted as if it existed inside the manifest repo.
+    # That allows us to use <include> with the relative file name.
+    with open(self.manifestFile, 'w') as fp:
+      fp.write("""<?xml version="1.0" encoding="UTF-8"?>
+<!--
+DO NOT EDIT THIS FILE!  It is generated by repo and changes will be discarded.
+If you want to use a different manifest, use `repo init -m <file>` instead.
+
+If you want to customize your checkout by overriding manifest settings, use
+the local_manifests/ directory instead.
+
+For more information on repo manifests, check out:
+https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
+-->
+<manifest>
+  <include name="%s" />
+</manifest>
+""" % (name,))
 
   def _RemoteToXml(self, r, doc, root):
     e = doc.createElement('remote')
@@ -427,6 +504,10 @@ class XmlManifest(object):
   def IsMirror(self):
     return self.manifestProject.config.GetBoolean('repo.mirror')
 
+  @property
+  def UseGitWorktrees(self):
+    return self.manifestProject.config.GetBoolean('repo.worktree')
+
   @property
   def IsArchive(self):
     return self.manifestProject.config.GetBoolean('repo.archive')
@@ -730,29 +811,14 @@ class XmlManifest(object):
     d.destBranchExpr = node.getAttribute('dest-branch') or None
     d.upstreamExpr = node.getAttribute('upstream') or None
 
-    sync_j = node.getAttribute('sync-j')
-    if sync_j == '' or sync_j is None:
-      d.sync_j = 1
-    else:
-      d.sync_j = int(sync_j)
+    d.sync_j = XmlInt(node, 'sync-j', 1)
+    if d.sync_j <= 0:
+      raise ManifestParseError('%s: sync-j must be greater than 0, not "%s"' %
+                               (self.manifestFile, d.sync_j))
 
-    sync_c = node.getAttribute('sync-c')
-    if not sync_c:
-      d.sync_c = False
-    else:
-      d.sync_c = sync_c.lower() in ("yes", "true", "1")
-
-    sync_s = node.getAttribute('sync-s')
-    if not sync_s:
-      d.sync_s = False
-    else:
-      d.sync_s = sync_s.lower() in ("yes", "true", "1")
-
-    sync_tags = node.getAttribute('sync-tags')
-    if not sync_tags:
-      d.sync_tags = True
-    else:
-      d.sync_tags = sync_tags.lower() in ("yes", "true", "1")
+    d.sync_c = XmlBool(node, 'sync-c', False)
+    d.sync_s = XmlBool(node, 'sync-s', False)
+    d.sync_tags = XmlBool(node, 'sync-tags', True)
     return d
 
   def _ParseNotice(self, node):
@@ -829,39 +895,15 @@ class XmlManifest(object):
       raise ManifestParseError("project %s path cannot be absolute in %s" %
                                (name, self.manifestFile))
 
-    rebase = node.getAttribute('rebase')
-    if not rebase:
-      rebase = True
-    else:
-      rebase = rebase.lower() in ("yes", "true", "1")
+    rebase = XmlBool(node, 'rebase', True)
+    sync_c = XmlBool(node, 'sync-c', False)
+    sync_s = XmlBool(node, 'sync-s', self._default.sync_s)
+    sync_tags = XmlBool(node, 'sync-tags', self._default.sync_tags)
 
-    sync_c = node.getAttribute('sync-c')
-    if not sync_c:
-      sync_c = False
-    else:
-      sync_c = sync_c.lower() in ("yes", "true", "1")
-
-    sync_s = node.getAttribute('sync-s')
-    if not sync_s:
-      sync_s = self._default.sync_s
-    else:
-      sync_s = sync_s.lower() in ("yes", "true", "1")
-
-    sync_tags = node.getAttribute('sync-tags')
-    if not sync_tags:
-      sync_tags = self._default.sync_tags
-    else:
-      sync_tags = sync_tags.lower() in ("yes", "true", "1")
-
-    clone_depth = node.getAttribute('clone-depth')
-    if clone_depth:
-      try:
-        clone_depth = int(clone_depth)
-        if clone_depth <= 0:
-          raise ValueError()
-      except ValueError:
-        raise ManifestParseError('invalid clone-depth %s in %s' %
-                                 (clone_depth, self.manifestFile))
+    clone_depth = XmlInt(node, 'clone-depth')
+    if clone_depth is not None and clone_depth <= 0:
+      raise ManifestParseError('%s: clone-depth must be greater than 0, not "%s"' %
+                               (self.manifestFile, clone_depth))
 
     dest_branch = node.getAttribute('dest-branch') or self._default.destBranchExpr
 
@@ -873,8 +915,10 @@ class XmlManifest(object):
     groups = self._ParseGroups(groups)
 
     if parent is None:
-      relpath, worktree, gitdir, objdir = self.GetProjectPaths(name, path)
+      relpath, worktree, gitdir, objdir, use_git_worktrees = \
+          self.GetProjectPaths(name, path)
     else:
+      use_git_worktrees = False
       relpath, worktree, gitdir, objdir = \
           self.GetSubprojectPaths(parent, name, path)
 
@@ -882,7 +926,7 @@ class XmlManifest(object):
     groups.extend(set(default_groups).difference(groups))
 
     if self.IsMirror and node.hasAttribute('force-path'):
-      if node.getAttribute('force-path').lower() in ("yes", "true", "1"):
+      if XmlBool(node, 'force-path', False):
         gitdir = os.path.join(self.topdir, '%s.git' % path)
 
     project = Project(manifest=self,
@@ -903,6 +947,7 @@ class XmlManifest(object):
                       upstream=upstream,
                       parent=parent,
                       dest_branch=dest_branch,
+                      use_git_worktrees=use_git_worktrees,
                       **extra_proj_attrs)
 
     for n in node.childNodes:
@@ -918,6 +963,7 @@ class XmlManifest(object):
     return project
 
   def GetProjectPaths(self, name, path):
+    use_git_worktrees = False
     relpath = path
     if self.IsMirror:
       worktree = None
@@ -926,8 +972,15 @@ class XmlManifest(object):
     else:
       worktree = os.path.join(self.topdir, path).replace('\\', '/')
       gitdir = os.path.join(self.repodir, 'projects', '%s.git' % path)
-      objdir = os.path.join(self.repodir, 'project-objects', '%s.git' % name)
-    return relpath, worktree, gitdir, objdir
+      # We allow people to mix git worktrees & non-git worktrees for now.
+      # This allows for in situ migration of repo clients.
+      if os.path.exists(gitdir) or not self.UseGitWorktrees:
+        objdir = os.path.join(self.repodir, 'project-objects', '%s.git' % name)
+      else:
+        use_git_worktrees = True
+        gitdir = os.path.join(self.repodir, 'worktrees', '%s.git' % name)
+        objdir = gitdir
+    return relpath, worktree, gitdir, objdir, use_git_worktrees
 
   def GetProjectsWithName(self, name):
     return self._projects.get(name, [])
@@ -987,19 +1040,30 @@ class XmlManifest(object):
     # Assume paths might be used on case-insensitive filesystems.
     path = path.lower()
 
+    # Split up the path by its components.  We can't use os.path.sep exclusively
+    # as some platforms (like Windows) will convert / to \ and that bypasses all
+    # our constructed logic here.  Especially since manifest authors only use
+    # / in their paths.
+    resep = re.compile(r'[/%s]' % re.escape(os.path.sep))
+    parts = resep.split(path)
+
     # Some people use src="." to create stable links to projects.  Lets allow
     # that but reject all other uses of "." to keep things simple.
-    parts = path.split(os.path.sep)
     if parts != ['.']:
       for part in set(parts):
         if part in {'.', '..', '.git'} or part.startswith('.repo'):
           return 'bad component: %s' % (part,)
 
-    if not symlink and path.endswith(os.path.sep):
+    if not symlink and resep.match(path[-1]):
       return 'dirs not allowed'
 
+    # NB: The two abspath checks here are to handle platforms with multiple
+    # filesystem path styles (e.g. Windows).
     norm = os.path.normpath(path)
-    if norm == '..' or norm.startswith('../') or norm.startswith(os.path.sep):
+    if (norm == '..' or
+        (len(norm) >= 3 and norm.startswith('..') and resep.match(norm[0])) or
+        os.path.isabs(norm) or
+        norm.startswith('/')):
       return 'path cannot be outside'
 
   @classmethod

pager.py

@@ -59,7 +59,8 @@ def _PipePager(pager):
   global pager_process, old_stdout, old_stderr
   assert pager_process is None, "Only one active pager process at a time"
   # Create pager process, piping stdout/err into its stdin
-  pager_process = subprocess.Popen([pager], stdin=subprocess.PIPE, stdout=sys.stdout, stderr=sys.stderr)
+  pager_process = subprocess.Popen([pager], stdin=subprocess.PIPE, stdout=sys.stdout,
+                                   stderr=sys.stderr)
   old_stdout = sys.stdout
   old_stderr = sys.stderr
   sys.stdout = pager_process.stdin

platform_utils.py

@@ -90,6 +90,11 @@ class _FileDescriptorStreamsNonBlocking(FileDescriptorStreams):
   """ Implementation of FileDescriptorStreams for platforms that support
   non blocking I/O.
   """
+  def __init__(self):
+    super(_FileDescriptorStreamsNonBlocking, self).__init__()
+    self._poll = select.poll()
+    self._fd_to_stream = {}
+
   class Stream(object):
     """ Encapsulates a file descriptor """
 
@@ -114,11 +119,18 @@ class _FileDescriptorStreamsNonBlocking(FileDescriptorStreams):
       self.fd.close()
 
   def _create_stream(self, fd, dest, std_name):
-    return self.Stream(fd, dest, std_name)
+    stream = self.Stream(fd, dest, std_name)
+    self._fd_to_stream[stream.fileno()] = stream
+    self._poll.register(stream, select.POLLIN)
+    return stream
+
+  def remove(self, stream):
+    self._poll.unregister(stream)
+    del self._fd_to_stream[stream.fileno()]
+    super(_FileDescriptorStreamsNonBlocking, self).remove(stream)
 
   def select(self):
-    ready_streams, _, _ = select.select(self.streams, [], [])
-    return ready_streams
+    return [self._fd_to_stream[fd] for fd, _ in self._poll.poll()]
 
 
 class _FileDescriptorStreamsThreads(FileDescriptorStreams):
@@ -179,7 +191,7 @@ class _FileDescriptorStreamsThreads(FileDescriptorStreams):
       for line in iter(self.fd.readline, b''):
         self.queue.put(_FileDescriptorStreamsThreads.QueueItem(self, line))
       self.fd.close()
-      self.queue.put(_FileDescriptorStreamsThreads.QueueItem(self, None))
+      self.queue.put(_FileDescriptorStreamsThreads.QueueItem(self, b''))
 
 
 def symlink(source, link_name):

platform_utils_win32.py

@@ -152,7 +152,8 @@ def create_dirsymlink(source, link_name):
 
 
 def _create_symlink(source, link_name, dwFlags):
-  if not CreateSymbolicLinkW(link_name, source, dwFlags | SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE):
+  if not CreateSymbolicLinkW(link_name, source,
+                             dwFlags | SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE):
     # See https://github.com/golang/go/pull/24307/files#diff-b87bc12e4da2497308f9ef746086e4f0
     # "the unprivileged create flag is unsupported below Windows 10 (1703, v10.0.14972).
     # retry without it."
@@ -218,8 +219,8 @@ def _preserve_encoding(source, target):
   if is_python3():
     return target
 
-  if isinstance(source, unicode):
-    return unicode(target)
+  if isinstance(source, unicode):  # noqa: F821
+    return unicode(target)  # noqa: F821
   return str(target)
 
 

project.py

@@ -42,7 +42,7 @@ import platform_utils
 import progress
 from repo_trace import IsTrace, Trace
 
-from git_refs import GitRefs, HEAD, R_HEADS, R_TAGS, R_PUB, R_M
+from git_refs import GitRefs, HEAD, R_HEADS, R_TAGS, R_PUB, R_M, R_WORKTREE_M
 
 from pyversion import is_python3
 if is_python3():
@@ -52,7 +52,7 @@ else:
   import urlparse
   urllib = imp.new_module('urllib')
   urllib.parse = urlparse
-  input = raw_input
+  input = raw_input  # noqa: F821
 
 
 def _lwrite(path, content):
@@ -198,18 +198,22 @@ class ReviewableBranch(object):
     return self._base_exists
 
   def UploadForReview(self, people,
+                      dryrun=False,
                       auto_topic=False,
-                      draft=False,
+                      hashtags=(),
+                      labels=(),
                       private=False,
                       notify=None,
                       wip=False,
                       dest_branch=None,
                       validate_certs=True,
                       push_options=None):
-    self.project.UploadForReview(self.name,
-                                 people,
+    self.project.UploadForReview(branch=self.name,
+                                 people=people,
+                                 dryrun=dryrun,
                                  auto_topic=auto_topic,
-                                 draft=draft,
+                                 hashtags=hashtags,
+                                 labels=labels,
                                  private=private,
                                  notify=notify,
                                  wip=wip,
@@ -271,7 +275,12 @@ def _SafeExpandPath(base, subpath, skipfinal=False):
   NB: We rely on a number of paths already being filtered out while parsing the
   manifest.  See the validation logic in manifest_xml.py for more details.
   """
-  components = subpath.split(os.path.sep)
+  # Split up the path by its components.  We can't use os.path.sep exclusively
+  # as some platforms (like Windows) will convert / to \ and that bypasses all
+  # our constructed logic here.  Especially since manifest authors only use
+  # / in their paths.
+  resep = re.compile(r'[/%s]' % re.escape(os.path.sep))
+  components = resep.split(subpath)
   if skipfinal:
     # Whether the caller handles the final component itself.
     finalpart = components.pop()
@@ -862,6 +871,7 @@ class Project(object):
                clone_depth=None,
                upstream=None,
                parent=None,
+               use_git_worktrees=False,
                is_derived=False,
                dest_branch=None,
                optimized_fetch=False,
@@ -885,6 +895,7 @@ class Project(object):
       sync_tags: The `sync-tags` attribute of manifest.xml's project element.
       upstream: The `upstream` attribute of manifest.xml's project element.
       parent: The parent Project object.
+      use_git_worktrees: Whether to use `git worktree` for this project.
       is_derived: False if the project was explicitly defined in the manifest;
                   True if the project is a discovered submodule.
       dest_branch: The branch to which to push changes for review by default.
@@ -919,6 +930,10 @@ class Project(object):
     self.clone_depth = clone_depth
     self.upstream = upstream
     self.parent = parent
+    # NB: Do not use this setting in __init__ to change behavior so that the
+    # manifest.git checkout can inspect & change it after instantiating.  See
+    # the XmlManifest init code for more info.
+    self.use_git_worktrees = use_git_worktrees
     self.is_derived = is_derived
     self.optimized_fetch = optimized_fetch
     self.subprojects = []
@@ -971,11 +986,9 @@ class Project(object):
     return None
 
   def IsRebaseInProgress(self):
-    w = self.worktree
-    g = os.path.join(w, '.git')
-    return os.path.exists(os.path.join(g, 'rebase-apply')) \
-        or os.path.exists(os.path.join(g, 'rebase-merge')) \
-        or os.path.exists(os.path.join(w, '.dotest'))
+    return (os.path.exists(self.work_git.GetDotgitPath('rebase-apply')) or
+            os.path.exists(self.work_git.GetDotgitPath('rebase-merge')) or
+            os.path.exists(os.path.join(self.worktree, '.dotest')))
 
   def IsDirty(self, consider_untracked=True):
     """Is the working directory modified in some way?
@@ -1330,8 +1343,10 @@ class Project(object):
 
   def UploadForReview(self, branch=None,
                       people=([], []),
+                      dryrun=False,
                       auto_topic=False,
-                      draft=False,
+                      hashtags=(),
+                      labels=(),
                       private=False,
                       notify=None,
                       wip=False,
@@ -1366,6 +1381,8 @@ class Project(object):
     if url is None:
       raise UploadError('review not configured')
     cmd = ['push']
+    if dryrun:
+      cmd.append('-n')
 
     if url.startswith('ssh://'):
       cmd.append('--receive-pack=gerrit receive-pack')
@@ -1379,15 +1396,12 @@ class Project(object):
     if dest_branch.startswith(R_HEADS):
       dest_branch = dest_branch[len(R_HEADS):]
 
-    upload_type = 'for'
-    if draft:
-      upload_type = 'drafts'
-
-    ref_spec = '%s:refs/%s/%s' % (R_HEADS + branch.name, upload_type,
-                                  dest_branch)
+    ref_spec = '%s:refs/for/%s' % (R_HEADS + branch.name, dest_branch)
     opts = []
     if auto_topic:
       opts += ['topic=' + branch.name]
+    opts += ['t=%s' % p for p in hashtags]
+    opts += ['l=%s' % p for p in labels]
 
     opts += ['r=%s' % p for p in people[0]]
     opts += ['cc=%s' % p for p in people[1]]
@@ -1427,11 +1441,12 @@ class Project(object):
 
   def Sync_NetworkHalf(self,
                        quiet=False,
+                       verbose=False,
                        is_new=None,
                        current_branch_only=False,
                        force_sync=False,
                        clone_bundle=True,
-                       no_tags=False,
+                       tags=True,
                        archive=False,
                        optimized_fetch=False,
                        prune=False,
@@ -1470,9 +1485,9 @@ class Project(object):
     if is_new is None:
       is_new = not self.Exists
     if is_new:
-      self._InitGitDir(force_sync=force_sync)
+      self._InitGitDir(force_sync=force_sync, quiet=quiet)
     else:
-      self._UpdateHooks()
+      self._UpdateHooks(quiet=quiet)
     self._InitRemote()
 
     if is_new:
@@ -1486,9 +1501,9 @@ class Project(object):
     else:
       alt_dir = None
 
-    if clone_bundle \
-            and alt_dir is None \
-            and self._ApplyCloneBundle(initial=is_new, quiet=quiet):
+    if (clone_bundle
+            and alt_dir is None
+            and self._ApplyCloneBundle(initial=is_new, quiet=quiet, verbose=verbose)):
       is_new = False
 
     if not current_branch_only:
@@ -1500,25 +1515,25 @@ class Project(object):
       elif self.manifest.default.sync_c:
         current_branch_only = True
 
-    if not no_tags:
-      if not self.sync_tags:
-        no_tags = True
+    if not self.sync_tags:
+      tags = False
 
     if self.clone_depth:
       depth = self.clone_depth
     else:
       depth = self.manifest.manifestProject.config.GetString('repo.depth')
 
-    need_to_fetch = not (optimized_fetch and
-                         (ID_RE.match(self.revisionExpr) and
-                          self._CheckForImmutableRevision()))
-    if (need_to_fetch and
-        not self._RemoteFetch(initial=is_new, quiet=quiet, alt_dir=alt_dir,
-                              current_branch_only=current_branch_only,
-                              no_tags=no_tags, prune=prune, depth=depth,
-                              submodules=submodules, force_sync=force_sync,
-                              clone_filter=clone_filter)):
-      return False
+    # See if we can skip the network fetch entirely.
+    if not (optimized_fetch and
+            (ID_RE.match(self.revisionExpr) and
+             self._CheckForImmutableRevision())):
+      if not self._RemoteFetch(
+              initial=is_new, quiet=quiet, verbose=verbose, alt_dir=alt_dir,
+              current_branch_only=current_branch_only,
+              tags=tags, prune=prune, depth=depth,
+              submodules=submodules, force_sync=force_sync,
+              clone_filter=clone_filter):
+        return False
 
     mp = self.manifest.manifestProject
     dissociate = mp.config.GetBoolean('repo.dissociate')
@@ -1816,19 +1831,123 @@ class Project(object):
                             patch_id,
                             self.bare_git.rev_parse('FETCH_HEAD'))
 
-# Branch Management ##
-  def GetHeadPath(self):
-    """Return the full path to the HEAD ref."""
-    dotgit = os.path.join(self.worktree, '.git')
-    if os.path.isfile(dotgit):
-      # Git worktrees use a "gitdir:" syntax to point to the scratch space.
-      with open(dotgit) as fp:
-        setting = fp.read()
-        assert setting.startswith('gitdir:')
-        gitdir = setting.split(':', 1)[1].strip()
-      dotgit = os.path.join(self.worktree, gitdir)
-    return os.path.join(dotgit, HEAD)
+  def DeleteWorktree(self, quiet=False, force=False):
+    """Delete the source checkout and any other housekeeping tasks.
 
+    This currently leaves behind the internal .repo/ cache state.  This helps
+    when switching branches or manifest changes get reverted as we don't have
+    to redownload all the git objects.  But we should do some GC at some point.
+
+    Args:
+      quiet: Whether to hide normal messages.
+      force: Always delete tree even if dirty.
+
+    Returns:
+      True if the worktree was completely cleaned out.
+    """
+    if self.IsDirty():
+      if force:
+        print('warning: %s: Removing dirty project: uncommitted changes lost.' %
+              (self.relpath,), file=sys.stderr)
+      else:
+        print('error: %s: Cannot remove project: uncommitted changes are '
+              'present.\n' % (self.relpath,), file=sys.stderr)
+        return False
+
+    if not quiet:
+      print('%s: Deleting obsolete checkout.' % (self.relpath,))
+
+    # Unlock and delink from the main worktree.  We don't use git's worktree
+    # remove because it will recursively delete projects -- we handle that
+    # ourselves below.  https://crbug.com/git/48
+    if self.use_git_worktrees:
+      needle = platform_utils.realpath(self.gitdir)
+      # Find the git worktree commondir under .repo/worktrees/.
+      output = self.bare_git.worktree('list', '--porcelain').splitlines()[0]
+      assert output.startswith('worktree '), output
+      commondir = output[9:]
+      # Walk each of the git worktrees to see where they point.
+      configs = os.path.join(commondir, 'worktrees')
+      for name in os.listdir(configs):
+        gitdir = os.path.join(configs, name, 'gitdir')
+        with open(gitdir) as fp:
+          relpath = fp.read().strip()
+        # Resolve the checkout path and see if it matches this project.
+        fullpath = platform_utils.realpath(os.path.join(configs, name, relpath))
+        if fullpath == needle:
+          platform_utils.rmtree(os.path.join(configs, name))
+
+    # Delete the .git directory first, so we're less likely to have a partially
+    # working git repository around. There shouldn't be any git projects here,
+    # so rmtree works.
+
+    # Try to remove plain files first in case of git worktrees.  If this fails
+    # for any reason, we'll fall back to rmtree, and that'll display errors if
+    # it can't remove things either.
+    try:
+      platform_utils.remove(self.gitdir)
+    except OSError:
+      pass
+    try:
+      platform_utils.rmtree(self.gitdir)
+    except OSError as e:
+      if e.errno != errno.ENOENT:
+        print('error: %s: %s' % (self.gitdir, e), file=sys.stderr)
+        print('error: %s: Failed to delete obsolete checkout; remove manually, '
+              'then run `repo sync -l`.' % (self.relpath,), file=sys.stderr)
+        return False
+
+    # Delete everything under the worktree, except for directories that contain
+    # another git project.
+    dirs_to_remove = []
+    failed = False
+    for root, dirs, files in platform_utils.walk(self.worktree):
+      for f in files:
+        path = os.path.join(root, f)
+        try:
+          platform_utils.remove(path)
+        except OSError as e:
+          if e.errno != errno.ENOENT:
+            print('error: %s: Failed to remove: %s' % (path, e), file=sys.stderr)
+            failed = True
+      dirs[:] = [d for d in dirs
+                 if not os.path.lexists(os.path.join(root, d, '.git'))]
+      dirs_to_remove += [os.path.join(root, d) for d in dirs
+                         if os.path.join(root, d) not in dirs_to_remove]
+    for d in reversed(dirs_to_remove):
+      if platform_utils.islink(d):
+        try:
+          platform_utils.remove(d)
+        except OSError as e:
+          if e.errno != errno.ENOENT:
+            print('error: %s: Failed to remove: %s' % (d, e), file=sys.stderr)
+            failed = True
+      elif not platform_utils.listdir(d):
+        try:
+          platform_utils.rmdir(d)
+        except OSError as e:
+          if e.errno != errno.ENOENT:
+            print('error: %s: Failed to remove: %s' % (d, e), file=sys.stderr)
+            failed = True
+    if failed:
+      print('error: %s: Failed to delete obsolete checkout.' % (self.relpath,),
+            file=sys.stderr)
+      print('       Remove manually, then run `repo sync -l`.', file=sys.stderr)
+      return False
+
+    # Try deleting parent dirs if they are empty.
+    path = self.worktree
+    while path != self.manifest.topdir:
+      try:
+        platform_utils.rmdir(path)
+      except OSError as e:
+        if e.errno != errno.ENOENT:
+          break
+      path = os.path.dirname(path)
+
+    return True
+
+# Branch Management ##
   def StartBranch(self, name, branch_merge='', revision=None):
     """Create a new branch off the manifest's revision.
     """
@@ -1862,13 +1981,9 @@ class Project(object):
       except KeyError:
         head = None
     if revid and head and revid == head:
-      ref = os.path.join(self.gitdir, R_HEADS + name)
-      try:
-        os.makedirs(os.path.dirname(ref))
-      except OSError:
-        pass
-      _lwrite(ref, '%s\n' % revid)
-      _lwrite(self.GetHeadPath(), 'ref: %s%s\n' % (R_HEADS, name))
+      ref = R_HEADS + name
+      self.work_git.update_ref(ref, revid)
+      self.work_git.symbolic_ref(HEAD, ref)
       branch.Save()
       return True
 
@@ -1915,7 +2030,8 @@ class Project(object):
       # Same revision; just update HEAD to point to the new
       # target branch, but otherwise take no other action.
       #
-      _lwrite(self.GetHeadPath(), 'ref: %s%s\n' % (R_HEADS, name))
+      _lwrite(self.work_git.GetDotgitPath(subpath=HEAD),
+              'ref: %s%s\n' % (R_HEADS, name))
       return True
 
     return GitCommand(self,
@@ -1948,7 +2064,7 @@ class Project(object):
 
       revid = self.GetRevisionId(all_refs)
       if head == revid:
-        _lwrite(self.GetHeadPath(), '%s\n' % revid)
+        _lwrite(self.work_git.GetDotgitPath(subpath=HEAD), '%s\n' % revid)
       else:
         self._Checkout(revid, quiet=True)
 
@@ -2166,6 +2282,24 @@ class Project(object):
     return result
 
 # Direct Git Commands ##
+  def EnableRepositoryExtension(self, key, value='true', version=1):
+    """Enable git repository extension |key| with |value|.
+
+    Args:
+      key: The extension to enabled.  Omit the "extensions." prefix.
+      value: The value to use for the extension.
+      version: The minimum git repository version needed.
+    """
+    # Make sure the git repo version is new enough already.
+    found_version = self.config.GetInt('core.repositoryFormatVersion')
+    if found_version is None:
+      found_version = 0
+    if found_version < version:
+      self.config.SetString('core.repositoryFormatVersion', str(version))
+
+    # Enable the extension!
+    self.config.SetString('extensions.%s' % (key,), value)
+
   def _CheckForImmutableRevision(self):
     try:
       # if revision (sha or tag) is not present then following function
@@ -2193,8 +2327,9 @@ class Project(object):
                    current_branch_only=False,
                    initial=False,
                    quiet=False,
+                   verbose=False,
                    alt_dir=None,
-                   no_tags=False,
+                   tags=True,
                    prune=False,
                    depth=None,
                    submodules=False,
@@ -2223,7 +2358,7 @@ class Project(object):
 
       if is_sha1 or tag_name is not None:
         if self._CheckForImmutableRevision():
-          if not quiet:
+          if verbose:
             print('Skipped fetching project %s (already have persistent ref)'
                   % self.name)
           return True
@@ -2293,7 +2428,7 @@ class Project(object):
     if clone_filter:
       git_require((2, 19, 0), fail=True, msg='partial clones')
       cmd.append('--filter=%s' % clone_filter)
-      self.config.SetString('extensions.partialclone', self.remote.name)
+      self.EnableRepositoryExtension('partialclone', self.remote.name)
 
     if depth:
       cmd.append('--depth=%s' % depth)
@@ -2305,8 +2440,10 @@ class Project(object):
       if os.path.exists(os.path.join(self.gitdir, 'shallow')):
         cmd.append('--depth=2147483647')
 
-    if quiet:
+    if not verbose:
       cmd.append('--quiet')
+    if not quiet and sys.stdout.isatty():
+      cmd.append('--progress')
     if not self.worktree:
       cmd.append('--update-head-ok')
     cmd.append(name)
@@ -2352,7 +2489,7 @@ class Project(object):
 
     # If using depth then we should not get all the tags since they may
     # be outside of the depth.
-    if no_tags or depth:
+    if not tags or depth:
       cmd.append('--no-tags')
     else:
       cmd.append('--tags')
@@ -2362,7 +2499,8 @@ class Project(object):
 
     ok = False
     for _i in range(2):
-      gitcmd = GitCommand(self, cmd, bare=True, ssh_proxy=ssh_proxy)
+      gitcmd = GitCommand(self, cmd, bare=True, ssh_proxy=ssh_proxy,
+                          merge_output=True, capture_stdout=quiet)
       ret = gitcmd.Wait()
       if ret == 0:
         ok = True
@@ -2385,6 +2523,8 @@ class Project(object):
       elif ret < 0:
         # Git died with a signal, exit immediately
         break
+      if not verbose:
+        print('%s:\n%s' % (self.name, gitcmd.stdout), file=sys.stderr)
       time.sleep(random.randint(30, 45))
 
     if initial:
@@ -2400,21 +2540,17 @@ class Project(object):
       # got what we wanted, else trigger a second run of all
       # refs.
       if not self._CheckForImmutableRevision():
-        if current_branch_only and depth:
-          # Sync the current branch only with depth set to None
-          return self._RemoteFetch(name=name,
-                                   current_branch_only=current_branch_only,
-                                   initial=False, quiet=quiet, alt_dir=alt_dir,
-                                   depth=None, clone_filter=clone_filter)
-        else:
-          # Avoid infinite recursion: sync all branches with depth set to None
-          return self._RemoteFetch(name=name, current_branch_only=False,
-                                   initial=False, quiet=quiet, alt_dir=alt_dir,
-                                   depth=None, clone_filter=clone_filter)
+        # Sync the current branch only with depth set to None.
+        # We always pass depth=None down to avoid infinite recursion.
+        return self._RemoteFetch(
+            name=name, quiet=quiet, verbose=verbose,
+            current_branch_only=current_branch_only and depth,
+            initial=False, alt_dir=alt_dir,
+            depth=None, clone_filter=clone_filter)
 
     return ok
 
-  def _ApplyCloneBundle(self, initial=False, quiet=False):
+  def _ApplyCloneBundle(self, initial=False, quiet=False, verbose=False):
     if initial and \
         (self.manifest.manifestProject.config.GetString('repo.depth') or
          self.clone_depth):
@@ -2438,13 +2574,16 @@ class Project(object):
       return False
 
     if not exist_dst:
-      exist_dst = self._FetchBundle(bundle_url, bundle_tmp, bundle_dst, quiet)
+      exist_dst = self._FetchBundle(bundle_url, bundle_tmp, bundle_dst, quiet,
+                                    verbose)
     if not exist_dst:
       return False
 
     cmd = ['fetch']
-    if quiet:
+    if not verbose:
       cmd.append('--quiet')
+    if not quiet and sys.stdout.isatty():
+      cmd.append('--progress')
     if not self.worktree:
       cmd.append('--update-head-ok')
     cmd.append(bundle_dst)
@@ -2459,13 +2598,13 @@ class Project(object):
       platform_utils.remove(bundle_tmp)
     return ok
 
-  def _FetchBundle(self, srcUrl, tmpPath, dstPath, quiet):
+  def _FetchBundle(self, srcUrl, tmpPath, dstPath, quiet, verbose):
     if os.path.exists(dstPath):
       platform_utils.remove(dstPath)
 
     cmd = ['curl', '--fail', '--output', tmpPath, '--netrc', '--location']
     if quiet:
-      cmd += ['--silent']
+      cmd += ['--silent', '--show-error']
     if os.path.exists(tmpPath):
       size = os.stat(tmpPath).st_size
       if size >= 1024:
@@ -2487,22 +2626,28 @@ class Project(object):
 
       if IsTrace():
         Trace('%s', ' '.join(cmd))
+      if verbose:
+        print('%s: Downloading bundle: %s' % (self.name, srcUrl))
+      stdout = None if verbose else subprocess.PIPE
+      stderr = None if verbose else subprocess.STDOUT
       try:
-        proc = subprocess.Popen(cmd)
+        proc = subprocess.Popen(cmd, stdout=stdout, stderr=stderr)
       except OSError:
         return False
 
-      curlret = proc.wait()
+      (output, _) = proc.communicate()
+      curlret = proc.returncode
 
       if curlret == 22:
         # From curl man page:
         # 22: HTTP page not retrieved. The requested url was not found or
         # returned another error with the HTTP error code being 400 or above.
         # This return code only appears if -f, --fail is used.
-        if not quiet:
-          print("Server does not provide clone.bundle; ignoring.",
-                file=sys.stderr)
+        if verbose:
+          print('Server does not provide clone.bundle; ignoring.')
         return False
+      elif curlret and not verbose and output:
+        print('%s' % output, file=sys.stderr)
 
     if os.path.exists(tmpPath):
       if curlret == 0 and self._IsValidBundle(tmpPath, quiet):
@@ -2536,8 +2681,12 @@ class Project(object):
       if self._allrefs:
         raise GitError('%s checkout %s ' % (self.name, rev))
 
-  def _CherryPick(self, rev):
+  def _CherryPick(self, rev, ffonly=False, record_origin=False):
     cmd = ['cherry-pick']
+    if ffonly:
+      cmd.append('--ff')
+    if record_origin:
+      cmd.append('-x')
     cmd.append(rev)
     cmd.append('--')
     if GitCommand(self, cmd).Wait() != 0:
@@ -2584,13 +2733,13 @@ class Project(object):
       raise GitError('%s rebase %s ' % (self.name, upstream))
 
   def _FastForward(self, head, ffonly=False):
-    cmd = ['merge', head]
+    cmd = ['merge', '--no-stat', head]
     if ffonly:
       cmd.append("--ff-only")
     if GitCommand(self, cmd).Wait() != 0:
       raise GitError('%s merge %s ' % (self.name, head))
 
-  def _InitGitDir(self, mirror_git=None, force_sync=False):
+  def _InitGitDir(self, mirror_git=None, force_sync=False, quiet=False):
     init_git_dir = not os.path.exists(self.gitdir)
     init_obj_dir = not os.path.exists(self.objdir)
     try:
@@ -2599,6 +2748,20 @@ class Project(object):
         os.makedirs(self.objdir)
         self.bare_objdir.init()
 
+        if self.use_git_worktrees:
+          # Set up the m/ space to point to the worktree-specific ref space.
+          # We'll update the worktree-specific ref space on each checkout.
+          if self.manifest.branch:
+            self.bare_git.symbolic_ref(
+                '-m', 'redirecting to worktree scope',
+                R_M + self.manifest.branch,
+                R_WORKTREE_M + self.manifest.branch)
+
+          # Enable per-worktree config file support if possible.  This is more a
+          # nice-to-have feature for users rather than a hard requirement.
+          if git_require((2, 19, 0)):
+            self.EnableRepositoryExtension('worktreeConfig')
+
       # If we have a separate directory to hold refs, initialize it as well.
       if self.objdir != self.gitdir:
         if init_git_dir:
@@ -2618,7 +2781,8 @@ class Project(object):
               if self.worktree and os.path.exists(platform_utils.realpath
                                                   (self.worktree)):
                 platform_utils.rmtree(platform_utils.realpath(self.worktree))
-              return self._InitGitDir(mirror_git=mirror_git, force_sync=False)
+              return self._InitGitDir(mirror_git=mirror_git, force_sync=False,
+                                      quiet=quiet)
             except Exception:
               raise e
           raise e
@@ -2632,13 +2796,15 @@ class Project(object):
             mirror_git = os.path.join(ref_dir, self.name + '.git')
           repo_git = os.path.join(ref_dir, '.repo', 'projects',
                                   self.relpath + '.git')
+          worktrees_git = os.path.join(ref_dir, '.repo', 'worktrees',
+                                       self.name + '.git')
 
           if os.path.exists(mirror_git):
             ref_dir = mirror_git
-
           elif os.path.exists(repo_git):
             ref_dir = repo_git
-
+          elif os.path.exists(worktrees_git):
+            ref_dir = worktrees_git
           else:
             ref_dir = None
 
@@ -2650,7 +2816,7 @@ class Project(object):
             _lwrite(os.path.join(self.gitdir, 'objects/info/alternates'),
                     os.path.join(ref_dir, 'objects') + '\n')
 
-        self._UpdateHooks()
+        self._UpdateHooks(quiet=quiet)
 
         m = self.manifest.manifestProject.config
         for key in ['user.name', 'user.email']:
@@ -2669,11 +2835,11 @@ class Project(object):
         platform_utils.rmtree(self.gitdir)
       raise
 
-  def _UpdateHooks(self):
+  def _UpdateHooks(self, quiet=False):
     if os.path.exists(self.gitdir):
-      self._InitHooks()
+      self._InitHooks(quiet=quiet)
 
-  def _InitHooks(self):
+  def _InitHooks(self, quiet=False):
     hooks = platform_utils.realpath(self._gitdir_path('hooks'))
     if not os.path.exists(hooks):
       os.makedirs(hooks)
@@ -2693,18 +2859,23 @@ class Project(object):
       if platform_utils.islink(dst):
         continue
       if os.path.exists(dst):
-        if filecmp.cmp(stock_hook, dst, shallow=False):
-          platform_utils.remove(dst)
-        else:
-          _warn("%s: Not replacing locally modified %s hook",
-                self.relpath, name)
-          continue
+        # If the files are the same, we'll leave it alone.  We create symlinks
+        # below by default but fallback to hardlinks if the OS blocks them.
+        # So if we're here, it's probably because we made a hardlink below.
+        if not filecmp.cmp(stock_hook, dst, shallow=False):
+          if not quiet:
+            _warn("%s: Not replacing locally modified %s hook",
+                  self.relpath, name)
+        continue
       try:
         platform_utils.symlink(
             os.path.relpath(stock_hook, os.path.dirname(dst)), dst)
       except OSError as e:
         if e.errno == errno.EPERM:
-          raise GitError(self._get_symlink_error_message())
+          try:
+            os.link(stock_hook, dst)
+          except OSError:
+            raise GitError(self._get_symlink_error_message())
         else:
           raise
 
@@ -2724,27 +2895,43 @@ class Project(object):
 
   def _InitMRef(self):
     if self.manifest.branch:
-      self._InitAnyMRef(R_M + self.manifest.branch)
+      if self.use_git_worktrees:
+        # We can't update this ref with git worktrees until it exists.
+        # We'll wait until the initial checkout to set it.
+        if not os.path.exists(self.worktree):
+          return
+
+        base = R_WORKTREE_M
+        active_git = self.work_git
+      else:
+        base = R_M
+        active_git = self.bare_git
+
+      self._InitAnyMRef(base + self.manifest.branch, active_git)
 
   def _InitMirrorHead(self):
-    self._InitAnyMRef(HEAD)
+    self._InitAnyMRef(HEAD, self.bare_git)
 
-  def _InitAnyMRef(self, ref):
+  def _InitAnyMRef(self, ref, active_git):
     cur = self.bare_ref.symref(ref)
 
     if self.revisionId:
       if cur != '' or self.bare_ref.get(ref) != self.revisionId:
         msg = 'manifest set to %s' % self.revisionId
         dst = self.revisionId + '^0'
-        self.bare_git.UpdateRef(ref, dst, message=msg, detach=True)
+        active_git.UpdateRef(ref, dst, message=msg, detach=True)
     else:
       remote = self.GetRemote(self.remote.name)
       dst = remote.ToLocal(self.revisionExpr)
       if cur != dst:
         msg = 'manifest set to %s' % self.revisionExpr
-        self.bare_git.symbolic_ref('-m', msg, ref, dst)
+        active_git.symbolic_ref('-m', msg, ref, dst)
 
   def _CheckDirReference(self, srcdir, destdir, share_refs):
+    # Git worktrees don't use symlinks to share at all.
+    if self.use_git_worktrees:
+      return
+
     symlink_files = self.shareable_files[:]
     symlink_dirs = self.shareable_dirs[:]
     if share_refs:
@@ -2844,11 +3031,43 @@ class Project(object):
         else:
           raise
 
+  def _InitGitWorktree(self):
+    """Init the project using git worktrees."""
+    self.bare_git.worktree('prune')
+    self.bare_git.worktree('add', '-ff', '--checkout', '--detach', '--lock',
+                           self.worktree, self.GetRevisionId())
+
+    # Rewrite the internal state files to use relative paths between the
+    # checkouts & worktrees.
+    dotgit = os.path.join(self.worktree, '.git')
+    with open(dotgit, 'r') as fp:
+      # Figure out the checkout->worktree path.
+      setting = fp.read()
+      assert setting.startswith('gitdir:')
+      git_worktree_path = setting.split(':', 1)[1].strip()
+    # Some platforms (e.g. Windows) won't let us update dotgit in situ because
+    # of file permissions.  Delete it and recreate it from scratch to avoid.
+    platform_utils.remove(dotgit)
+    # Use relative path from checkout->worktree.
+    with open(dotgit, 'w') as fp:
+      print('gitdir:', os.path.relpath(git_worktree_path, self.worktree),
+            file=fp)
+    # Use relative path from worktree->checkout.
+    with open(os.path.join(git_worktree_path, 'gitdir'), 'w') as fp:
+      print(os.path.relpath(dotgit, git_worktree_path), file=fp)
+
+    self._InitMRef()
+
   def _InitWorkTree(self, force_sync=False, submodules=False):
     realdotgit = os.path.join(self.worktree, '.git')
     tmpdotgit = realdotgit + '.tmp'
     init_dotgit = not os.path.exists(realdotgit)
     if init_dotgit:
+      if self.use_git_worktrees:
+        self._InitGitWorktree()
+        self._CopyAndLinkFiles()
+        return
+
       dotgit = tmpdotgit
       platform_utils.rmtree(tmpdotgit, ignore_errors=True)
       os.makedirs(tmpdotgit)
@@ -3026,11 +3245,28 @@ class Project(object):
       finally:
         p.Wait()
 
-    def GetHead(self):
+    def GetDotgitPath(self, subpath=None):
+      """Return the full path to the .git dir.
+
+      As a convenience, append |subpath| if provided.
+      """
       if self._bare:
-        path = os.path.join(self._project.gitdir, HEAD)
+        dotgit = self._gitdir
       else:
-        path = self._project.GetHeadPath()
+        dotgit = os.path.join(self._project.worktree, '.git')
+        if os.path.isfile(dotgit):
+          # Git worktrees use a "gitdir:" syntax to point to the scratch space.
+          with open(dotgit) as fp:
+            setting = fp.read()
+          assert setting.startswith('gitdir:')
+          gitdir = setting.split(':', 1)[1].strip()
+          dotgit = os.path.normpath(os.path.join(self._project.worktree, gitdir))
+
+      return dotgit if subpath is None else os.path.join(dotgit, subpath)
+
+    def GetHead(self):
+      """Return the ref that HEAD points to."""
+      path = self.GetDotgitPath(subpath=HEAD)
       try:
         with open(path) as fd:
           line = fd.readline()

release/README.md

@@ -0,0 +1,2 @@
+These are helper tools for managing official releases.
+See the [release process](../docs/release-process.md) document for more details.

release/sign-launcher.py

@@ -0,0 +1,114 @@
+#!/usr/bin/env python3
+# Copyright (C) 2020 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helper tool for signing repo launcher scripts correctly.
+
+This is intended to be run only by the official Repo release managers.
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+
+import util
+
+
+def sign(opts):
+  """Sign the launcher!"""
+  output = ''
+  for key in opts.keys:
+    # We use ! at the end of the key so that gpg uses this specific key.
+    # Otherwise it uses the key as a lookup into the overall key and uses the
+    # default signing key.  i.e. It will see that KEYID_RSA is a subkey of
+    # another key, and use the primary key to sign instead of the subkey.
+    cmd = ['gpg', '--homedir', opts.gpgdir, '-u', f'{key}!', '--batch', '--yes',
+           '--armor', '--detach-sign', '--output', '-', opts.launcher]
+    ret = util.run(opts, cmd, encoding='utf-8', stdout=subprocess.PIPE)
+    output += ret.stdout
+
+  # Save the combined signatures into one file.
+  with open(f'{opts.launcher}.asc', 'w', encoding='utf-8') as fp:
+    fp.write(output)
+
+
+def check(opts):
+  """Check the signature."""
+  util.run(opts, ['gpg', '--verify', f'{opts.launcher}.asc'])
+
+
+def postmsg(opts):
+  """Helpful info to show at the end for release manager."""
+  print(f"""
+Repo launcher bucket:
+  gs://git-repo-downloads/
+
+To upload this launcher directly:
+  gsutil cp -a public-read {opts.launcher} {opts.launcher}.asc gs://git-repo-downloads/
+
+NB: You probably want to upload it with a specific version first, e.g.:
+  gsutil cp -a public-read {opts.launcher} gs://git-repo-downloads/repo-3.0
+  gsutil cp -a public-read {opts.launcher}.asc gs://git-repo-downloads/repo-3.0.asc
+""")
+
+
+def get_parser():
+  """Get a CLI parser."""
+  parser = argparse.ArgumentParser(description=__doc__)
+  parser.add_argument('-n', '--dry-run',
+                      dest='dryrun', action='store_true',
+                      help='show everything that would be done')
+  parser.add_argument('--gpgdir',
+                      default=os.path.join(util.HOMEDIR, '.gnupg', 'repo'),
+                      help='path to dedicated gpg dir with release keys '
+                           '(default: ~/.gnupg/repo/)')
+  parser.add_argument('--keyid', dest='keys', default=[], action='append',
+                      help='alternative signing keys to use')
+  parser.add_argument('launcher',
+                      default=os.path.join(util.TOPDIR, 'repo'), nargs='?',
+                      help='the launcher script to sign')
+  return parser
+
+
+def main(argv):
+  """The main func!"""
+  parser = get_parser()
+  opts = parser.parse_args(argv)
+
+  if not os.path.exists(opts.gpgdir):
+    parser.error(f'--gpgdir does not exist: {opts.gpgdir}')
+  if not os.path.exists(opts.launcher):
+    parser.error(f'launcher does not exist: {opts.launcher}')
+
+  opts.launcher = os.path.relpath(opts.launcher)
+  print(f'Signing "{opts.launcher}" launcher script and saving to '
+        f'"{opts.launcher}.asc"')
+
+  if opts.keys:
+    print(f'Using custom keys to sign: {" ".join(opts.keys)}')
+  else:
+    print('Using official Repo release keys to sign')
+    opts.keys = [util.KEYID_DSA, util.KEYID_RSA, util.KEYID_ECC]
+    util.import_release_key(opts)
+
+  sign(opts)
+  check(opts)
+  postmsg(opts)
+
+  return 0
+
+
+if __name__ == '__main__':
+  sys.exit(main(sys.argv[1:]))

release/sign-tag.py

@@ -0,0 +1,144 @@
+#!/usr/bin/env python3
+# Copyright (C) 2020 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helper tool for signing repo release tags correctly.
+
+This is intended to be run only by the official Repo release managers, but it
+could be run by people maintaining their own fork of the project.
+
+NB: Avoid new releases on off-hours.  If something goes wrong, staff/oncall need
+to be active in order to respond quickly & effectively.  Recommend sticking to:
+* Mon - Thu, 9:00 - 14:00 PT (i.e. MTV time)
+* Avoid US holidays (and large international ones if possible)
+* Follow the normal Google production freeze schedule
+"""
+
+import argparse
+import os
+import re
+import subprocess
+import sys
+
+import util
+
+
+# We currently sign with the old DSA key as it's been around the longest.
+# We should transition to RSA by Jun 2020, and ECC by Jun 2021.
+KEYID = util.KEYID_DSA
+
+# Regular expression to validate tag names.
+RE_VALID_TAG = r'^v([0-9]+[.])+[0-9]+$'
+
+
+def sign(opts):
+  """Tag the commit & sign it!"""
+  # We use ! at the end of the key so that gpg uses this specific key.
+  # Otherwise it uses the key as a lookup into the overall key and uses the
+  # default signing key.  i.e. It will see that KEYID_RSA is a subkey of
+  # another key, and use the primary key to sign instead of the subkey.
+  cmd = ['git', 'tag', '-s', opts.tag, '-u', f'{opts.key}!',
+         '-m', f'repo {opts.tag}', opts.commit]
+
+  key = 'GNUPGHOME'
+  print('+', f'export {key}="{opts.gpgdir}"')
+  oldvalue = os.getenv(key)
+  os.putenv(key, opts.gpgdir)
+  util.run(opts, cmd)
+  if oldvalue is None:
+    os.unsetenv(key)
+  else:
+    os.putenv(key, oldvalue)
+
+
+def check(opts):
+  """Check the signature."""
+  util.run(opts, ['git', 'tag', '--verify', opts.tag])
+
+
+def postmsg(opts):
+  """Helpful info to show at the end for release manager."""
+  cmd = ['git', 'rev-parse', 'remotes/origin/stable']
+  ret = util.run(opts, cmd, encoding='utf-8', stdout=subprocess.PIPE)
+  current_release = ret.stdout.strip()
+
+  cmd = ['git', 'log', '--format=%h (%aN) %s', '--no-merges',
+         f'remotes/origin/stable..{opts.tag}']
+  ret = util.run(opts, cmd, encoding='utf-8', stdout=subprocess.PIPE)
+  shortlog = ret.stdout.strip()
+
+  print(f"""
+Here's the short log since the last release.
+{shortlog}
+
+To push release to the public:
+  git push origin {opts.commit}:stable {opts.tag} -n
+NB: People will start upgrading to this version immediately.
+
+To roll back a release:
+  git push origin --force {current_release}:stable -n
+""")
+
+
+def get_parser():
+  """Get a CLI parser."""
+  parser = argparse.ArgumentParser(
+      description=__doc__,
+      formatter_class=argparse.RawDescriptionHelpFormatter)
+  parser.add_argument('-n', '--dry-run',
+                      dest='dryrun', action='store_true',
+                      help='show everything that would be done')
+  parser.add_argument('--gpgdir',
+                      default=os.path.join(util.HOMEDIR, '.gnupg', 'repo'),
+                      help='path to dedicated gpg dir with release keys '
+                           '(default: ~/.gnupg/repo/)')
+  parser.add_argument('-f', '--force', action='store_true',
+                      help='force signing of any tag')
+  parser.add_argument('--keyid', dest='key',
+                      help='alternative signing key to use')
+  parser.add_argument('tag',
+                      help='the tag to create (e.g. "v2.0")')
+  parser.add_argument('commit', default='HEAD', nargs='?',
+                      help='the commit to tag')
+  return parser
+
+
+def main(argv):
+  """The main func!"""
+  parser = get_parser()
+  opts = parser.parse_args(argv)
+
+  if not os.path.exists(opts.gpgdir):
+    parser.error(f'--gpgdir does not exist: {opts.gpgdir}')
+
+  if not opts.force and not re.match(RE_VALID_TAG, opts.tag):
+    parser.error(f'tag "{opts.tag}" does not match regex "{RE_VALID_TAG}"; '
+                 'use --force to sign anyways')
+
+  if opts.key:
+    print(f'Using custom key to sign: {opts.key}')
+  else:
+    print('Using official Repo release key to sign')
+    opts.key = KEYID
+    util.import_release_key(opts)
+
+  sign(opts)
+  check(opts)
+  postmsg(opts)
+
+  return 0
+
+
+if __name__ == '__main__':
+  sys.exit(main(sys.argv[1:]))

release/util.py

@@ -0,0 +1,73 @@
+# Copyright (C) 2020 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Random utility code for release tools."""
+
+import os
+import re
+import subprocess
+import sys
+
+
+assert sys.version_info >= (3, 6), 'This module requires Python 3.6+'
+
+
+TOPDIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+HOMEDIR = os.path.expanduser('~')
+
+
+# These are the release keys we sign with.
+KEYID_DSA = '8BB9AD793E8E6153AF0F9A4416530D5E920F5C65'
+KEYID_RSA = 'A34A13BE8E76BFF46A0C022DA2E75A824AAB9624'
+KEYID_ECC = 'E1F9040D7A3F6DAFAC897CD3D3B95DA243E48A39'
+
+
+def cmdstr(cmd):
+  """Get a nicely quoted shell command."""
+  ret = []
+  for arg in cmd:
+    if not re.match(r'^[a-zA-Z0-9/_.=-]+$', arg):
+      arg = f'"{arg}"'
+    ret.append(arg)
+  return ' '.join(ret)
+
+
+def run(opts, cmd, check=True, **kwargs):
+  """Helper around subprocess.run to include logging."""
+  print('+', cmdstr(cmd))
+  if opts.dryrun:
+    cmd = ['true', '--'] + cmd
+  try:
+    return subprocess.run(cmd, check=check, **kwargs)
+  except subprocess.CalledProcessError as e:
+    print(f'aborting: {e}', file=sys.stderr)
+    sys.exit(1)
+
+
+def import_release_key(opts):
+  """Import the public key of the official release repo signing key."""
+  # Extract the key from our repo launcher.
+  launcher = getattr(opts, 'launcher', os.path.join(TOPDIR, 'repo'))
+  print(f'Importing keys from "{launcher}" launcher script')
+  with open(launcher, encoding='utf-8') as fp:
+    data = fp.read()
+
+  keys = re.findall(
+      r'\n-----BEGIN PGP PUBLIC KEY BLOCK-----\n[^-]*'
+      r'\n-----END PGP PUBLIC KEY BLOCK-----\n', data, flags=re.M)
+  run(opts, ['gpg', '--import'], input='\n'.join(keys).encode('utf-8'))
+
+  print('Marking keys as fully trusted')
+  run(opts, ['gpg', '--import-ownertrust'],
+      input=f'{KEYID_DSA}:6:\n'.encode('utf-8'))

repo

@@ -1,5 +1,19 @@
 #!/usr/bin/env python
 # -*- coding:utf-8 -*-
+#
+# Copyright (C) 2008 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
 """Repo launcher.
 
@@ -13,12 +27,34 @@ from __future__ import print_function
 import datetime
 import os
 import platform
+import shlex
 import subprocess
 import sys
 
 
+# Keep basic logic in sync with repo_trace.py.
+class Trace(object):
+  """Trace helper logic."""
+
+  REPO_TRACE = 'REPO_TRACE'
+
+  def __init__(self):
+    self.set(os.environ.get(self.REPO_TRACE) == '1')
+
+  def set(self, value):
+    self.enabled = bool(value)
+
+  def print(self, *args, **kwargs):
+    if self.enabled:
+      print(*args, **kwargs)
+
+
+trace = Trace()
+
+
 def exec_command(cmd):
   """Execute |cmd| or return None on failure."""
+  trace.print(':', ' '.join(cmd))
   try:
     if platform.system() == 'Windows':
       ret = subprocess.call(cmd)
@@ -84,14 +120,11 @@ def check_python_version():
 
 
 if __name__ == '__main__':
-  # TODO(vapier): Enable this on Windows once we have Python 3 issues fixed.
-  if platform.system() != 'Windows':
-    check_python_version()
+  check_python_version()
 
 
 # repo default configuration
 #
-import os
 REPO_URL = os.environ.get('REPO_URL', None)
 if not REPO_URL:
   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
@@ -99,25 +132,11 @@ REPO_REV = os.environ.get('REPO_REV')
 if not REPO_REV:
   REPO_REV = 'stable'
 
-# Copyright (C) 2008 Google Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
 # increment this whenever we make important changes to this script
-VERSION = (2, 3)
+VERSION = (2, 5)
 
 # increment this if the MAINTAINER_KEYS block is modified
-KEYRING_VERSION = (2, 0)
+KEYRING_VERSION = (2, 3)
 
 # Each individual key entry is created by using:
 # gpg --armor --export keyid
@@ -125,7 +144,6 @@ MAINTAINER_KEYS = """
 
      Repo Maintainer <repo@android.kernel.org>
 -----BEGIN PGP PUBLIC KEY BLOCK-----
-Version: GnuPG v1.4.2.2 (GNU/Linux)
 
 mQGiBEj3ugERBACrLJh/ZPyVSKeClMuznFIrsQ+hpNnmJGw1a9GXKYKk8qHPhAZf
 WKtrBqAVMNRLhL85oSlekRz98u41H5si5zcuv+IXJDF5MJYcB8f22wAy15lUqPWi
@@ -161,8 +179,39 @@ p3v5ILwfC7hVx4jHSnOgZ65L9s8EQdVr1ckN9243yta7rNgwfcqb60ILMFF1BRk/
 5xGrFy8tfAaeBMIQ17gvFSp/suc9DYO0ICK2BISzq+F+ZiAKsjMYOBNdH/h0zobQ
 HTHs37+/QLMomGEGKZMWi0dShU2J5mNRQu3Hhxl3hHDVbt5CeJBb26aQcQrFz69W
 zE3GNvmJosh6leayjtI9P2A6iEkEGBECAAkFAkj3uiACGwwACgkQFlMNXpIPXGWp
-TACbBS+Up3RpfYVfd63c1cDdlru13pQAn3NQy/SN858MkxN+zym86UBgOad2
-=CMiZ
+TACbBS+Up3RpfYVfd63c1cDdlru13pQAn3NQy/SN858MkxN+zym86UBgOad2uQIN
+BF5FqOoBEAC8aRtWEtXzeuoQhdFrLTqYs2dy6kl9y+j3DMQYAMs8je582qzUigIO
+ZZxq7T/3WQgghsdw9yPvdzlw9tKdet2TJkR1mtBfSjZQrkKwR0pQP4AD7t/90Whu
+R8Wlu8ysapE2hLxMH5Y2znRQX2LkUYmk0K2ik9AgZEh3AFEg3YLl2pGnSjeSp3ch
+cLX2n/rVZf5LXluZGRG+iov1Ka+8m+UqzohMA1DYNECJW6KPgXsNX++i8/iwZVic
+PWzhRJSQC+QiAZNsKT6HNNKs97YCUVzhjBLnRSxRBPkr0hS/VMWY2V4pbASljWyd
+GYmlDcxheLne0yjes0bJAdvig5rB42FOV0FCM4bDYOVwKfZ7SpzGCYXxtlwe0XNG
+tLW9WA6tICVqNZ/JNiRTBLrsGSkyrEhDPKnIHlHRI5Zux6IHwMVB0lQKHjSop+t6
+oyubqWcPCGGYdz2QGQHNz7huC/Zn0wS4hsoiSwPv6HCq3jNyUkOJ7wZ3ouv60p2I
+kPurgviVaRaPSKTYdKfkcJOtFeqOh1na5IHkXsD9rNctB7tSgfsm0G6qJIVe3ZmJ
+7QAyHBfuLrAWCq5xS8EHDlvxPdAD8EEsa9T32YxcHKIkxr1eSwrUrKb8cPhWq1pp
+Jiylw6G1fZ02VKixqmPC4oFMyg1PO8L2tcQTrnVmZvfFGiaekHKdhQARAQABiQKW
+BBgRAgAgFiEEi7mteT6OYVOvD5pEFlMNXpIPXGUFAl5FqOoCGwICQAkQFlMNXpIP
+XGXBdCAEGQEKAB0WIQSjShO+jna/9GoMAi2i51qCSquWJAUCXkWo6gAKCRCi51qC
+SquWJLzgD/0YEZYS7yKxhP+kk94TcTYMBMSZpU5KFClB77yu4SI1LeXq4ocBT4sp
+EPaOsQiIx//j59J67b7CBe4UeRA6D2n0pw+bCKuc731DFi5X9C1zq3a7E67SQ2yd
+FbYE2fnpVnMqb62g4sTh7JmdxEtXCWBUWL0OEoWouBW1PkFDHx2kYLC7YpZt3+4t
+VtNhSfV8NS6PF8ep3JXHVd2wsC3DQtggeId5GM44o8N0SkwQHNjK8ZD+VZ74ZnhZ
+HeyHskomiOC61LrZWQvxD6VqtfnBQ5GvONO8QuhkiFwMMOnpPVj2k7ngSkd5o27K
+6c53ZESOlR4bAfl0i3RZYC9B5KerGkBE3dTgTzmGjOaahl2eLz4LDPdTwMtS+sAU
+1hPPvZTQeYDdV62bOWUyteMoJu354GgZPQ9eItWYixpNCyOGNcJXl6xk3/OuoP6f
+MciFV8aMxs/7mUR8q1Ei3X9MKu+bbODYj2rC1tMkLj1OaAJkfvRuYrKsQpoUsn4q
+VT9+aciNpU/I7M30watlWo7RfUFI3zaGdMDcMFju1cWt2Un8E3gtscGufzbz1Z5Z
+Gak+tCOWUyuYNWX3noit7Dk6+3JGHGaQettldNu2PLM9SbIXd2EaqK/eEv9BS3dd
+ItkZwzyZXSaQ9UqAceY1AHskJJ5KVXIRLuhP5jBWWo3fnRMyMYt2nwNBAJ9B9TA8
+VlBniwIl5EzCvOFOTGrtewCdHOvr3N3ieypGz1BzyCN9tJMO3G24MwReRal9Fgkr
+BgEEAdpHDwEBB0BhPE/je6OuKgWzJ1mnrUmHhn4IMOHp+58+T5kHU3Oy6YjXBBgR
+AgAgFiEEi7mteT6OYVOvD5pEFlMNXpIPXGUFAl5FqX0CGwIAgQkQFlMNXpIPXGV2
+IAQZFggAHRYhBOH5BA16P22vrIl809O5XaJD5Io5BQJeRal9AAoJENO5XaJD5Io5
+MEkA/3uLmiwANOcgE0zB9zga0T/KkYhYOWFx7zRyDhrTf9spAPwIfSBOAGtwxjLO
+DCce5OaQJl/YuGHvXq2yx5h7T8pdAZ+PAJ4qfIk2LLSidsplTDXOKhOQAuOqUQCf
+cZ7aFsJF4PtcDrfdejyAxbtsSHI=
+=82Tj
 -----END PGP PUBLIC KEY BLOCK-----
 """
 
@@ -202,26 +251,39 @@ else:
 home_dot_repo = os.path.expanduser('~/.repoconfig')
 gpg_dir = os.path.join(home_dot_repo, 'gnupg')
 
-extra_args = []
-init_optparse = optparse.OptionParser(usage="repo init -u url [options]")
 
-def _InitParser():
-  """Setup the init subcommand parser."""
+def GetParser(gitc_init=False):
+  """Setup the CLI parser."""
+  if gitc_init:
+    usage = 'repo gitc-init -u url -c client [options]'
+  else:
+    usage = 'repo init -u url [options]'
+
+  parser = optparse.OptionParser(usage=usage)
+
   # Logging.
-  group = init_optparse.add_option_group('Logging options')
+  group = parser.add_option_group('Logging options')
+  group.add_option('-v', '--verbose',
+                   dest='output_mode', action='store_true',
+                   help='show all output')
   group.add_option('-q', '--quiet',
-                   action='store_true', default=False,
-                   help='be quiet')
+                   dest='output_mode', action='store_false',
+                   help='only show errors')
 
   # Manifest.
-  group = init_optparse.add_option_group('Manifest options')
+  group = parser.add_option_group('Manifest options')
   group.add_option('-u', '--manifest-url',
                    help='manifest repository location', metavar='URL')
   group.add_option('-b', '--manifest-branch',
                    help='manifest branch or revision', metavar='REVISION')
   group.add_option('-m', '--manifest-name',
                    help='initial manifest file', metavar='NAME.xml')
-  group.add_option('--current-branch',
+  cbr_opts = ['--current-branch']
+  # The gitc-init subcommand allocates -c itself, but a lot of init users
+  # want -c, so try to satisfy both as best we can.
+  if not gitc_init:
+    cbr_opts += ['-c']
+  group.add_option(*cbr_opts,
                    dest='current_branch_only', action='store_true',
                    help='fetch only current manifest branch from server')
   group.add_option('--mirror', action='store_true',
@@ -240,6 +302,8 @@ def _InitParser():
   group.add_option('--clone-filter', action='store', default='blob:none',
                    help='filter for use with --partial-clone '
                         '[default: %default]')
+  group.add_option('--worktree', action='store_true',
+                   help=optparse.SUPPRESS_HELP)
   group.add_option('--archive', action='store_true',
                    help='checkout an archive instead of a git repository for '
                         'each project. See git archive.')
@@ -253,36 +317,104 @@ def _InitParser():
                    help='restrict manifest projects to ones with a specified '
                         'platform group [auto|all|none|linux|darwin|...]',
                    metavar='PLATFORM')
-  group.add_option('--no-clone-bundle', action='store_true',
+  group.add_option('--no-clone-bundle',
+                   dest='clone_bundle', default=True, action='store_false',
                    help='disable use of /clone.bundle on HTTP/HTTPS')
-  group.add_option('--no-tags', action='store_true',
+  group.add_option('--no-tags',
+                   dest='tags', default=True, action='store_false',
                    help="don't fetch tags in the manifest")
 
   # Tool.
-  group = init_optparse.add_option_group('repo Version options')
+  group = parser.add_option_group('repo Version options')
   group.add_option('--repo-url', metavar='URL',
                    help='repo repository location ($REPO_URL)')
-  group.add_option('--repo-branch', metavar='REVISION',
+  group.add_option('--repo-rev', metavar='REV',
                    help='repo branch or revision ($REPO_REV)')
-  group.add_option('--no-repo-verify', action='store_true',
+  group.add_option('--repo-branch', dest='repo_rev',
+                   help=optparse.SUPPRESS_HELP)
+  group.add_option('--no-repo-verify',
+                   dest='repo_verify', default=True, action='store_false',
                    help='do not verify repo source code')
 
   # Other.
-  group = init_optparse.add_option_group('Other options')
+  group = parser.add_option_group('Other options')
   group.add_option('--config-name',
                    action='store_true', default=False,
                    help='Always prompt for name/e-mail')
 
+  # gitc-init specific settings.
+  if gitc_init:
+    group = parser.add_option_group('GITC options')
+    group.add_option('-f', '--manifest-file',
+                     help='Optional manifest file to use for this GITC client.')
+    group.add_option('-c', '--gitc-client',
+                     help='Name of the gitc_client instance to create or modify.')
+
+  return parser
+
+
+# This is a poor replacement for subprocess.run until we require Python 3.6+.
+RunResult = collections.namedtuple(
+    'RunResult', ('returncode', 'stdout', 'stderr'))
+
+
+class RunError(Exception):
+  """Error when running a command failed."""
+
+
+def run_command(cmd, **kwargs):
+  """Run |cmd| and return its output."""
+  check = kwargs.pop('check', False)
+  if kwargs.pop('capture_output', False):
+    kwargs.setdefault('stdout', subprocess.PIPE)
+    kwargs.setdefault('stderr', subprocess.PIPE)
+  cmd_input = kwargs.pop('input', None)
+
+  def decode(output):
+    """Decode |output| to text."""
+    if output is None:
+      return output
+    try:
+      return output.decode('utf-8')
+    except UnicodeError:
+      print('repo: warning: Invalid UTF-8 output:\ncmd: %r\n%r' % (cmd, output),
+            file=sys.stderr)
+      # TODO(vapier): Once we require Python 3, use 'backslashreplace'.
+      return output.decode('utf-8', 'replace')
+
+  # Run & package the results.
+  proc = subprocess.Popen(cmd, **kwargs)
+  (stdout, stderr) = proc.communicate(input=cmd_input)
+  dbg = ': ' + ' '.join(cmd)
+  if cmd_input is not None:
+    dbg += ' 0<|'
+  if stdout == subprocess.PIPE:
+    dbg += ' 1>|'
+  if stderr == subprocess.PIPE:
+    dbg += ' 2>|'
+  elif stderr == subprocess.STDOUT:
+    dbg += ' 2>&1'
+  trace.print(dbg)
+  ret = RunResult(proc.returncode, decode(stdout), decode(stderr))
+
+  # If things failed, print useful debugging output.
+  if check and ret.returncode:
+    print('repo: error: "%s" failed with exit status %s' %
+          (cmd[0], ret.returncode), file=sys.stderr)
+    print('  cwd: %s\n  cmd: %r' %
+          (kwargs.get('cwd', os.getcwd()), cmd), file=sys.stderr)
+
+    def _print_output(name, output):
+      if output:
+        print('  %s:\n  >> %s' % (name, '\n  >> '.join(output.splitlines())),
+              file=sys.stderr)
+
+    _print_output('stdout', ret.stdout)
+    _print_output('stderr', ret.stderr)
+    raise RunError(ret)
+
+  return ret
 
-def _GitcInitOptions(init_optparse_arg):
-  init_optparse_arg.set_usage("repo gitc-init -u url -c client [options]")
-  g = init_optparse_arg.add_option_group('GITC options')
-  g.add_option('-f', '--manifest-file',
-               dest='manifest_file',
-               help='Optional manifest file to use for this GITC client.')
-  g.add_option('-c', '--gitc-client',
-               dest='gitc_client',
-               help='The name of the gitc_client instance to create or modify.')
 
 _gitc_manifest_dir = None
 
@@ -331,31 +463,47 @@ class CloneFailure(Exception):
   """
 
 
+def check_repo_verify(repo_verify, quiet=False):
+  """Check the --repo-verify state."""
+  if not repo_verify:
+    print('repo: warning: verification of repo code has been disabled;\n'
+          'repo will not be able to verify the integrity of itself.\n',
+          file=sys.stderr)
+    return False
+
+  if NeedSetupGnuPG():
+    return SetupGnuPG(quiet)
+
+  return True
+
+
+def check_repo_rev(dst, rev, repo_verify=True, quiet=False):
+  """Check that |rev| is valid."""
+  do_verify = check_repo_verify(repo_verify, quiet=quiet)
+  remote_ref, local_rev = resolve_repo_rev(dst, rev)
+  if not quiet and not remote_ref.startswith('refs/heads/'):
+    print('warning: repo is not tracking a remote branch, so it will not '
+          'receive updates', file=sys.stderr)
+  if do_verify:
+    rev = verify_rev(dst, remote_ref, local_rev, quiet)
+  else:
+    rev = local_rev
+  return (remote_ref, rev)
+
+
 def _Init(args, gitc_init=False):
   """Installs repo by cloning it over the network.
   """
-  if gitc_init:
-    _GitcInitOptions(init_optparse)
-  opt, args = init_optparse.parse_args(args)
+  parser = GetParser(gitc_init=gitc_init)
+  opt, args = parser.parse_args(args)
   if args:
-    init_optparse.print_usage()
+    parser.print_usage()
     sys.exit(1)
+  opt.quiet = opt.output_mode is False
+  opt.verbose = opt.output_mode is True
 
-  url = opt.repo_url
-  if not url:
-    url = REPO_URL
-    extra_args.append('--repo-url=%s' % url)
-
-  branch = opt.repo_branch
-  if not branch:
-    branch = REPO_REV
-    extra_args.append('--repo-branch=%s' % branch)
-
-  if branch.startswith('refs/heads/'):
-    branch = branch[len('refs/heads/'):]
-  if branch.startswith('refs/'):
-    print("fatal: invalid branch name '%s'" % branch, file=sys.stderr)
-    raise CloneFailure()
+  url = opt.repo_url or REPO_URL
+  rev = opt.repo_rev or REPO_REV
 
   try:
     if gitc_init:
@@ -390,23 +538,13 @@ def _Init(args, gitc_init=False):
 
   _CheckGitVersion()
   try:
-    if opt.no_repo_verify:
-      do_verify = False
-    else:
-      if NeedSetupGnuPG():
-        do_verify = SetupGnuPG(opt.quiet)
-      else:
-        do_verify = True
-
+    if not opt.quiet:
+      print('Downloading Repo source from', url)
     dst = os.path.abspath(os.path.join(repodir, S_repo))
-    _Clone(url, dst, opt.quiet, not opt.no_clone_bundle)
+    _Clone(url, dst, opt.clone_bundle, opt.quiet, opt.verbose)
 
-    if do_verify:
-      rev = _Verify(dst, branch, opt.quiet)
-    else:
-      rev = 'refs/remotes/origin/%s^0' % branch
-
-    _Checkout(dst, branch, rev, opt.quiet)
+    remote_ref, rev = check_repo_rev(dst, rev, opt.repo_verify, quiet=opt.quiet)
+    _Checkout(dst, remote_ref, rev, opt.quiet)
 
     if not os.path.isfile(os.path.join(dst, 'repo')):
       print("warning: '%s' does not look like a git-repo repository, is "
@@ -419,15 +557,34 @@ def _Init(args, gitc_init=False):
     raise
 
 
+def run_git(*args, **kwargs):
+  """Run git and return execution details."""
+  kwargs.setdefault('capture_output', True)
+  kwargs.setdefault('check', True)
+  try:
+    return run_command([GIT] + list(args), **kwargs)
+  except OSError as e:
+    print(file=sys.stderr)
+    print('repo: error: "%s" is not available' % GIT, file=sys.stderr)
+    print('repo: error: %s' % e, file=sys.stderr)
+    print(file=sys.stderr)
+    print('Please make sure %s is installed and in your path.' % GIT,
+          file=sys.stderr)
+    sys.exit(1)
+  except RunError:
+    raise CloneFailure()
+
+
 # The git version info broken down into components for easy analysis.
 # Similar to Python's sys.version_info.
 GitVersion = collections.namedtuple(
     'GitVersion', ('major', 'minor', 'micro', 'full'))
 
+
 def ParseGitVersion(ver_str=None):
   if ver_str is None:
     # Load the version ourselves.
-    ver_str = _GetGitVersion()
+    ver_str = run_git('--version').stdout
 
   if not ver_str.startswith('git version '):
     return None
@@ -444,38 +601,16 @@ def ParseGitVersion(ver_str=None):
   return GitVersion(*to_tuple)
 
 
-def _GetGitVersion():
-  cmd = [GIT, '--version']
-  try:
-    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
-  except OSError as e:
-    print(file=sys.stderr)
-    print("fatal: '%s' is not available" % GIT, file=sys.stderr)
-    print('fatal: %s' % e, file=sys.stderr)
-    print(file=sys.stderr)
-    print('Please make sure %s is installed and in your path.' % GIT,
-          file=sys.stderr)
-    raise
-
-  ver_str = proc.stdout.read().strip()
-  proc.stdout.close()
-  proc.wait()
-  return ver_str.decode('utf-8')
-
-
 def _CheckGitVersion():
-  try:
-    ver_act = ParseGitVersion()
-  except OSError:
-    raise CloneFailure()
-
+  ver_act = ParseGitVersion()
   if ver_act is None:
     print('fatal: unable to detect git version', file=sys.stderr)
     raise CloneFailure()
 
   if ver_act < MIN_GIT_VERSION:
     need = '.'.join(map(str, MIN_GIT_VERSION))
-    print('fatal: git %s or later required' % need, file=sys.stderr)
+    print('fatal: git %s or later required; found %s' % (need, ver_act.full),
+          file=sys.stderr)
     raise CloneFailure()
 
 
@@ -547,40 +682,54 @@ def SetupGnuPG(quiet):
             file=sys.stderr)
       sys.exit(1)
 
-  env = os.environ.copy()
-  _setenv('GNUPGHOME', gpg_dir, env)
-
-  cmd = ['gpg', '--import']
+  if not quiet:
+    print('repo: Updating release signing keys to keyset ver %s' %
+          ('.'.join(str(x) for x in KEYRING_VERSION),))
+  # NB: We use --homedir (and cwd below) because some environments (Windows) do
+  # not correctly handle full native paths.  We avoid the issue by changing to
+  # the right dir with cwd=gpg_dir before executing gpg, and then telling gpg to
+  # use the cwd (.) as its homedir which leaves the path resolution logic to it.
+  cmd = ['gpg', '--homedir', '.', '--import']
   try:
-    proc = subprocess.Popen(cmd,
-                            env=env,
-                            stdin=subprocess.PIPE)
-  except OSError as e:
+    # gpg can be pretty chatty.  Always capture the output and if something goes
+    # wrong, the builtin check failure will dump stdout & stderr for debugging.
+    run_command(cmd, stdin=subprocess.PIPE, capture_output=True,
+                cwd=gpg_dir, check=True,
+                input=MAINTAINER_KEYS.encode('utf-8'))
+  except OSError:
     if not quiet:
       print('warning: gpg (GnuPG) is not available.', file=sys.stderr)
       print('warning: Installing it is strongly encouraged.', file=sys.stderr)
       print(file=sys.stderr)
     return False
 
-  proc.stdin.write(MAINTAINER_KEYS.encode('utf-8'))
-  proc.stdin.close()
-
-  if proc.wait() != 0:
-    print('fatal: registering repo maintainer keys failed', file=sys.stderr)
-    sys.exit(1)
-  print()
-
   with open(os.path.join(home_dot_repo, 'keyring-version'), 'w') as fd:
     fd.write('.'.join(map(str, KEYRING_VERSION)) + '\n')
   return True
 
 
-def _SetConfig(local, name, value):
+def _SetConfig(cwd, name, value):
   """Set a git configuration option to the specified value.
   """
-  cmd = [GIT, 'config', name, value]
-  if subprocess.Popen(cmd, cwd=local).wait() != 0:
-    raise CloneFailure()
+  run_git('config', name, value, cwd=cwd)
+
+
+def _GetRepoConfig(name):
+  """Read a repo configuration option."""
+  config = os.path.join(home_dot_repo, 'config')
+  if not os.path.exists(config):
+    return None
+
+  cmd = ['config', '--file', config, '--get', name]
+  ret = run_git(*cmd, check=False)
+  if ret.returncode == 0:
+    return ret.stdout
+  elif ret.returncode == 1:
+    return None
+  else:
+    print('repo: error: git %s failed:\n%s' % (' '.join(cmd), ret.stderr),
+          file=sys.stderr)
+    raise RunError()
 
 
 def _InitHttp():
@@ -594,7 +743,7 @@ def _InitHttp():
       p = n.hosts[host]
       mgr.add_password(p[1], 'http://%s/' % host, p[0], p[2])
       mgr.add_password(p[1], 'https://%s/' % host, p[0], p[2])
-  except:
+  except Exception:
     pass
   handlers.append(urllib.request.HTTPBasicAuthHandler(mgr))
   handlers.append(urllib.request.HTTPDigestAuthHandler(mgr))
@@ -608,39 +757,29 @@ def _InitHttp():
   urllib.request.install_opener(urllib.request.build_opener(*handlers))
 
 
-def _Fetch(url, local, src, quiet):
-  if not quiet:
-    print('Get %s' % url, file=sys.stderr)
-
-  cmd = [GIT, 'fetch']
-  if quiet:
+def _Fetch(url, cwd, src, quiet, verbose):
+  cmd = ['fetch']
+  if not verbose:
     cmd.append('--quiet')
+  err = None
+  if not quiet and sys.stdout.isatty():
+    cmd.append('--progress')
+  elif not verbose:
     err = subprocess.PIPE
-  else:
-    err = None
   cmd.append(src)
   cmd.append('+refs/heads/*:refs/remotes/origin/*')
   cmd.append('+refs/tags/*:refs/tags/*')
-
-  proc = subprocess.Popen(cmd, cwd=local, stderr=err)
-  if err:
-    proc.stderr.read()
-    proc.stderr.close()
-  if proc.wait() != 0:
-    raise CloneFailure()
+  run_git(*cmd, stderr=err, capture_output=False, cwd=cwd)
 
 
-def _DownloadBundle(url, local, quiet):
+def _DownloadBundle(url, cwd, quiet, verbose):
   if not url.endswith('/'):
     url += '/'
   url += 'clone.bundle'
 
-  proc = subprocess.Popen(
-      [GIT, 'config', '--get-regexp', 'url.*.insteadof'],
-      cwd=local,
-      stdout=subprocess.PIPE)
-  for line in proc.stdout:
-    line = line.decode('utf-8')
+  ret = run_git('config', '--get-regexp', 'url.*.insteadof', cwd=cwd,
+                check=False)
+  for line in ret.stdout.splitlines():
     m = re.compile(r'^url\.(.*)\.insteadof (.*)$').match(line)
     if m:
       new_url = m.group(1)
@@ -648,13 +787,11 @@ def _DownloadBundle(url, local, quiet):
       if url.startswith(old_url):
         url = new_url + url[len(old_url):]
         break
-  proc.stdout.close()
-  proc.wait()
 
   if not url.startswith('http:') and not url.startswith('https:'):
     return False
 
-  dest = open(os.path.join(local, '.git', 'clone.bundle'), 'w+b')
+  dest = open(os.path.join(cwd, '.git', 'clone.bundle'), 'w+b')
   try:
     try:
       r = urllib.request.urlopen(url)
@@ -669,8 +806,8 @@ def _DownloadBundle(url, local, quiet):
       print('fatal: error %s' % e.reason, file=sys.stderr)
       raise CloneFailure()
     try:
-      if not quiet:
-        print('Get %s' % url, file=sys.stderr)
+      if verbose:
+        print('Downloading clone bundle %s' % url, file=sys.stderr)
       while True:
         buf = r.read(8192)
         if not buf:
@@ -682,121 +819,139 @@ def _DownloadBundle(url, local, quiet):
     dest.close()
 
 
-def _ImportBundle(local):
-  path = os.path.join(local, '.git', 'clone.bundle')
+def _ImportBundle(cwd):
+  path = os.path.join(cwd, '.git', 'clone.bundle')
   try:
-    _Fetch(local, local, path, True)
+    _Fetch(cwd, cwd, path, True, False)
   finally:
     os.remove(path)
 
 
-def _Clone(url, local, quiet, clone_bundle):
+def _Clone(url, cwd, clone_bundle, quiet, verbose):
   """Clones a git repository to a new subdirectory of repodir
   """
+  if verbose:
+    print('Cloning git repository', url)
+
   try:
-    os.mkdir(local)
+    os.mkdir(cwd)
   except OSError as e:
-    print('fatal: cannot make %s directory: %s' % (local, e.strerror),
+    print('fatal: cannot make %s directory: %s' % (cwd, e.strerror),
           file=sys.stderr)
     raise CloneFailure()
 
-  cmd = [GIT, 'init', '--quiet']
-  try:
-    proc = subprocess.Popen(cmd, cwd=local)
-  except OSError as e:
-    print(file=sys.stderr)
-    print("fatal: '%s' is not available" % GIT, file=sys.stderr)
-    print('fatal: %s' % e, file=sys.stderr)
-    print(file=sys.stderr)
-    print('Please make sure %s is installed and in your path.' % GIT,
-          file=sys.stderr)
-    raise CloneFailure()
-  if proc.wait() != 0:
-    print('fatal: could not create %s' % local, file=sys.stderr)
-    raise CloneFailure()
+  run_git('init', '--quiet', cwd=cwd)
 
   _InitHttp()
-  _SetConfig(local, 'remote.origin.url', url)
-  _SetConfig(local,
+  _SetConfig(cwd, 'remote.origin.url', url)
+  _SetConfig(cwd,
              'remote.origin.fetch',
              '+refs/heads/*:refs/remotes/origin/*')
-  if clone_bundle and _DownloadBundle(url, local, quiet):
-    _ImportBundle(local)
-  _Fetch(url, local, 'origin', quiet)
+  if clone_bundle and _DownloadBundle(url, cwd, quiet, verbose):
+    _ImportBundle(cwd)
+  _Fetch(url, cwd, 'origin', quiet, verbose)
 
 
-def _Verify(cwd, branch, quiet):
-  """Verify the branch has been signed by a tag.
+def resolve_repo_rev(cwd, committish):
+  """Figure out what REPO_REV represents.
+
+  We support:
+  * refs/heads/xxx: Branch.
+  * refs/tags/xxx: Tag.
+  * xxx: Branch or tag or commit.
+
+  Args:
+    cwd: The git checkout to run in.
+    committish: The REPO_REV argument to resolve.
+
+  Returns:
+    A tuple of (remote ref, commit) as makes sense for the committish.
+    For branches, this will look like ('refs/heads/stable', <revision>).
+    For tags, this will look like ('refs/tags/v1.0', <revision>).
+    For commits, this will be (<revision>, <revision>).
   """
-  cmd = [GIT, 'describe', 'origin/%s' % branch]
-  proc = subprocess.Popen(cmd,
-                          stdout=subprocess.PIPE,
-                          stderr=subprocess.PIPE,
-                          cwd=cwd)
-  cur = proc.stdout.read().strip().decode('utf-8')
-  proc.stdout.close()
+  def resolve(committish):
+    ret = run_git('rev-parse', '--verify', '%s^{commit}' % (committish,),
+                  cwd=cwd, check=False)
+    return None if ret.returncode else ret.stdout.strip()
 
-  proc.stderr.read()
-  proc.stderr.close()
+  # An explicit branch.
+  if committish.startswith('refs/heads/'):
+    remote_ref = committish
+    committish = committish[len('refs/heads/'):]
+    rev = resolve('refs/remotes/origin/%s' % committish)
+    if rev is None:
+      print('repo: error: unknown branch "%s"' % (committish,),
+            file=sys.stderr)
+      raise CloneFailure()
+    return (remote_ref, rev)
 
-  if proc.wait() != 0 or not cur:
-    print(file=sys.stderr)
-    print("fatal: branch '%s' has not been signed" % branch, file=sys.stderr)
-    raise CloneFailure()
+  # An explicit tag.
+  if committish.startswith('refs/tags/'):
+    remote_ref = committish
+    committish = committish[len('refs/tags/'):]
+    rev = resolve(remote_ref)
+    if rev is None:
+      print('repo: error: unknown tag "%s"' % (committish,),
+            file=sys.stderr)
+      raise CloneFailure()
+    return (remote_ref, rev)
+
+  # See if it's a short branch name.
+  rev = resolve('refs/remotes/origin/%s' % committish)
+  if rev:
+    return ('refs/heads/%s' % (committish,), rev)
+
+  # See if it's a tag.
+  rev = resolve('refs/tags/%s' % committish)
+  if rev:
+    return ('refs/tags/%s' % (committish,), rev)
+
+  # See if it's a commit.
+  rev = resolve(committish)
+  if rev and rev.lower().startswith(committish.lower()):
+    return (rev, rev)
+
+  # Give up!
+  print('repo: error: unable to resolve "%s"' % (committish,), file=sys.stderr)
+  raise CloneFailure()
+
+
+def verify_rev(cwd, remote_ref, rev, quiet):
+  """Verify the commit has been signed by a tag."""
+  ret = run_git('describe', rev, cwd=cwd)
+  cur = ret.stdout.strip()
 
   m = re.compile(r'^(.*)-[0-9]{1,}-g[0-9a-f]{1,}$').match(cur)
   if m:
     cur = m.group(1)
     if not quiet:
       print(file=sys.stderr)
-      print("info: Ignoring branch '%s'; using tagged release '%s'"
-            % (branch, cur), file=sys.stderr)
+      print("warning: '%s' is not signed; falling back to signed release '%s'"
+            % (remote_ref, cur), file=sys.stderr)
       print(file=sys.stderr)
 
   env = os.environ.copy()
   _setenv('GNUPGHOME', gpg_dir, env)
-
-  cmd = [GIT, 'tag', '-v', cur]
-  proc = subprocess.Popen(cmd,
-                          stdout=subprocess.PIPE,
-                          stderr=subprocess.PIPE,
-                          cwd=cwd,
-                          env=env)
-  out = proc.stdout.read().decode('utf-8')
-  proc.stdout.close()
-
-  err = proc.stderr.read().decode('utf-8')
-  proc.stderr.close()
-
-  if proc.wait() != 0:
-    print(file=sys.stderr)
-    print(out, file=sys.stderr)
-    print(err, file=sys.stderr)
-    print(file=sys.stderr)
-    raise CloneFailure()
+  run_git('tag', '-v', cur, cwd=cwd, env=env)
   return '%s^0' % cur
 
 
-def _Checkout(cwd, branch, rev, quiet):
+def _Checkout(cwd, remote_ref, rev, quiet):
   """Checkout an upstream branch into the repository and track it.
   """
-  cmd = [GIT, 'update-ref', 'refs/heads/default', rev]
-  if subprocess.Popen(cmd, cwd=cwd).wait() != 0:
-    raise CloneFailure()
+  run_git('update-ref', 'refs/heads/default', rev, cwd=cwd)
 
   _SetConfig(cwd, 'branch.default.remote', 'origin')
-  _SetConfig(cwd, 'branch.default.merge', 'refs/heads/%s' % branch)
+  _SetConfig(cwd, 'branch.default.merge', remote_ref)
 
-  cmd = [GIT, 'symbolic-ref', 'HEAD', 'refs/heads/default']
-  if subprocess.Popen(cmd, cwd=cwd).wait() != 0:
-    raise CloneFailure()
+  run_git('symbolic-ref', 'HEAD', 'refs/heads/default', cwd=cwd)
 
-  cmd = [GIT, 'read-tree', '--reset', '-u']
+  cmd = ['read-tree', '--reset', '-u']
   if not quiet:
     cmd.append('-v')
   cmd.append('HEAD')
-  if subprocess.Popen(cmd, cwd=cwd).wait() != 0:
-    raise CloneFailure()
+  run_git(*cmd, cwd=cwd)
 
 
 def _FindRepo():
@@ -822,6 +977,25 @@ class _Options(object):
   version = False
 
 
+def _ExpandAlias(name):
+  """Look up user registered aliases."""
+  # We don't resolve aliases for existing subcommands.  This matches git.
+  if name in {'gitc-init', 'help', 'init'}:
+    return name, []
+
+  alias = _GetRepoConfig('alias.%s' % (name,))
+  if alias is None:
+    return name, []
+
+  args = alias.strip().split(' ', 1)
+  name = args[0]
+  if len(args) == 2:
+    args = shlex.split(args[1])
+  else:
+    args = []
+  return name, args
+
+
 def _ParseArguments(args):
   cmd = None
   opt = _Options()
@@ -833,6 +1007,8 @@ def _ParseArguments(args):
       opt.help = True
     elif a == '--version':
       opt.version = True
+    elif a == '--trace':
+      trace.set(True)
     elif not a.startswith('-'):
       cmd = a
       arg = args[i + 1:]
@@ -863,12 +1039,9 @@ For access to the full online help, install repo ("repo init").
 
 def _Help(args):
   if args:
-    if args[0] == 'init':
-      init_optparse.print_help()
-      sys.exit(0)
-    elif args[0] == 'gitc-init':
-      _GitcInitOptions(init_optparse)
-      init_optparse.print_help()
+    if args[0] in {'init', 'gitc-init'}:
+      parser = GetParser(gitc_init=args[0] == 'gitc-init')
+      parser.print_help()
       sys.exit(0)
     else:
       print("error: '%s' is not a bootstrap command.\n"
@@ -886,6 +1059,14 @@ def _Version():
   print('       (from %s)' % (__file__,))
   print('git %s' % (ParseGitVersion().full,))
   print('Python %s' % sys.version)
+  uname = platform.uname()
+  if sys.version_info.major < 3:
+    # Python 3 returns a named tuple, but Python 2 is simpler.
+    print(uname)
+  else:
+    print('OS %s %s (%s)' % (uname.system, uname.release, uname.version))
+    print('CPU %s (%s)' %
+          (uname.machine, uname.processor if uname.processor else 'unknown'))
   sys.exit(0)
 
 
@@ -921,21 +1102,18 @@ def _SetDefaultsTo(gitdir):
   global REPO_REV
 
   REPO_URL = gitdir
-  proc = subprocess.Popen([GIT,
-                           '--git-dir=%s' % gitdir,
-                           'symbolic-ref',
-                           'HEAD'],
-                          stdout=subprocess.PIPE,
-                          stderr=subprocess.PIPE)
-  REPO_REV = proc.stdout.read().strip().decode('utf-8')
-  proc.stdout.close()
+  ret = run_git('--git-dir=%s' % gitdir, 'symbolic-ref', 'HEAD', check=False)
+  if ret.returncode:
+    # If we're not tracking a branch (bisect/etc...), then fall back to commit.
+    print('repo: warning: %s has no current branch; using HEAD' % gitdir,
+          file=sys.stderr)
+    try:
+      ret = run_git('rev-parse', 'HEAD', cwd=gitdir)
+    except CloneFailure:
+      print('fatal: %s has invalid HEAD' % gitdir, file=sys.stderr)
+      sys.exit(1)
 
-  proc.stderr.read()
-  proc.stderr.close()
-
-  if proc.wait() != 0:
-    print('fatal: %s has no current branch' % gitdir, file=sys.stderr)
-    sys.exit(1)
+  REPO_REV = ret.stdout.strip()
 
 
 def main(orig_args):
@@ -959,8 +1137,12 @@ def main(orig_args):
           'command from the corresponding client under /gitc/',
           file=sys.stderr)
     sys.exit(1)
-  _InitParser()
   if not repo_main:
+    # Only expand aliases here since we'll be parsing the CLI ourselves.
+    # If we had repo_main, alias expansion would happen in main.py.
+    cmd, alias_args = _ExpandAlias(cmd)
+    args = alias_args + args
+
     if opt.help:
       _Usage()
     if cmd == 'help':
@@ -994,7 +1176,6 @@ def main(orig_args):
         '--wrapper-path=%s' % wrapper_path,
         '--']
   me.extend(orig_args)
-  me.extend(extra_args)
   exec_command(me)
   print("fatal: unable to start %s" % repo_main, file=sys.stderr)
   sys.exit(148)

run_tests

@@ -42,9 +42,11 @@ def main(argv):
   """The main entry."""
   # Add the repo tree to PYTHONPATH as the tests expect to be able to import
   # modules directly.
-  topdir = os.path.dirname(os.path.realpath(__file__))
-  pythonpath = os.environ.get('PYTHONPATH', '')
-  os.environ['PYTHONPATH'] = '%s:%s' % (topdir, pythonpath)
+  pythonpath = os.path.dirname(os.path.realpath(__file__))
+  oldpythonpath = os.environ.get('PYTHONPATH', None)
+  if oldpythonpath is not None:
+    pythonpath += os.pathsep + oldpythonpath
+  os.environ['PYTHONPATH'] = pythonpath
 
   return run_pytest('pytest', argv)
 

subcmds/__init__.py

@@ -16,6 +16,7 @@
 
 import os
 
+# A mapping of the subcommand name to the class that implements it.
 all_commands = {}
 
 my_dir = os.path.dirname(__file__)
@@ -37,7 +38,7 @@ for py in os.listdir(my_dir):
                      ['%s' % name])
     mod = getattr(mod, name)
     try:
-      cmd = getattr(mod, clsn)()
+      cmd = getattr(mod, clsn)
     except AttributeError:
       raise SyntaxError('%s/%s does not define class %s' % (
           __name__, py, clsn))
@@ -46,5 +47,5 @@ for py in os.listdir(my_dir):
     cmd.NAME = name
     all_commands[name] = cmd
 
-if 'help' in all_commands:
-  all_commands['help'].commands = all_commands
+# Add 'branch' as an alias for 'branches'.
+all_commands['branch'] = all_commands['branches']

subcmds/abandon.py

@@ -15,9 +15,11 @@
 # limitations under the License.
 
 from __future__ import print_function
-import sys
-from command import Command
+
 from collections import defaultdict
+import sys
+
+from command import Command
 from git_command import git
 from progress import Progress
 
@@ -35,6 +37,9 @@ It is equivalent to "git branch -D <branchname>".
 """
 
   def _Options(self, p):
+    p.add_option('-q', '--quiet',
+                 action='store_true', default=False,
+                 help='be quiet')
     p.add_option('--all',
                  dest='all', action='store_true',
                  help='delete all branches in all projects')
@@ -91,11 +96,14 @@ It is equivalent to "git branch -D <branchname>".
             file=sys.stderr)
       sys.exit(1)
     else:
-      print('Abandoned branches:', file=sys.stderr)
+      # Everything below here is displaying status.
+      if opt.quiet:
+        return
+      print('Abandoned branches:')
       for br in success.keys():
         if len(all_projects) > 1 and len(all_projects) == len(success[br]):
           result = "all project"
         else:
           result = "%s" % (
               ('\n' + ' ' * width + '| ').join(p.relpath for p in success[br]))
-        print("%s%s| %s\n" % (br, ' ' * (width - len(br)), result), file=sys.stderr)
+        print("%s%s| %s\n" % (br, ' ' * (width - len(br)), result))

subcmds/diff.py

@@ -29,10 +29,6 @@ to the Unix 'patch' command.
 """
 
   def _Options(self, p):
-    def cmd(option, opt_str, value, parser):
-      setattr(parser.values, option.dest, list(parser.rargs))
-      while parser.rargs:
-        del parser.rargs[0]
     p.add_option('-u', '--absolute',
                  dest='absolute', action='store_true',
                  help='Paths are relative to the repository root')

subcmds/diffmanifests.py

@@ -79,7 +79,7 @@ synced and their revisions won't be found.
                  metavar='<FORMAT>',
                  help='print the log using a custom git pretty format string')
 
-  def _printRawDiff(self, diff):
+  def _printRawDiff(self, diff, pretty_format=None):
     for project in diff['added']:
       self.printText("A %s %s" % (project.relpath, project.revisionExpr))
       self.out.nl()
@@ -92,7 +92,7 @@ synced and their revisions won't be found.
       self.printText("C %s %s %s" % (project.relpath, project.revisionExpr,
                                      otherProject.revisionExpr))
       self.out.nl()
-      self._printLogs(project, otherProject, raw=True, color=False)
+      self._printLogs(project, otherProject, raw=True, color=False, pretty_format=pretty_format)
 
     for project, otherProject in diff['unreachable']:
       self.printText("U %s %s %s" % (project.relpath, project.revisionExpr,
@@ -203,6 +203,6 @@ synced and their revisions won't be found.
 
     diff = manifest1.projectsDiff(manifest2)
     if opt.raw:
-      self._printRawDiff(diff)
+      self._printRawDiff(diff, pretty_format=opt.pretty_format)
     else:
       self._printDiff(diff, color=opt.color, pretty_format=opt.pretty_format)

subcmds/download.py

@@ -37,9 +37,13 @@ If no project is specified try to use current directory as a project.
 """
 
   def _Options(self, p):
+    p.add_option('-b', '--branch',
+                 help='create a new branch first')
     p.add_option('-c', '--cherry-pick',
                  dest='cherrypick', action='store_true',
                  help="cherry-pick instead of checkout")
+    p.add_option('-x', '--record-origin', action='store_true',
+                 help='pass -x when cherry-picking')
     p.add_option('-r', '--revert',
                  dest='revert', action='store_true',
                  help="revert instead of checkout")
@@ -78,6 +82,14 @@ If no project is specified try to use current directory as a project.
         project = self.GetProjects([a])[0]
     return to_get
 
+  def ValidateOptions(self, opt, args):
+    if opt.record_origin:
+      if not opt.cherrypick:
+        self.OptionParser.error('-x only makes sense with --cherry-pick')
+
+      if opt.ffonly:
+        self.OptionParser.error('-x and --ff are mutually exclusive options')
+
   def Execute(self, opt, args):
     for project, change_id, ps_id in self._ParseChangeIds(args):
       dl = project.DownloadPatchSet(change_id, ps_id)
@@ -99,17 +111,36 @@ If no project is specified try to use current directory as a project.
               file=sys.stderr)
         for c in dl.commits:
           print('  %s' % (c), file=sys.stderr)
-      if opt.cherrypick:
-        try:
-          project._CherryPick(dl.commit)
-        except GitError:
-          print('[%s] Could not complete the cherry-pick of %s'
-                % (project.name, dl.commit), file=sys.stderr)
-          sys.exit(1)
 
+      if opt.cherrypick:
+        mode = 'cherry-pick'
       elif opt.revert:
-        project._Revert(dl.commit)
+        mode = 'revert'
       elif opt.ffonly:
-        project._FastForward(dl.commit, ffonly=True)
+        mode = 'fast-forward merge'
       else:
-        project._Checkout(dl.commit)
+        mode = 'checkout'
+
+      # We'll combine the branch+checkout operation, but all the rest need a
+      # dedicated branch start.
+      if opt.branch and mode != 'checkout':
+        project.StartBranch(opt.branch)
+
+      try:
+        if opt.cherrypick:
+          project._CherryPick(dl.commit, ffonly=opt.ffonly,
+                              record_origin=opt.record_origin)
+        elif opt.revert:
+          project._Revert(dl.commit)
+        elif opt.ffonly:
+          project._FastForward(dl.commit, ffonly=True)
+        else:
+          if opt.branch:
+            project.StartBranch(opt.branch, revision=dl.commit)
+          else:
+            project._Checkout(dl.commit)
+
+      except GitError:
+        print('[%s] Could not complete the %s of %s'
+              % (project.name, mode, dl.commit), file=sys.stderr)
+        sys.exit(1)

subcmds/forall.py

@@ -127,7 +127,8 @@ without iterating through the remaining projects.
                  help="Execute the command only on projects matching regex or wildcard expression")
     p.add_option('-i', '--inverse-regex',
                  dest='inverse_regex', action='store_true',
-                 help="Execute the command only on projects not matching regex or wildcard expression")
+                 help="Execute the command only on projects not matching regex or "
+                      "wildcard expression")
     p.add_option('-g', '--groups',
                  dest='groups',
                  help="Execute the command only on projects matching the specified groups")
@@ -309,8 +310,6 @@ def DoWork(project, mirror, opt, cmd, shell, cnt, config):
   def setenv(name, val):
     if val is None:
       val = ''
-    if hasattr(val, 'encode'):
-      val = val.encode()
     env[name] = val
 
   setenv('REPO_PROJECT', project['name'])

subcmds/gitc_delete.py

@@ -22,7 +22,7 @@ import platform_utils
 
 from pyversion import is_python3
 if not is_python3():
-  input = raw_input
+  input = raw_input  # noqa: F821
 
 
 class GitcDelete(Command, GitcClientCommand):

subcmds/gitc_init.py

@@ -62,7 +62,8 @@ use for this GITC client.
   def Execute(self, opt, args):
     gitc_client = gitc_utils.parse_clientdir(os.getcwd())
     if not gitc_client or (opt.gitc_client and gitc_client != opt.gitc_client):
-      print('fatal: Please update your repo command. See go/gitc for instructions.', file=sys.stderr)
+      print('fatal: Please update your repo command. See go/gitc for instructions.',
+            file=sys.stderr)
       sys.exit(1)
     self.client_dir = os.path.join(gitc_utils.get_gitc_manifest_dir(),
                                    gitc_client)

subcmds/help.py

@@ -19,6 +19,7 @@ import re
 import sys
 from formatter import AbstractFormatter, DumbWriter
 
+from subcmds import all_commands
 from color import Coloring
 from command import PagedCommand, MirrorSafeCommand, GitcAvailableCommand, GitcClientCommand
 import gitc_utils
@@ -42,7 +43,7 @@ Displays detailed usage information about a command.
     fmt = '  %%-%ds  %%s' % maxlen
 
     for name in commandNames:
-      command = self.commands[name]
+      command = all_commands[name]()
       try:
         summary = command.helpSummary.strip()
       except AttributeError:
@@ -52,7 +53,7 @@ Displays detailed usage information about a command.
   def _PrintAllCommands(self):
     print('usage: repo COMMAND [ARGS]')
     print('The complete list of recognized repo commands are:')
-    commandNames = list(sorted(self.commands))
+    commandNames = list(sorted(all_commands))
     self._PrintCommands(commandNames)
     print("See 'repo help <command>' for more information on a "
           'specific command.')
@@ -73,7 +74,7 @@ Displays detailed usage information about a command.
       return False
 
     commandNames = list(sorted([name
-                                for name, command in self.commands.items()
+                                for name, command in all_commands.items()
                                 if command.common and gitc_supported(command)]))
     self._PrintCommands(commandNames)
 
@@ -132,8 +133,8 @@ Displays detailed usage information about a command.
     out._PrintSection('Description', 'helpDescription')
 
   def _PrintAllCommandHelp(self):
-    for name in sorted(self.commands):
-      cmd = self.commands[name]
+    for name in sorted(all_commands):
+      cmd = all_commands[name]()
       cmd.manifest = self.manifest
       self._PrintCommandHelp(cmd, header_prefix='[%s] ' % (name,))
 
@@ -158,7 +159,7 @@ Displays detailed usage information about a command.
       name = args[0]
 
       try:
-        cmd = self.commands[name]
+        cmd = all_commands[name]()
       except KeyError:
         print("repo: '%s' is not a repo command." % name, file=sys.stderr)
         sys.exit(1)

subcmds/init.py

@@ -15,6 +15,8 @@
 # limitations under the License.
 
 from __future__ import print_function
+
+import optparse
 import os
 import platform
 import re
@@ -36,6 +38,7 @@ from project import SyncBuffer
 from git_config import GitConfig
 from git_command import git_require, MIN_GIT_VERSION_SOFT, MIN_GIT_VERSION_HARD
 import platform_utils
+from wrapper import Wrapper
 
 
 class Init(InteractiveCommand, MirrorSafeCommand):
@@ -85,9 +88,12 @@ to update the working directory files.
   def _Options(self, p, gitc_init=False):
     # Logging
     g = p.add_option_group('Logging options')
+    g.add_option('-v', '--verbose',
+                 dest='output_mode', action='store_true',
+                 help='show all output')
     g.add_option('-q', '--quiet',
-                 dest="quiet", action="store_true", default=False,
-                 help="be quiet")
+                 dest='output_mode', action='store_false',
+                 help='only show errors')
 
     # Manifest
     g = p.add_option_group('Manifest options')
@@ -128,6 +134,10 @@ to update the working directory files.
     g.add_option('--clone-filter', action='store', default='blob:none',
                  dest='clone_filter',
                  help='filter for use with --partial-clone [default: %default]')
+    # TODO(vapier): Expose option with real help text once this has been in the
+    # wild for a while w/out significant bug reports.  Goal is by ~Sep 2020.
+    g.add_option('--worktree', action='store_true',
+                 help=optparse.SUPPRESS_HELP)
     g.add_option('--archive',
                  dest='archive', action='store_true',
                  help='checkout an archive instead of a git repository for '
@@ -146,10 +156,10 @@ to update the working directory files.
                       'platform group [auto|all|none|linux|darwin|...]',
                  metavar='PLATFORM')
     g.add_option('--no-clone-bundle',
-                 dest='no_clone_bundle', action='store_true',
+                 dest='clone_bundle', default=True, action='store_false',
                  help='disable use of /clone.bundle on HTTP/HTTPS')
     g.add_option('--no-tags',
-                 dest='no_tags', action='store_true',
+                 dest='tags', default=True, action='store_false',
                  help="don't fetch tags in the manifest")
 
     # Tool
@@ -157,11 +167,12 @@ to update the working directory files.
     g.add_option('--repo-url',
                  dest='repo_url',
                  help='repo repository location', metavar='URL')
-    g.add_option('--repo-branch',
-                 dest='repo_branch',
-                 help='repo branch or revision', metavar='REVISION')
+    g.add_option('--repo-rev', metavar='REV',
+                 help='repo branch or revision')
+    g.add_option('--repo-branch', dest='repo_rev',
+                 help=optparse.SUPPRESS_HELP)
     g.add_option('--no-repo-verify',
-                 dest='no_repo_verify', action='store_true',
+                 dest='repo_verify', default=True, action='store_false',
                  help='do not verify repo source code')
 
     # Other
@@ -184,7 +195,8 @@ to update the working directory files.
         sys.exit(1)
 
       if not opt.quiet:
-        print('Get %s' % GitConfig.ForUser().UrlInsteadOf(opt.manifest_url),
+        print('Downloading manifest from %s' %
+              (GitConfig.ForUser().UrlInsteadOf(opt.manifest_url),),
               file=sys.stderr)
 
       # The manifest project object doesn't keep track of the path on the
@@ -246,6 +258,20 @@ to update the working directory files.
     if opt.dissociate:
       m.config.SetString('repo.dissociate', 'true')
 
+    if opt.worktree:
+      if opt.mirror:
+        print('fatal: --mirror and --worktree are incompatible',
+              file=sys.stderr)
+        sys.exit(1)
+      if opt.submodules:
+        print('fatal: --submodules and --worktree are incompatible',
+              file=sys.stderr)
+        sys.exit(1)
+      m.config.SetString('repo.worktree', 'true')
+      if is_new:
+        m.use_git_worktrees = True
+      print('warning: --worktree is experimental!', file=sys.stderr)
+
     if opt.archive:
       if is_new:
         m.config.SetString('repo.archive', 'true')
@@ -280,10 +306,10 @@ to update the working directory files.
     if opt.submodules:
       m.config.SetString('repo.submodules', 'true')
 
-    if not m.Sync_NetworkHalf(is_new=is_new, quiet=opt.quiet,
-                              clone_bundle=not opt.no_clone_bundle,
+    if not m.Sync_NetworkHalf(is_new=is_new, quiet=opt.quiet, verbose=opt.verbose,
+                              clone_bundle=opt.clone_bundle,
                               current_branch_only=opt.current_branch_only,
-                              no_tags=opt.no_tags, submodules=opt.submodules,
+                              tags=opt.tags, submodules=opt.submodules,
                               clone_filter=opt.clone_filter):
       r = m.GetRemote(m.remote.name)
       print('fatal: cannot obtain manifest %s' % r.url, file=sys.stderr)
@@ -327,7 +353,7 @@ to update the working directory files.
       return value
     return a
 
-  def _ShouldConfigureUser(self):
+  def _ShouldConfigureUser(self, opt):
     gc = self.manifest.globalConfig
     mp = self.manifest.manifestProject
 
@@ -339,21 +365,24 @@ to update the working directory files.
       mp.config.SetString('user.name', gc.GetString('user.name'))
       mp.config.SetString('user.email', gc.GetString('user.email'))
 
-    print()
-    print('Your identity is: %s <%s>' % (mp.config.GetString('user.name'),
-                                         mp.config.GetString('user.email')))
-    print('If you want to change this, please re-run \'repo init\' with --config-name')
+    if not opt.quiet:
+      print()
+      print('Your identity is: %s <%s>' % (mp.config.GetString('user.name'),
+                                           mp.config.GetString('user.email')))
+      print("If you want to change this, please re-run 'repo init' with --config-name")
     return False
 
-  def _ConfigureUser(self):
+  def _ConfigureUser(self, opt):
     mp = self.manifest.manifestProject
 
     while True:
-      print()
+      if not opt.quiet:
+        print()
       name = self._Prompt('Your Name', mp.UserName)
       email = self._Prompt('Your Email', mp.UserEmail)
 
-      print()
+      if not opt.quiet:
+        print()
       print('Your identity is: %s <%s>' % (name, email))
       print('is this correct [y/N]? ', end='')
       # TODO: When we require Python 3, use flush=True w/print above.
@@ -425,15 +454,16 @@ to update the working directory files.
       # We store the depth in the main manifest project.
       self.manifest.manifestProject.config.SetString('repo.depth', depth)
 
-  def _DisplayResult(self):
+  def _DisplayResult(self, opt):
     if self.manifest.IsMirror:
       init_type = 'mirror '
     else:
       init_type = ''
 
-    print()
-    print('repo %shas been initialized in %s'
-          % (init_type, self.manifest.topdir))
+    if not opt.quiet:
+      print()
+      print('repo %shas been initialized in %s' %
+            (init_type, self.manifest.topdir))
 
     current_dir = os.getcwd()
     if current_dir != self.manifest.topdir:
@@ -459,12 +489,37 @@ to update the working directory files.
             % ('.'.join(str(x) for x in MIN_GIT_VERSION_SOFT),),
             file=sys.stderr)
 
+    opt.quiet = opt.output_mode is False
+    opt.verbose = opt.output_mode is True
+
+    rp = self.manifest.repoProject
+
+    # Handle new --repo-url requests.
+    if opt.repo_url:
+      remote = rp.GetRemote('origin')
+      remote.url = opt.repo_url
+      remote.Save()
+
+    # Handle new --repo-rev requests.
+    if opt.repo_rev:
+      wrapper = Wrapper()
+      remote_ref, rev = wrapper.check_repo_rev(
+          rp.gitdir, opt.repo_rev, repo_verify=opt.repo_verify, quiet=opt.quiet)
+      branch = rp.GetBranch('default')
+      branch.merge = remote_ref
+      rp.work_git.update_ref('refs/heads/default', rev)
+      branch.Save()
+
+    if opt.worktree:
+      # Older versions of git supported worktree, but had dangerous gc bugs.
+      git_require((2, 15, 0), fail=True, msg='git gc worktree corruption')
+
     self._SyncManifest(opt)
     self._LinkManifest(opt.manifest_name)
 
     if os.isatty(0) and os.isatty(1) and not self.manifest.IsMirror:
-      if opt.config_name or self._ShouldConfigureUser():
-        self._ConfigureUser()
+      if opt.config_name or self._ShouldConfigureUser(opt):
+        self._ConfigureUser(opt)
       self._ConfigureColor()
 
-    self._DisplayResult()
+    self._DisplayResult(opt)

subcmds/manifest.py

@@ -25,7 +25,7 @@ class Manifest(PagedCommand):
   common = False
   helpSummary = "Manifest inspection utility"
   helpUsage = """
-%prog [-o {-|NAME.xml} [-r]]
+%prog [-o {-|NAME.xml}] [-m MANIFEST.xml] [-r]
 """
   _helpDescription = """
 
@@ -50,6 +50,8 @@ in a Git repository for use during future 'repo init' invocations.
     p.add_option('-r', '--revision-as-HEAD',
                  dest='peg_rev', action='store_true',
                  help='Save revisions as current HEAD')
+    p.add_option('-m', '--manifest-name',
+                 help='temporary manifest to use for this sync', metavar='NAME.xml')
     p.add_option('--suppress-upstream-revision', dest='peg_rev_upstream',
                  default=True, action='store_false',
                  help='If in -r mode, do not write the upstream field.  '
@@ -62,6 +64,10 @@ in a Git repository for use during future 'repo init' invocations.
                  metavar='-|NAME.xml')
 
   def _Output(self, opt):
+    # If alternate manifest is specified, override the manifest file that we're using.
+    if opt.manifest_name:
+      self.manifest.Override(opt.manifest_name, False)
+
     if opt.output_file == '-':
       fd = sys.stdout
     else:

subcmds/rebase.py

@@ -53,7 +53,7 @@ branch but need to incorporate new upstream changes "underneath" them.
                  dest='force_rebase', action='store_true',
                  help='Pass --force-rebase to git rebase')
     p.add_option('--no-ff',
-                 dest='no_ff', action='store_true',
+                 dest='ff', default=True, action='store_false',
                  help='Pass --no-ff to git rebase')
     p.add_option('-q', '--quiet',
                  dest='quiet', action='store_true',
@@ -93,7 +93,7 @@ branch but need to incorporate new upstream changes "underneath" them.
       common_args.append('--quiet')
     if opt.force_rebase:
       common_args.append('--force-rebase')
-    if opt.no_ff:
+    if not opt.ff:
       common_args.append('--no-ff')
     if opt.autosquash:
       common_args.append('--autosquash')

subcmds/selfupdate.py

@@ -40,7 +40,7 @@ need to be performed by an end-user.
   def _Options(self, p):
     g = p.add_option_group('repo Version options')
     g.add_option('--no-repo-verify',
-                 dest='no_repo_verify', action='store_true',
+                 dest='repo_verify', default=True, action='store_false',
                  help='do not verify repo source code')
     g.add_option('--repo-upgraded',
                  dest='repo_upgraded', action='store_true',
@@ -60,5 +60,5 @@ need to be performed by an end-user.
 
       rp.bare_git.gc('--auto')
       _PostRepoFetch(rp,
-                     no_repo_verify=opt.no_repo_verify,
+                     repo_verify=opt.repo_verify,
                      verbose=True)

subcmds/status.py

@@ -16,6 +16,10 @@
 
 from __future__ import print_function
 
+import glob
+import itertools
+import os
+
 from command import PagedCommand
 
 try:
@@ -23,11 +27,6 @@ try:
 except ImportError:
   import dummy_threading as _threading
 
-import glob
-
-import itertools
-import os
-
 from color import Coloring
 import platform_utils
 

subcmds/sync.py

@@ -15,6 +15,7 @@
 # limitations under the License.
 
 from __future__ import print_function
+
 import json
 import netrc
 from optparse import SUPPRESS_HELP
@@ -234,9 +235,12 @@ later is required to fix a server side protocol bug.
     p.add_option('-c', '--current-branch',
                  dest='current_branch_only', action='store_true',
                  help='fetch only current branch from server')
+    p.add_option('-v', '--verbose',
+                 dest='output_mode', action='store_true',
+                 help='show all sync output')
     p.add_option('-q', '--quiet',
-                 dest='quiet', action='store_true',
-                 help='be more quiet')
+                 dest='output_mode', action='store_false',
+                 help='only show errors')
     p.add_option('-j', '--jobs',
                  dest='jobs', action='store', type='int',
                  help="projects to fetch simultaneously (default %d)" % self.jobs)
@@ -244,7 +248,7 @@ later is required to fix a server side protocol bug.
                  dest='manifest_name',
                  help='temporary manifest to use for this sync', metavar='NAME.xml')
     p.add_option('--no-clone-bundle',
-                 dest='no_clone_bundle', action='store_true',
+                 dest='clone_bundle', default=True, action='store_false',
                  help='disable use of /clone.bundle on HTTP/HTTPS')
     p.add_option('-u', '--manifest-server-username', action='store',
                  dest='manifest_server_username',
@@ -256,7 +260,7 @@ later is required to fix a server side protocol bug.
                  dest='fetch_submodules', action='store_true',
                  help='fetch submodules from server')
     p.add_option('--no-tags',
-                 dest='no_tags', action='store_true',
+                 dest='tags', default=True, action='store_false',
                  help="don't fetch tags")
     p.add_option('--optimized-fetch',
                  dest='optimized_fetch', action='store_true',
@@ -273,7 +277,7 @@ later is required to fix a server side protocol bug.
 
     g = p.add_option_group('repo Version options')
     g.add_option('--no-repo-verify',
-                 dest='no_repo_verify', action='store_true',
+                 dest='repo_verify', default=True, action='store_false',
                  help='do not verify repo source code')
     g.add_option('--repo-upgraded',
                  dest='repo_upgraded', action='store_true',
@@ -332,10 +336,11 @@ later is required to fix a server side protocol bug.
       try:
         success = project.Sync_NetworkHalf(
             quiet=opt.quiet,
+            verbose=opt.verbose,
             current_branch_only=opt.current_branch_only,
             force_sync=opt.force_sync,
-            clone_bundle=not opt.no_clone_bundle,
-            no_tags=opt.no_tags, archive=self.manifest.IsArchive,
+            clone_bundle=opt.clone_bundle,
+            tags=opt.tags, archive=self.manifest.IsArchive,
             optimized_fetch=opt.optimized_fetch,
             prune=opt.prune,
             clone_filter=clone_filter)
@@ -565,12 +570,12 @@ later is required to fix a server side protocol bug.
     gc_gitdirs = {}
     for project in projects:
       # Make sure pruning never kicks in with shared projects.
-      if len(project.manifest.GetProjectsWithName(project.name)) > 1:
+      if (not project.use_git_worktrees and
+              len(project.manifest.GetProjectsWithName(project.name)) > 1):
         print('%s: Shared project %s found, disabling pruning.' %
               (project.relpath, project.name))
         if git_require((2, 7, 0)):
-          project.config.SetString('core.repositoryFormatVersion', '1')
-          project.config.SetString('extensions.preciousObjects', 'true')
+          project.EnableRepositoryExtension('preciousObjects')
         else:
           # This isn't perfect, but it's the best we can do with old git.
           print('%s: WARNING: shared projects are unreliable when using old '
@@ -627,65 +632,6 @@ later is required to fix a server side protocol bug.
     else:
       self.manifest._Unload()
 
-  def _DeleteProject(self, path):
-    print('Deleting obsolete path %s' % path, file=sys.stderr)
-
-    # Delete the .git directory first, so we're less likely to have a partially
-    # working git repository around. There shouldn't be any git projects here,
-    # so rmtree works.
-    try:
-      platform_utils.rmtree(os.path.join(path, '.git'))
-    except OSError as e:
-      print('Failed to remove %s (%s)' % (os.path.join(path, '.git'), str(e)), file=sys.stderr)
-      print('error: Failed to delete obsolete path %s' % path, file=sys.stderr)
-      print('       remove manually, then run sync again', file=sys.stderr)
-      return 1
-
-    # Delete everything under the worktree, except for directories that contain
-    # another git project
-    dirs_to_remove = []
-    failed = False
-    for root, dirs, files in platform_utils.walk(path):
-      for f in files:
-        try:
-          platform_utils.remove(os.path.join(root, f))
-        except OSError as e:
-          print('Failed to remove %s (%s)' % (os.path.join(root, f), str(e)), file=sys.stderr)
-          failed = True
-      dirs[:] = [d for d in dirs
-                 if not os.path.lexists(os.path.join(root, d, '.git'))]
-      dirs_to_remove += [os.path.join(root, d) for d in dirs
-                         if os.path.join(root, d) not in dirs_to_remove]
-    for d in reversed(dirs_to_remove):
-      if platform_utils.islink(d):
-        try:
-          platform_utils.remove(d)
-        except OSError as e:
-          print('Failed to remove %s (%s)' % (os.path.join(root, d), str(e)), file=sys.stderr)
-          failed = True
-      elif len(platform_utils.listdir(d)) == 0:
-        try:
-          platform_utils.rmdir(d)
-        except OSError as e:
-          print('Failed to remove %s (%s)' % (os.path.join(root, d), str(e)), file=sys.stderr)
-          failed = True
-          continue
-    if failed:
-      print('error: Failed to delete obsolete path %s' % path, file=sys.stderr)
-      print('       remove manually, then run sync again', file=sys.stderr)
-      return 1
-
-    # Try deleting parent dirs if they are empty
-    project_dir = path
-    while project_dir != self.manifest.topdir:
-      if len(platform_utils.listdir(project_dir)) == 0:
-        platform_utils.rmdir(project_dir)
-      else:
-        break
-      project_dir = os.path.dirname(project_dir)
-
-    return 0
-
   def UpdateProjectList(self, opt):
     new_project_paths = []
     for project in self.GetProjects(None, missing_ok=True):
@@ -712,23 +658,15 @@ later is required to fix a server side protocol bug.
                 remote=RemoteSpec('origin'),
                 gitdir=gitdir,
                 objdir=gitdir,
+                use_git_worktrees=os.path.isfile(gitdir),
                 worktree=os.path.join(self.manifest.topdir, path),
                 relpath=path,
                 revisionExpr='HEAD',
                 revisionId=None,
                 groups=None)
-
-            if project.IsDirty() and opt.force_remove_dirty:
-              print('WARNING: Removing dirty project "%s": uncommitted changes '
-                    'erased' % project.relpath, file=sys.stderr)
-              self._DeleteProject(project.worktree)
-            elif project.IsDirty():
-              print('error: Cannot remove project "%s": uncommitted changes '
-                    'are present' % project.relpath, file=sys.stderr)
-              print('       commit changes, then run sync again',
-                    file=sys.stderr)
-              return 1
-            elif self._DeleteProject(project.worktree):
+            if not project.DeleteWorktree(
+                    quiet=opt.quiet,
+                    force=opt.force_remove_dirty):
               return 1
 
     new_project_paths.sort()
@@ -790,13 +728,13 @@ later is required to fix a server side protocol bug.
         if branch.startswith(R_HEADS):
           branch = branch[len(R_HEADS):]
 
-        env = os.environ.copy()
-        if 'SYNC_TARGET' in env:
-          target = env['SYNC_TARGET']
+        if 'SYNC_TARGET' in os.environ:
+          target = os.environ['SYNC_TARGET']
           [success, manifest_str] = server.GetApprovedManifest(branch, target)
-        elif 'TARGET_PRODUCT' in env and 'TARGET_BUILD_VARIANT' in env:
-          target = '%s-%s' % (env['TARGET_PRODUCT'],
-                              env['TARGET_BUILD_VARIANT'])
+        elif ('TARGET_PRODUCT' in os.environ and
+              'TARGET_BUILD_VARIANT' in os.environ):
+          target = '%s-%s' % (os.environ['TARGET_PRODUCT'],
+                              os.environ['TARGET_BUILD_VARIANT'])
           [success, manifest_str] = server.GetApprovedManifest(branch, target)
         else:
           [success, manifest_str] = server.GetApprovedManifest(branch)
@@ -835,9 +773,9 @@ later is required to fix a server side protocol bug.
     """Fetch & update the local manifest project."""
     if not opt.local_only:
       start = time.time()
-      success = mp.Sync_NetworkHalf(quiet=opt.quiet,
+      success = mp.Sync_NetworkHalf(quiet=opt.quiet, verbose=opt.verbose,
                                     current_branch_only=opt.current_branch_only,
-                                    no_tags=opt.no_tags,
+                                    tags=opt.tags,
                                     optimized_fetch=opt.optimized_fetch,
                                     submodules=self.manifest.HasSubmodules,
                                     clone_filter=self.manifest.CloneFilter)
@@ -883,6 +821,9 @@ later is required to fix a server side protocol bug.
       soft_limit, _ = _rlimit_nofile()
       self.jobs = min(self.jobs, (soft_limit - 5) // 3)
 
+    opt.quiet = opt.output_mode is False
+    opt.verbose = opt.output_mode is True
+
     if opt.manifest_name:
       self.manifest.Override(opt.manifest_name)
 
@@ -904,6 +845,13 @@ later is required to fix a server side protocol bug.
 
     rp = self.manifest.repoProject
     rp.PreSync()
+    cb = rp.CurrentBranch
+    if cb:
+      base = rp.GetBranch(cb).merge
+      if not base or not base.startswith('refs/heads/'):
+        print('warning: repo is not tracking a remote branch, so it will not '
+              'receive updates; run `repo init --repo-rev=stable` to fix.',
+              file=sys.stderr)
 
     mp = self.manifest.manifestProject
     mp.PreSync()
@@ -970,7 +918,7 @@ later is required to fix a server side protocol bug.
 
       fetched = self._Fetch(to_fetch, opt, err_event)
 
-      _PostRepoFetch(rp, opt.no_repo_verify)
+      _PostRepoFetch(rp, opt.repo_verify)
       if opt.network_only:
         # bail out now; the rest touches the working tree
         if err_event.isSet():
@@ -1060,11 +1008,11 @@ def _PostRepoUpgrade(manifest, quiet=False):
       project.PostRepoUpgrade()
 
 
-def _PostRepoFetch(rp, no_repo_verify=False, verbose=False):
+def _PostRepoFetch(rp, repo_verify=True, verbose=False):
   if rp.HasChanges:
     print('info: A new version of repo is available', file=sys.stderr)
     print(file=sys.stderr)
-    if no_repo_verify or _VerifyTag(rp):
+    if not repo_verify or _VerifyTag(rp):
       syncbuf = SyncBuffer(rp.config)
       rp.Sync_LocalHalf(syncbuf)
       if not syncbuf.Finish():
@@ -1104,8 +1052,8 @@ def _VerifyTag(project):
     return False
 
   env = os.environ.copy()
-  env['GIT_DIR'] = project.gitdir.encode()
-  env['GNUPGHOME'] = gpg_dir.encode()
+  env['GIT_DIR'] = project.gitdir
+  env['GNUPGHOME'] = gpg_dir
 
   cmd = [GIT, 'tag', '-v', cur]
   proc = subprocess.Popen(cmd,
@@ -1195,7 +1143,7 @@ class PersistentTransport(xmlrpc.client.Transport):
       # Since we're only using them for HTTP, copy the file temporarily,
       # stripping those prefixes away.
       if cookiefile:
-        tmpcookiefile = tempfile.NamedTemporaryFile()
+        tmpcookiefile = tempfile.NamedTemporaryFile(mode='w')
         tmpcookiefile.write("# HTTP Cookie File")
         try:
           with open(cookiefile) as f:

subcmds/upload.py

@@ -27,7 +27,7 @@ from project import RepoHook
 
 from pyversion import is_python3
 if not is_python3():
-  input = raw_input
+  input = raw_input  # noqa: F821
 else:
   unicode = str
 
@@ -130,6 +130,23 @@ is set to "true" then repo will assume you always want the equivalent
 of the -t option to the repo command. If unset or set to "false" then
 repo will make use of only the command line option.
 
+review.URL.uploadhashtags:
+
+To add hashtags whenever uploading a commit, you can set a per-project
+or global Git option to do so. The value of review.URL.uploadhashtags
+will be used as comma delimited hashtags like the --hashtag option.
+
+review.URL.uploadlabels:
+
+To add labels whenever uploading a commit, you can set a per-project
+or global Git option to do so. The value of review.URL.uploadlabels
+will be used as comma delimited labels like the --label option.
+
+review.URL.uploadnotify:
+
+Control e-mail notifications when uploading.
+https://gerrit-review.googlesource.com/Documentation/user-upload.html#notify
+
 # References
 
 Gerrit Code Review:  https://www.gerritcodereview.com/
@@ -140,6 +157,15 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
     p.add_option('-t',
                  dest='auto_topic', action='store_true',
                  help='Send local branch name to Gerrit Code Review')
+    p.add_option('--hashtag', '--ht',
+                 dest='hashtags', action='append', default=[],
+                 help='Add hashtags (comma delimited) to the review.')
+    p.add_option('--hashtag-branch', '--htb',
+                 action='store_true',
+                 help='Add local branch name as a hashtag.')
+    p.add_option('-l', '--label',
+                 dest='labels', action='append', default=[],
+                 help='Add a label when uploading.')
     p.add_option('--re', '--reviewers',
                  type='string', action='append', dest='reviewers',
                  help='Request reviews from these people.')
@@ -152,9 +178,6 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
     p.add_option('--cbr', '--current-branch',
                  dest='current_branch', action='store_true',
                  help='Upload current git branch.')
-    p.add_option('-d', '--draft',
-                 action='store_true', dest='draft', default=False,
-                 help='If specified, upload as a draft.')
     p.add_option('--ne', '--no-emails',
                  action='store_false', dest='notify', default=True,
                  help='If specified, do not send emails on upload.')
@@ -172,6 +195,12 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
                  type='string', action='store', dest='dest_branch',
                  metavar='BRANCH',
                  help='Submit for review on this target branch.')
+    p.add_option('-n', '--dry-run',
+                 dest='dryrun', default=False, action='store_true',
+                 help='Do everything except actually upload the CL.')
+    p.add_option('-y', '--yes',
+                 default=False, action='store_true',
+                 help='Answer yes to all safe prompts.')
     p.add_option('--no-cert-checks',
                  dest='validate_certs', action='store_false', default=True,
                  help='Disable verifying ssl certs (unsafe).')
@@ -220,7 +249,7 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
 
       destination = opt.dest_branch or project.dest_branch or project.revisionExpr
       print('Upload project %s/ to remote branch %s%s:' %
-            (project.relpath, destination, ' (draft)' if opt.draft else ''))
+            (project.relpath, destination, ' (private)' if opt.private else ''))
       print('  branch %s (%2d commit%s, %s):' % (
           name,
           len(commit_list),
@@ -232,8 +261,12 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
       print('to %s (y/N)? ' % remote.review, end='')
       # TODO: When we require Python 3, use flush=True w/print above.
       sys.stdout.flush()
-      answer = sys.stdin.readline().strip().lower()
-      answer = answer in ('y', 'yes', '1', 'true', 't')
+      if opt.yes:
+        print('<--yes>')
+        answer = True
+      else:
+        answer = sys.stdin.readline().strip().lower()
+        answer = answer in ('y', 'yes', '1', 'true', 't')
 
     if answer:
       if len(branch.commits) > UNUSUAL_COMMIT_THRESHOLD:
@@ -372,7 +405,11 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
             print('Continue uploading? (y/N) ', end='')
             # TODO: When we require Python 3, use flush=True w/print above.
             sys.stdout.flush()
-            a = sys.stdin.readline().strip().lower()
+            if opt.yes:
+              print('<--yes>')
+              a = 'yes'
+            else:
+              a = sys.stdin.readline().strip().lower()
             if a not in ('y', 'yes', 't', 'true', 'on'):
               print("skipping upload", file=sys.stderr)
               branch.uploaded = False
@@ -384,6 +421,42 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
           key = 'review.%s.uploadtopic' % branch.project.remote.review
           opt.auto_topic = branch.project.config.GetBoolean(key)
 
+        def _ExpandCommaList(value):
+          """Split |value| up into comma delimited entries."""
+          if not value:
+            return
+          for ret in value.split(','):
+            ret = ret.strip()
+            if ret:
+              yield ret
+
+        # Check if hashtags should be included.
+        key = 'review.%s.uploadhashtags' % branch.project.remote.review
+        hashtags = set(_ExpandCommaList(branch.project.config.GetString(key)))
+        for tag in opt.hashtags:
+          hashtags.update(_ExpandCommaList(tag))
+        if opt.hashtag_branch:
+          hashtags.add(branch.name)
+
+        # Check if labels should be included.
+        key = 'review.%s.uploadlabels' % branch.project.remote.review
+        labels = set(_ExpandCommaList(branch.project.config.GetString(key)))
+        for label in opt.labels:
+          labels.update(_ExpandCommaList(label))
+        # Basic sanity check on label syntax.
+        for label in labels:
+          if not re.match(r'^.+[+-][0-9]+$', label):
+            print('repo: error: invalid label syntax "%s": labels use forms '
+                  'like CodeReview+1 or Verified-1' % (label,), file=sys.stderr)
+            sys.exit(1)
+
+        # Handle e-mail notifications.
+        if opt.notify is False:
+          notify = 'NONE'
+        else:
+          key = 'review.%s.uploadnotify' % branch.project.remote.review
+          notify = branch.project.config.GetString(key)
+
         destination = opt.dest_branch or branch.project.dest_branch
 
         # Make sure our local branch is not setup to track a different remote branch
@@ -400,10 +473,12 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
             continue
 
         branch.UploadForReview(people,
+                               dryrun=opt.dryrun,
                                auto_topic=opt.auto_topic,
-                               draft=opt.draft,
+                               hashtags=hashtags,
+                               labels=labels,
                                private=opt.private,
-                               notify=None if opt.notify else 'NONE',
+                               notify=notify,
                                wip=opt.wip,
                                dest_branch=destination,
                                validate_certs=opt.validate_certs,
@@ -486,8 +561,12 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
         pending.append((project, avail))
 
     if not pending:
-      print("no branches ready for upload", file=sys.stderr)
-      return
+      if branch is None:
+        print('repo: error: no branches ready for upload', file=sys.stderr)
+      else:
+        print('repo: error: no branches named "%s" ready for upload' %
+              (branch,), file=sys.stderr)
+      return 1
 
     if not opt.bypass_hooks:
       hook = RepoHook('pre-upload', self.manifest.repo_hooks_project,

subcmds/version.py

@@ -15,7 +15,10 @@
 # limitations under the License.
 
 from __future__ import print_function
+
+import platform
 import sys
+
 from command import Command, MirrorSafeCommand
 from git_command import git, RepoSourceVersion, user_agent
 from git_refs import HEAD
@@ -41,9 +44,9 @@ class Version(Command, MirrorSafeCommand):
     print('repo version %s' % rp_ver)
     print('       (from %s)' % rem.url)
 
-    if Version.wrapper_path is not None:
-      print('repo launcher version %s' % Version.wrapper_version)
-      print('       (from %s)' % Version.wrapper_path)
+    if self.wrapper_path is not None:
+      print('repo launcher version %s' % self.wrapper_version)
+      print('       (from %s)' % self.wrapper_path)
 
       if src_ver != rp_ver:
         print('       (currently at %s)' % src_ver)
@@ -52,3 +55,11 @@ class Version(Command, MirrorSafeCommand):
     print('git %s' % git.version_tuple().full)
     print('git User-Agent %s' % user_agent.git)
     print('Python %s' % sys.version)
+    uname = platform.uname()
+    if sys.version_info.major < 3:
+      # Python 3 returns a named tuple, but Python 2 is simpler.
+      print(uname)
+    else:
+      print('OS %s %s (%s)' % (uname.system, uname.release, uname.version))
+      print('CPU %s (%s)' %
+            (uname.machine, uname.processor if uname.processor else 'unknown'))

tests/fixtures/test.gitconfig

@@ -1,3 +1,13 @@
 [section]
 	empty
 	nonempty = true
+	boolinvalid = oops
+	booltrue = true
+	boolfalse = false
+	intinvalid = oops
+	inthex = 0x10
+	inthexk = 0x10k
+	int = 10
+	intk = 10k
+	intm = 10m
+	intg = 10g

tests/test_git_command.py

@@ -21,7 +21,13 @@ from __future__ import print_function
 import re
 import unittest
 
+try:
+  from unittest import mock
+except ImportError:
+  import mock
+
 import git_command
+import wrapper
 
 
 class GitCallUnitTest(unittest.TestCase):
@@ -76,3 +82,45 @@ class UserAgentUnitTest(unittest.TestCase):
     # the general form.
     m = re.match(r'^git/[^ ]+ ([^ ]+) git-repo/[^ ]+', ua)
     self.assertIsNotNone(m)
+
+
+class GitRequireTests(unittest.TestCase):
+  """Test the git_require helper."""
+
+  def setUp(self):
+    ver = wrapper.GitVersion(1, 2, 3, 4)
+    mock.patch.object(git_command.git, 'version_tuple', return_value=ver).start()
+
+  def tearDown(self):
+    mock.patch.stopall()
+
+  def test_older_nonfatal(self):
+    """Test non-fatal require calls with old versions."""
+    self.assertFalse(git_command.git_require((2,)))
+    self.assertFalse(git_command.git_require((1, 3)))
+    self.assertFalse(git_command.git_require((1, 2, 4)))
+    self.assertFalse(git_command.git_require((1, 2, 3, 5)))
+
+  def test_newer_nonfatal(self):
+    """Test non-fatal require calls with newer versions."""
+    self.assertTrue(git_command.git_require((0,)))
+    self.assertTrue(git_command.git_require((1, 0)))
+    self.assertTrue(git_command.git_require((1, 2, 0)))
+    self.assertTrue(git_command.git_require((1, 2, 3, 0)))
+
+  def test_equal_nonfatal(self):
+    """Test require calls with equal values."""
+    self.assertTrue(git_command.git_require((1, 2, 3, 4), fail=False))
+    self.assertTrue(git_command.git_require((1, 2, 3, 4), fail=True))
+
+  def test_older_fatal(self):
+    """Test fatal require calls with old versions."""
+    with self.assertRaises(SystemExit) as e:
+      git_command.git_require((2,), fail=True)
+      self.assertNotEqual(0, e.code)
+
+  def test_older_fatal_msg(self):
+    """Test fatal require calls with old versions and message."""
+    with self.assertRaises(SystemExit) as e:
+      git_command.git_require((2,), fail=True, msg='so sad')
+      self.assertNotEqual(0, e.code)

tests/test_git_config.py

@@ -71,6 +71,43 @@ class GitConfigUnitTest(unittest.TestCase):
     val = config.GetString('empty')
     self.assertEqual(val, None)
 
+  def test_GetBoolean_undefined(self):
+    """Test GetBoolean on key that doesn't exist."""
+    self.assertIsNone(self.config.GetBoolean('section.missing'))
+
+  def test_GetBoolean_invalid(self):
+    """Test GetBoolean on invalid boolean value."""
+    self.assertIsNone(self.config.GetBoolean('section.boolinvalid'))
+
+  def test_GetBoolean_true(self):
+    """Test GetBoolean on valid true boolean."""
+    self.assertTrue(self.config.GetBoolean('section.booltrue'))
+
+  def test_GetBoolean_false(self):
+    """Test GetBoolean on valid false boolean."""
+    self.assertFalse(self.config.GetBoolean('section.boolfalse'))
+
+  def test_GetInt_undefined(self):
+    """Test GetInt on key that doesn't exist."""
+    self.assertIsNone(self.config.GetInt('section.missing'))
+
+  def test_GetInt_invalid(self):
+    """Test GetInt on invalid integer value."""
+    self.assertIsNone(self.config.GetBoolean('section.intinvalid'))
+
+  def test_GetInt_valid(self):
+    """Test GetInt on valid integers."""
+    TESTS = (
+        ('inthex', 16),
+        ('inthexk', 16384),
+        ('int', 10),
+        ('intk', 10240),
+        ('intm', 10485760),
+        ('intg', 10737418240),
+    )
+    for key, value in TESTS:
+      self.assertEqual(value, self.config.GetInt('section.%s' % (key,)))
+
 
 if __name__ == '__main__':
   unittest.main()

tests/test_manifest_xml.py

@@ -18,7 +18,9 @@
 
 from __future__ import print_function
 
+import os
 import unittest
+import xml.dom.minidom
 
 import error
 import manifest_xml
@@ -78,8 +80,69 @@ class ManifestValidateFilePaths(unittest.TestCase):
         # Block Unicode characters that get normalized out by filesystems.
         u'foo\u200Cbar',
     )
+    # Make sure platforms that use path separators (e.g. Windows) are also
+    # rejected properly.
+    if os.path.sep != '/':
+      PATHS += tuple(x.replace('/', os.path.sep) for x in PATHS)
+
     for path in PATHS:
       self.assertRaises(
           error.ManifestInvalidPathError, self.check_both, path, 'a')
       self.assertRaises(
           error.ManifestInvalidPathError, self.check_both, 'a', path)
+
+
+class ValueTests(unittest.TestCase):
+  """Check utility parsing code."""
+
+  def _get_node(self, text):
+    return xml.dom.minidom.parseString(text).firstChild
+
+  def test_bool_default(self):
+    """Check XmlBool default handling."""
+    node = self._get_node('<node/>')
+    self.assertIsNone(manifest_xml.XmlBool(node, 'a'))
+    self.assertIsNone(manifest_xml.XmlBool(node, 'a', None))
+    self.assertEqual(123, manifest_xml.XmlBool(node, 'a', 123))
+
+    node = self._get_node('<node a=""/>')
+    self.assertIsNone(manifest_xml.XmlBool(node, 'a'))
+
+  def test_bool_invalid(self):
+    """Check XmlBool invalid handling."""
+    node = self._get_node('<node a="moo"/>')
+    self.assertEqual(123, manifest_xml.XmlBool(node, 'a', 123))
+
+  def test_bool_true(self):
+    """Check XmlBool true values."""
+    for value in ('yes', 'true', '1'):
+      node = self._get_node('<node a="%s"/>' % (value,))
+      self.assertTrue(manifest_xml.XmlBool(node, 'a'))
+
+  def test_bool_false(self):
+    """Check XmlBool false values."""
+    for value in ('no', 'false', '0'):
+      node = self._get_node('<node a="%s"/>' % (value,))
+      self.assertFalse(manifest_xml.XmlBool(node, 'a'))
+
+  def test_int_default(self):
+    """Check XmlInt default handling."""
+    node = self._get_node('<node/>')
+    self.assertIsNone(manifest_xml.XmlInt(node, 'a'))
+    self.assertIsNone(manifest_xml.XmlInt(node, 'a', None))
+    self.assertEqual(123, manifest_xml.XmlInt(node, 'a', 123))
+
+    node = self._get_node('<node a=""/>')
+    self.assertIsNone(manifest_xml.XmlInt(node, 'a'))
+
+  def test_int_good(self):
+    """Check XmlInt numeric handling."""
+    for value in (-1, 0, 1, 50000):
+      node = self._get_node('<node a="%s"/>' % (value,))
+      self.assertEqual(value, manifest_xml.XmlInt(node, 'a'))
+
+  def test_int_invalid(self):
+    """Check XmlInt invalid handling."""
+    with self.assertRaises(error.ManifestParseError):
+      node = self._get_node('<node a="xx"/>')
+      manifest_xml.XmlInt(node, 'a')

tests/test_project.py

@@ -27,6 +27,7 @@ import unittest
 
 import error
 import git_config
+import platform_utils
 import project
 
 
@@ -40,7 +41,7 @@ def TempGitTree():
     subprocess.check_call(['git', 'init'], cwd=tempdir)
     yield tempdir
   finally:
-    shutil.rmtree(tempdir)
+    platform_utils.rmtree(tempdir)
 
 
 class RepoHookShebang(unittest.TestCase):
@@ -243,17 +244,19 @@ class CopyFile(CopyLinkTestCase):
     src = os.path.join(self.worktree, 'foo.txt')
     sym = os.path.join(self.worktree, 'sym')
     self.touch(src)
-    os.symlink('foo.txt', sym)
+    platform_utils.symlink('foo.txt', sym)
     self.assertExists(sym)
     cf = self.CopyFile('sym', 'foo')
     self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
 
   def test_src_block_symlink_traversal(self):
     """Do not allow reading through a symlink dir."""
-    src = os.path.join(self.worktree, 'bar', 'passwd')
-    os.symlink('/etc', os.path.join(self.worktree, 'bar'))
+    realfile = os.path.join(self.tempdir, 'file.txt')
+    self.touch(realfile)
+    src = os.path.join(self.worktree, 'bar', 'file.txt')
+    platform_utils.symlink(self.tempdir, os.path.join(self.worktree, 'bar'))
     self.assertExists(src)
-    cf = self.CopyFile('bar/foo.txt', 'foo')
+    cf = self.CopyFile('bar/file.txt', 'foo')
     self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
 
   def test_src_block_copy_from_dir(self):
@@ -267,7 +270,7 @@ class CopyFile(CopyLinkTestCase):
     """Do not allow writing to a symlink."""
     src = os.path.join(self.worktree, 'foo.txt')
     self.touch(src)
-    os.symlink('dest', os.path.join(self.topdir, 'sym'))
+    platform_utils.symlink('dest', os.path.join(self.topdir, 'sym'))
     cf = self.CopyFile('foo.txt', 'sym')
     self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
 
@@ -275,7 +278,8 @@ class CopyFile(CopyLinkTestCase):
     """Do not allow writing through a symlink dir."""
     src = os.path.join(self.worktree, 'foo.txt')
     self.touch(src)
-    os.symlink('/tmp', os.path.join(self.topdir, 'sym'))
+    platform_utils.symlink(tempfile.gettempdir(),
+                           os.path.join(self.topdir, 'sym'))
     cf = self.CopyFile('foo.txt', 'sym/foo.txt')
     self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
 
@@ -303,7 +307,7 @@ class LinkFile(CopyLinkTestCase):
     dest = os.path.join(self.topdir, 'foo')
     self.assertExists(dest)
     self.assertTrue(os.path.islink(dest))
-    self.assertEqual('git-project/foo.txt', os.readlink(dest))
+    self.assertEqual(os.path.join('git-project', 'foo.txt'), os.readlink(dest))
 
   def test_src_subdir(self):
     """Link to a file in a subdir of a project."""
@@ -320,7 +324,7 @@ class LinkFile(CopyLinkTestCase):
     lf = self.LinkFile('.', 'foo/bar')
     lf._Link()
     self.assertExists(dest)
-    self.assertEqual('../git-project', os.readlink(dest))
+    self.assertEqual(os.path.join('..', 'git-project'), os.readlink(dest))
 
   def test_dest_subdir(self):
     """Link a file to a subdir of a checkout."""
@@ -354,10 +358,10 @@ class LinkFile(CopyLinkTestCase):
     self.touch(src)
     lf = self.LinkFile('foo.txt', 'sym')
     lf._Link()
-    self.assertEqual('git-project/foo.txt', os.readlink(dest))
+    self.assertEqual(os.path.join('git-project', 'foo.txt'), os.readlink(dest))
 
     # Point the symlink somewhere else.
     os.unlink(dest)
-    os.symlink('/', dest)
+    platform_utils.symlink(self.tempdir, dest)
     lf._Link()
-    self.assertEqual('git-project/foo.txt', os.readlink(dest))
+    self.assertEqual(os.path.join('git-project', 'foo.txt'), os.readlink(dest))

tests/test_wrapper.py

@@ -18,10 +18,14 @@
 
 from __future__ import print_function
 
+import contextlib
 import os
 import re
+import shutil
+import tempfile
 import unittest
 
+import platform_utils
 from pyversion import is_python3
 import wrapper
 
@@ -34,6 +38,18 @@ else:
   from StringIO import StringIO
 
 
+@contextlib.contextmanager
+def TemporaryDirectory():
+  """Create a new empty git checkout for testing."""
+  # TODO(vapier): Convert this to tempfile.TemporaryDirectory once we drop
+  # Python 2 support entirely.
+  try:
+    tempdir = tempfile.mkdtemp(prefix='repo-tests')
+    yield tempdir
+  finally:
+    platform_utils.rmtree(tempdir)
+
+
 def fixture(*paths):
   """Return a path relative to tests/fixtures.
   """
@@ -66,6 +82,20 @@ class RepoWrapperUnitTest(RepoWrapperTestCase):
     self.assertEqual('', stderr.getvalue())
     self.assertIn('repo launcher version', stdout.getvalue())
 
+  def test_init_parser(self):
+    """Make sure 'init' GetParser works."""
+    parser = self.wrapper.GetParser(gitc_init=False)
+    opts, args = parser.parse_args([])
+    self.assertEqual([], args)
+    self.assertIsNone(opts.manifest_url)
+
+  def test_gitc_init_parser(self):
+    """Make sure 'gitc-init' GetParser works."""
+    parser = self.wrapper.GetParser(gitc_init=True)
+    opts, args = parser.parse_args([])
+    self.assertEqual([], args)
+    self.assertIsNone(opts.manifest_file)
+
   def test_get_gitc_manifest_dir_no_gitc(self):
     """
     Test reading a missing gitc config file
@@ -101,7 +131,8 @@ class RepoWrapperUnitTest(RepoWrapperTestCase):
     self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/test/extra'), 'test')
     self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/test'), 'test')
     self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/test/'), 'test')
-    self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/test/extra'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/test/extra'),
+                     'test')
     self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/'), None)
     self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/'), None)
 
@@ -138,5 +169,316 @@ class SetGitTrace2ParentSid(RepoWrapperTestCase):
     self.assertRegex(value, self.VALID_FORMAT)
 
 
+class RunCommand(RepoWrapperTestCase):
+  """Check run_command behavior."""
+
+  def test_capture(self):
+    """Check capture_output handling."""
+    ret = self.wrapper.run_command(['echo', 'hi'], capture_output=True)
+    self.assertEqual(ret.stdout, 'hi\n')
+
+  def test_check(self):
+    """Check check handling."""
+    self.wrapper.run_command(['true'], check=False)
+    self.wrapper.run_command(['true'], check=True)
+    self.wrapper.run_command(['false'], check=False)
+    with self.assertRaises(self.wrapper.RunError):
+      self.wrapper.run_command(['false'], check=True)
+
+
+class RunGit(RepoWrapperTestCase):
+  """Check run_git behavior."""
+
+  def test_capture(self):
+    """Check capture_output handling."""
+    ret = self.wrapper.run_git('--version')
+    self.assertIn('git', ret.stdout)
+
+  def test_check(self):
+    """Check check handling."""
+    with self.assertRaises(self.wrapper.CloneFailure):
+      self.wrapper.run_git('--version-asdfasdf')
+    self.wrapper.run_git('--version-asdfasdf', check=False)
+
+
+class ParseGitVersion(RepoWrapperTestCase):
+  """Check ParseGitVersion behavior."""
+
+  def test_autoload(self):
+    """Check we can load the version from the live git."""
+    ret = self.wrapper.ParseGitVersion()
+    self.assertIsNotNone(ret)
+
+  def test_bad_ver(self):
+    """Check handling of bad git versions."""
+    ret = self.wrapper.ParseGitVersion(ver_str='asdf')
+    self.assertIsNone(ret)
+
+  def test_normal_ver(self):
+    """Check handling of normal git versions."""
+    ret = self.wrapper.ParseGitVersion(ver_str='git version 2.25.1')
+    self.assertEqual(2, ret.major)
+    self.assertEqual(25, ret.minor)
+    self.assertEqual(1, ret.micro)
+    self.assertEqual('2.25.1', ret.full)
+
+  def test_extended_ver(self):
+    """Check handling of extended distro git versions."""
+    ret = self.wrapper.ParseGitVersion(
+        ver_str='git version 1.30.50.696.g5e7596f4ac-goog')
+    self.assertEqual(1, ret.major)
+    self.assertEqual(30, ret.minor)
+    self.assertEqual(50, ret.micro)
+    self.assertEqual('1.30.50.696.g5e7596f4ac-goog', ret.full)
+
+
+class CheckGitVersion(RepoWrapperTestCase):
+  """Check _CheckGitVersion behavior."""
+
+  def test_unknown(self):
+    """Unknown versions should abort."""
+    with mock.patch.object(self.wrapper, 'ParseGitVersion', return_value=None):
+      with self.assertRaises(self.wrapper.CloneFailure):
+        self.wrapper._CheckGitVersion()
+
+  def test_old(self):
+    """Old versions should abort."""
+    with mock.patch.object(
+        self.wrapper, 'ParseGitVersion',
+        return_value=self.wrapper.GitVersion(1, 0, 0, '1.0.0')):
+      with self.assertRaises(self.wrapper.CloneFailure):
+        self.wrapper._CheckGitVersion()
+
+  def test_new(self):
+    """Newer versions should run fine."""
+    with mock.patch.object(
+        self.wrapper, 'ParseGitVersion',
+        return_value=self.wrapper.GitVersion(100, 0, 0, '100.0.0')):
+      self.wrapper._CheckGitVersion()
+
+
+class NeedSetupGnuPG(RepoWrapperTestCase):
+  """Check NeedSetupGnuPG behavior."""
+
+  def test_missing_dir(self):
+    """The ~/.repoconfig tree doesn't exist yet."""
+    with TemporaryDirectory() as tempdir:
+      self.wrapper.home_dot_repo = os.path.join(tempdir, 'foo')
+      self.assertTrue(self.wrapper.NeedSetupGnuPG())
+
+  def test_missing_keyring(self):
+    """The keyring-version file doesn't exist yet."""
+    with TemporaryDirectory() as tempdir:
+      self.wrapper.home_dot_repo = tempdir
+      self.assertTrue(self.wrapper.NeedSetupGnuPG())
+
+  def test_empty_keyring(self):
+    """The keyring-version file exists, but is empty."""
+    with TemporaryDirectory() as tempdir:
+      self.wrapper.home_dot_repo = tempdir
+      with open(os.path.join(tempdir, 'keyring-version'), 'w'):
+        pass
+      self.assertTrue(self.wrapper.NeedSetupGnuPG())
+
+  def test_old_keyring(self):
+    """The keyring-version file exists, but it's old."""
+    with TemporaryDirectory() as tempdir:
+      self.wrapper.home_dot_repo = tempdir
+      with open(os.path.join(tempdir, 'keyring-version'), 'w') as fp:
+        fp.write('1.0\n')
+      self.assertTrue(self.wrapper.NeedSetupGnuPG())
+
+  def test_new_keyring(self):
+    """The keyring-version file exists, and is up-to-date."""
+    with TemporaryDirectory() as tempdir:
+      self.wrapper.home_dot_repo = tempdir
+      with open(os.path.join(tempdir, 'keyring-version'), 'w') as fp:
+        fp.write('1000.0\n')
+      self.assertFalse(self.wrapper.NeedSetupGnuPG())
+
+
+class SetupGnuPG(RepoWrapperTestCase):
+  """Check SetupGnuPG behavior."""
+
+  def test_full(self):
+    """Make sure it works completely."""
+    with TemporaryDirectory() as tempdir:
+      self.wrapper.home_dot_repo = tempdir
+      self.assertTrue(self.wrapper.SetupGnuPG(True))
+      with open(os.path.join(tempdir, 'keyring-version'), 'r') as fp:
+        data = fp.read()
+      self.assertEqual('.'.join(str(x) for x in self.wrapper.KEYRING_VERSION),
+                       data.strip())
+
+
+class VerifyRev(RepoWrapperTestCase):
+  """Check verify_rev behavior."""
+
+  def test_verify_passes(self):
+    """Check when we have a valid signed tag."""
+    desc_result = self.wrapper.RunResult(0, 'v1.0\n', '')
+    gpg_result = self.wrapper.RunResult(0, '', '')
+    with mock.patch.object(self.wrapper, 'run_git',
+                           side_effect=(desc_result, gpg_result)):
+      ret = self.wrapper.verify_rev('/', 'refs/heads/stable', '1234', True)
+      self.assertEqual('v1.0^0', ret)
+
+  def test_unsigned_commit(self):
+    """Check we fall back to signed tag when we have an unsigned commit."""
+    desc_result = self.wrapper.RunResult(0, 'v1.0-10-g1234\n', '')
+    gpg_result = self.wrapper.RunResult(0, '', '')
+    with mock.patch.object(self.wrapper, 'run_git',
+                           side_effect=(desc_result, gpg_result)):
+      ret = self.wrapper.verify_rev('/', 'refs/heads/stable', '1234', True)
+      self.assertEqual('v1.0^0', ret)
+
+  def test_verify_fails(self):
+    """Check we fall back to signed tag when we have an unsigned commit."""
+    desc_result = self.wrapper.RunResult(0, 'v1.0-10-g1234\n', '')
+    gpg_result = Exception
+    with mock.patch.object(self.wrapper, 'run_git',
+                           side_effect=(desc_result, gpg_result)):
+      with self.assertRaises(Exception):
+        self.wrapper.verify_rev('/', 'refs/heads/stable', '1234', True)
+
+
+class GitCheckoutTestCase(RepoWrapperTestCase):
+  """Tests that use a real/small git checkout."""
+
+  GIT_DIR = None
+  REV_LIST = None
+
+  @classmethod
+  def setUpClass(cls):
+    # Create a repo to operate on, but do it once per-class.
+    cls.GIT_DIR = tempfile.mkdtemp(prefix='repo-rev-tests')
+    run_git = wrapper.Wrapper().run_git
+
+    remote = os.path.join(cls.GIT_DIR, 'remote')
+    os.mkdir(remote)
+    run_git('init', cwd=remote)
+    run_git('commit', '--allow-empty', '-minit', cwd=remote)
+    run_git('branch', 'stable', cwd=remote)
+    run_git('tag', 'v1.0', cwd=remote)
+    run_git('commit', '--allow-empty', '-m2nd commit', cwd=remote)
+    cls.REV_LIST = run_git('rev-list', 'HEAD', cwd=remote).stdout.splitlines()
+
+    run_git('init', cwd=cls.GIT_DIR)
+    run_git('fetch', remote, '+refs/heads/*:refs/remotes/origin/*', cwd=cls.GIT_DIR)
+
+  @classmethod
+  def tearDownClass(cls):
+    if not cls.GIT_DIR:
+      return
+
+    shutil.rmtree(cls.GIT_DIR)
+
+
+class ResolveRepoRev(GitCheckoutTestCase):
+  """Check resolve_repo_rev behavior."""
+
+  def test_explicit_branch(self):
+    """Check refs/heads/branch argument."""
+    rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, 'refs/heads/stable')
+    self.assertEqual('refs/heads/stable', rrev)
+    self.assertEqual(self.REV_LIST[1], lrev)
+
+    with self.assertRaises(wrapper.CloneFailure):
+      self.wrapper.resolve_repo_rev(self.GIT_DIR, 'refs/heads/unknown')
+
+  def test_explicit_tag(self):
+    """Check refs/tags/tag argument."""
+    rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, 'refs/tags/v1.0')
+    self.assertEqual('refs/tags/v1.0', rrev)
+    self.assertEqual(self.REV_LIST[1], lrev)
+
+    with self.assertRaises(wrapper.CloneFailure):
+      self.wrapper.resolve_repo_rev(self.GIT_DIR, 'refs/tags/unknown')
+
+  def test_branch_name(self):
+    """Check branch argument."""
+    rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, 'stable')
+    self.assertEqual('refs/heads/stable', rrev)
+    self.assertEqual(self.REV_LIST[1], lrev)
+
+    rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, 'master')
+    self.assertEqual('refs/heads/master', rrev)
+    self.assertEqual(self.REV_LIST[0], lrev)
+
+  def test_tag_name(self):
+    """Check tag argument."""
+    rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, 'v1.0')
+    self.assertEqual('refs/tags/v1.0', rrev)
+    self.assertEqual(self.REV_LIST[1], lrev)
+
+  def test_full_commit(self):
+    """Check specific commit argument."""
+    commit = self.REV_LIST[0]
+    rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, commit)
+    self.assertEqual(commit, rrev)
+    self.assertEqual(commit, lrev)
+
+  def test_partial_commit(self):
+    """Check specific (partial) commit argument."""
+    commit = self.REV_LIST[0][0:20]
+    rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, commit)
+    self.assertEqual(self.REV_LIST[0], rrev)
+    self.assertEqual(self.REV_LIST[0], lrev)
+
+  def test_unknown(self):
+    """Check unknown ref/commit argument."""
+    with self.assertRaises(wrapper.CloneFailure):
+      self.wrapper.resolve_repo_rev(self.GIT_DIR, 'boooooooya')
+
+
+class CheckRepoVerify(RepoWrapperTestCase):
+  """Check check_repo_verify behavior."""
+
+  def test_no_verify(self):
+    """Always fail with --no-repo-verify."""
+    self.assertFalse(self.wrapper.check_repo_verify(False))
+
+  def test_gpg_initialized(self):
+    """Should pass if gpg is setup already."""
+    with mock.patch.object(self.wrapper, 'NeedSetupGnuPG', return_value=False):
+      self.assertTrue(self.wrapper.check_repo_verify(True))
+
+  def test_need_gpg_setup(self):
+    """Should pass/fail based on gpg setup."""
+    with mock.patch.object(self.wrapper, 'NeedSetupGnuPG', return_value=True):
+      with mock.patch.object(self.wrapper, 'SetupGnuPG') as m:
+        m.return_value = True
+        self.assertTrue(self.wrapper.check_repo_verify(True))
+
+        m.return_value = False
+        self.assertFalse(self.wrapper.check_repo_verify(True))
+
+
+class CheckRepoRev(GitCheckoutTestCase):
+  """Check check_repo_rev behavior."""
+
+  def test_verify_works(self):
+    """Should pass when verification passes."""
+    with mock.patch.object(self.wrapper, 'check_repo_verify', return_value=True):
+      with mock.patch.object(self.wrapper, 'verify_rev', return_value='12345'):
+        rrev, lrev = self.wrapper.check_repo_rev(self.GIT_DIR, 'stable')
+    self.assertEqual('refs/heads/stable', rrev)
+    self.assertEqual('12345', lrev)
+
+  def test_verify_fails(self):
+    """Should fail when verification fails."""
+    with mock.patch.object(self.wrapper, 'check_repo_verify', return_value=True):
+      with mock.patch.object(self.wrapper, 'verify_rev', side_effect=Exception):
+        with self.assertRaises(Exception):
+          self.wrapper.check_repo_rev(self.GIT_DIR, 'stable')
+
+  def test_verify_ignore(self):
+    """Should pass when verification is disabled."""
+    with mock.patch.object(self.wrapper, 'verify_rev', side_effect=Exception):
+      rrev, lrev = self.wrapper.check_repo_rev(self.GIT_DIR, 'stable', repo_verify=False)
+    self.assertEqual('refs/heads/stable', rrev)
+    self.assertEqual(self.REV_LIST[1], lrev)
+
+
 if __name__ == '__main__':
   unittest.main()

tox.ini

@@ -17,9 +17,20 @@
 [tox]
 envlist = py27, py36, py37, py38
 
+[gh-actions]
+python =
+    2.7: py27
+    3.6: py36
+    3.7: py37
+    3.8: py38
+
 [testenv]
 deps = pytest
-commands = {toxinidir}/run_tests
+commands = {envpython} run_tests
+setenv =
+    GIT_AUTHOR_NAME = Repo test author
+    GIT_COMMITTER_NAME = Repo test committer
+    EMAIL = repo@gerrit.nodomain
 
 [testenv:py27]
 deps =