Fixing forall subcommand for Py3

Execution of 'repo forall -p -c' doesn't work with Py3 and ends up
with an error:

Got an error, terminating the pool: TypeError: can only concatenate
str (not "bytes") to str

That's fixed by using the decode() method.

Change-Id: Ice01aaa1822dde8d957b5bf096021dd5a2b7dd51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253659
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Jiri Tyr <jiri.tyr@gmail.com>
(cherry picked from commit 83a3227b62)

docs/internal-fs-layout.md

@@ -1,145 +0,0 @@
-# Repo internal filesystem layout
-
-A reference to the `.repo/` tree in repo client checkouts.
-Hopefully it's complete & up-to-date, but who knows!
-
-*** note
-**Warning**:
-This is meant for developers of the repo project itself as a quick reference.
-**Nothing** in here must be construed as ABI, or that repo itself will never
-change its internals in backwards incompatible ways.
-***
-
-[TOC]
-
-## .repo/ layout
-
-All content under `.repo/` is managed by `repo` itself with few exceptions.
-
-In general, you should not make manual changes in here.
-If a setting was initialized using an option to `repo init`, you should use that
-command to change the setting later on.
-It is always safe to re-run `repo init` in existing repo client checkouts.
-For example, if you want to change the manifest branch, you can simply run
-`repo init --manifest-branch=<new name>` and repo will take care of the rest.
-
-### repo/ state
-
-*   `repo/`: A git checkout of the repo project.  This is how `repo` re-execs
-    itself to get the latest released version.
-
-    It tracks the git repository at `REPO_URL` using the `REPO_REV` branch.
-    Those are specified at `repo init` time using the `--repo-url=<REPO_URL>`
-    and `--repo-branch=<REPO_REV>` options.
-
-    Any changes made to this directory will usually be automatically discarded
-    by repo itself when it checks for updates.  If you want to update to the
-    latest version of repo, use `repo selfupdate` instead.  If you want to
-    change the git URL/branch that this tracks, re-run `repo init` with the new
-    settings.
-
-*   `.repo_fetchtimes.json`: Used by `repo sync` to record stats when syncing
-    the various projects.
-
-### Manifests
-
-For more documentation on the manifest format, including the local_manifests
-support, see the [manifest-format.md] file.
-
-*   `manifests/`: A git checkout of the manifest project.  Its `.git/` state
-    points to the `manifest.git` bare checkout (see below).  It tracks the git
-    branch specified at `repo init` time via `--manifest-branch`.
-
-    The local branch name is always `default` regardless of the remote tracking
-    branch.  Do not get confused if the remote branch is not `default`, or if
-    there is a remote `default` that is completely different!
-
-    No manual changes should be made in here as it will just confuse repo and
-    it won't automatically recover causing no new changes to be picked up.
-
-*   `manifests.git/`: A bare checkout of the manifest project.  It tracks the
-    git repository specified at `repo init` time via `--manifest-url`.
-
-    No manual changes should be made in here as it will just confuse repo.
-    If you want to switch the tracking settings, re-run `repo init` with the
-    new settings.
-
-*   `manifest.xml -> manifests/<manifest-name>.xml`: A symlink to the manifest
-    that the user wishes to sync.  It is specified at `repo init` time via
-    `--manifest-name`.
-
-    Do not try to repoint this symlink to other files as it will confuse repo.
-    If you want to switch manifest files, re-run `repo init` with the new
-    setting.
-
-*   `manifests.git/.repo_config.json`: JSON cache of the `manifests.git/config`
-    file for repo to read/process quickly.
-
-*   `local_manifest.xml` (*Deprecated*): User-authored tweaks to the manifest
-    used to sync.  See [local manifests] for more details.
-*   `local_manifests/`: Directory of user-authored manifest fragments to tweak
-    the manifest used to sync.  See [local manifests] for more details.
-
-### Project objects
-
-*   `project.list`: Tracking file used by `repo sync` to determine when projects
-    are added or removed and need corresponding updates in the checkout.
-*   `projects/`: Bare checkouts of every project synced by the manifest.  The
-    filesystem layout matches the `<project path=...` setting in the manifest
-    (i.e. where it's checked out in the repo client source tree).  Those
-    checkouts will symlink their `.git/` state to paths under here.
-
-    Some git state is further split out under `project-objects/`.
-*   `project-objects/`: Git objects that are safe to share across multiple
-    git checkouts.  The filesystem layout matches the `<project name=...`
-    setting in the manifest (i.e. the path on the remote server).  This allows
-    for multiple checkouts of the same remote git repo to share their objects.
-    For example, you could have different branches of `foo/bar.git` checked
-    out to `foo/bar-master`, `foo/bar-release`, etc...  There will be multiple
-    trees under `projects/` for each one, but only one under `project-objects/`.
-
-    This can run into problems if different remotes use the same path on their
-    respective servers ...
-*   `subprojects/`: Like `projects/`, but for git submodules.
-*   `subproject-objects/`: Like `project-objects/`, but for git submodules.
-
-### Settings
-
-The `.repo/manifests.git/config` file is used to track settings for the entire
-repo client checkout.
-Most settings use the `[repo]` section to avoid conflicts with git.
-User controlled settings are initialized when running `repo init`.
-
-| Setting           | `repo init` Option        | Use/Meaning |
-|-------------------|---------------------------|-------------|
-| manifest.groups   | `--groups` & `--platform` | The manifest groups to sync |
-| repo.archive      | `--archive`               | Use `git archive` for checkouts |
-| repo.clonefilter  | `--clone-filter`          | Filter setting when using [partial git clones] |
-| repo.depth        | `--depth`                 | Create shallow checkouts when cloning |
-| repo.dissociate   | `--dissociate`            | Dissociate from any reference/mirrors after initial clone |
-| repo.mirror       | `--mirror`                | Checkout is a repo mirror |
-| repo.partialclone | `--partial-clone`         | Create [partial git clones] |
-| repo.reference    | `--reference`             | Reference repo client checkout |
-| repo.submodules   | `--submodules`            | Sync git submodules |
-| user.email        | `--config-name`           | User's e-mail address; Copied into `.git/config` when checking out a new project |
-| user.name         | `--config-name`           | User's name; Copied into `.git/config` when checking out a new project |
-
-[partial git clones]: https://git-scm.com/docs/gitrepository-layout#_code_partialclone_code
-
-## ~/ dotconfig layout
-
-Repo will create & maintain a few files in the user's home directory.
-
-*   `.repoconfig/`: Repo's per-user directory for all random config files/state.
-*   `.repoconfig/keyring-version`: Cache file for checking if the gnupg subdir
-    has all the same keys as the repo launcher.  Used to avoid running gpg
-    constantly as that can be quite slow.
-*   `.repoconfig/gnupg/`: GnuPG's internal state directory used when repo needs
-    to run `gpg`.  This provides isolation from the user's normal `~/.gnupg/`.
-
-*   `.repo_.gitconfig.json`: JSON cache of the `.gitconfig` file for repo to
-    read/process quickly.
-
-
-[manifest-format.md]: ./manifest-format.md
-[local manifests]: ./manifest-format.md#Local-Manifests

docs/manifest-format.md

@@ -89,7 +89,6 @@ following DTD:
   <!ATTLIST extend-project path CDATA #IMPLIED>
   <!ATTLIST extend-project groups CDATA #IMPLIED>
   <!ATTLIST extend-project revision CDATA #IMPLIED>
-  <!ATTLIST extend-project remote CDATA #IMPLIED>
 
   <!ELEMENT remove-project EMPTY>
   <!ATTLIST remove-project name  CDATA #REQUIRED>
@@ -307,9 +306,6 @@ belongs.  Same syntax as the corresponding element of `project`.
 Attribute `revision`: If specified, overrides the revision of the original
 project.  Same syntax as the corresponding element of `project`.
 
-Attribute `remote`: If specified, overrides the remote of the original
-project.  Same syntax as the corresponding element of `project`.
-
 ### Element annotation
 
 Zero or more annotation elements may be specified as children of a
@@ -342,7 +338,7 @@ It's just like copyfile and runs at the same time as copyfile but
 instead of copying it creates a symlink.
 
 The symlink is created at "dest" (relative to the top of the tree) and
-points to the path specified by "src" which is a path in the project.
+points to the path specified by "src".
 
 Parent directories of "dest" will be automatically created if missing.
 

docs/release-process.md

@@ -161,91 +161,7 @@ You can create a short changelog using the command:
 $ git log --format="%h (%aN) %s" --no-merges origin/stable..$r
 ```
 
-## Project References
 
-Here's a table showing the relationship of major tools, their EOL dates, and
-their status in Ubuntu & Debian.
-Those distros tend to be good indicators of how long we need to support things.
-
-Things in bold indicate stuff to take note of, but does not guarantee that we
-still support them.
-Things in italics are things we used to care about but probably don't anymore.
-
-|   Date   |     EOL      | [Git][rel-g] | [Python][rel-p] | [Ubuntu][rel-u] / [Debian][rel-d] | Git | Python |
-|:--------:|:------------:|--------------|-----------------|-----------------------------------|-----|--------|
-| Oct 2008 | *Oct 2013*   |              | 2.6.0           | *10.04 Lucid* - 10.10 Maverick / *Squeeze* |
-| Dec 2008 | *Feb 2009*   |              | 3.0.0           |
-| Feb 2009 | *Mar 2012*   |              |                 | Debian 5 Lenny       | 1.5.6.5 | 2.5.2 |
-| Jun 2009 | *Jun 2016*   |              | 3.1.0           | *10.04 Lucid* - 10.10 Maverick / *Squeeze* |
-| Feb 2010 | *Oct 2012*   | 1.7.0        |                 | *10.04 Lucid* - *12.04 Precise* - 12.10 Quantal |
-| Apr 2010 | *Apr 2015*   |              |                 | *10.04 Lucid*        | 1.7.0.4  | 2.6.5 3.1.2  |
-| Jul 2010 | *Dec 2019*   |              | **2.7.0**       | 11.04 Natty - **<current>** |
-| Oct 2010 |              |              |                 | 10.10 Maverick       | 1.7.1    | 2.6.6 3.1.3  |
-| Feb 2011 | *Feb 2016*   |              |                 | Debian 6 Squeeze     | 1.7.2.5  | 2.6.6 3.1.3  |
-| Apr 2011 |              |              |                 | 11.04 Natty          | 1.7.4    | 2.7.1 3.2.0  |
-| Oct 2011 | *Feb 2016*   |              | 3.2.0           | 11.04 Natty - 12.10 Quantal |
-| Oct 2011 |              |              |                 | 11.10 Ocelot         | 1.7.5.4  | 2.7.2 3.2.2  |
-| Apr 2012 | *Apr 2019*   |              |                 | *12.04 Precise*      | 1.7.9.5  | 2.7.3 3.2.3  |
-| Sep 2012 | *Sep 2017*   |              | 3.3.0           | 13.04 Raring - 13.10 Saucy |
-| Oct 2012 | *Dec 2014*   | 1.8.0        |                 | 13.04 Raring - 13.10 Saucy |
-| Oct 2012 |              |              |                 | 12.10 Quantal        | 1.7.10.4 | 2.7.3 3.2.3  |
-| Apr 2013 |              |              |                 | 13.04 Raring         | 1.8.1.2  | 2.7.4 3.3.1  |
-| May 2013 | *May 2018*   |              |                 | Debian 7 Wheezy      | 1.7.10.4 | 2.7.3 3.2.3  |
-| Oct 2013 |              |              |                 | 13.10 Saucy          | 1.8.3.2  | 2.7.5 3.3.2  |
-| Feb 2014 | *Dec 2014*   | **1.9.0**    |                 | **14.04 Trusty** |
-| Mar 2014 | *Mar 2019*   |              | **3.4.0**       | **14.04 Trusty** - 15.10 Wily / **Jessie** |
-| Apr 2014 | **Apr 2022** |              |                 | **14.04 Trusty**     | 1.9.1    | 2.7.5 3.4.0  |
-| May 2014 | *Dec 2014*   | 2.0.0        |
-| Aug 2014 | *Dec 2014*   | **2.1.0**    |                 | 14.10 Utopic - 15.04 Vivid / **Jessie** |
-| Oct 2014 |              |              |                 | 14.10 Utopic         | 2.1.0    | 2.7.8 3.4.2  |
-| Nov 2014 | *Sep 2015*   | 2.2.0        |
-| Feb 2015 | *Sep 2015*   | 2.3.0        |
-| Apr 2015 | *May 2017*   | 2.4.0        |
-| Apr 2015 | **Jun 2020** |              |                 | **Debian 8 Jessie**  | 2.1.4    | 2.7.9 3.4.2  |
-| Apr 2015 |              |              |                 | 15.04 Vivid          | 2.1.4    | 2.7.9 3.4.3  |
-| Jul 2015 | *May 2017*   | 2.5.0        |                 | 15.10 Wily |
-| Sep 2015 | *May 2017*   | 2.6.0        |
-| Sep 2015 | **Sep 2020** |              | **3.5.0**       | **16.04 Xenial** - 17.04 Zesty / **Stretch** |
-| Oct 2015 |              |              |                 | 15.10 Wily           | 2.5.0    | 2.7.9 3.4.3  |
-| Jan 2016 | *Jul 2017*   | **2.7.0**    |                 | **16.04 Xenial** |
-| Mar 2016 | *Jul 2017*   | 2.8.0        |
-| Apr 2016 | **Apr 2024** |              |                 | **16.04 Xenial**     | 2.7.4    | 2.7.11 3.5.1 |
-| Jun 2016 | *Jul 2017*   | 2.9.0        |                 | 16.10 Yakkety |
-| Sep 2016 | *Sep 2017*   | 2.10.0       |
-| Oct 2016 |              |              |                 | 16.10 Yakkety        | 2.9.3    | 2.7.11 3.5.1 |
-| Nov 2016 | *Sep 2017*   | **2.11.0**   |                 | 17.04 Zesty / **Stretch** |
-| Dec 2016 | **Dec 2021** |              | **3.6.0**       | 17.10 Artful - **18.04 Bionic** - 18.10 Cosmic |
-| Feb 2017 | *Sep 2017*   | 2.12.0       |
-| Apr 2017 |              |              |                 | 17.04 Zesty          | 2.11.0   | 2.7.13 3.5.3 |
-| May 2017 | *May 2018*   | 2.13.0       |
-| Jun 2017 | **Jun 2022** |              |                 | **Debian 9 Stretch** | 2.11.0   | 2.7.13 3.5.3 |
-| Aug 2017 | *Dec 2019*   | 2.14.0       |                 | 17.10 Artful |
-| Oct 2017 | *Dec 2019*   | 2.15.0       |
-| Oct 2017 |              |              |                 | 17.10 Artful         | 2.14.1   | 2.7.14 3.6.3 |
-| Jan 2018 | *Dec 2019*   | 2.16.0       |
-| Apr 2018 | *Dec 2019*   | 2.17.0       |                 | **18.04 Bionic**     |
-| Apr 2018 | **Apr 2028** |              |                 | **18.04 Bionic**     | 2.17.0   | 2.7.15 3.6.5 |
-| Jun 2018 | *Dec 2019*   | 2.18.0       |
-| Jun 2018 | **Jun 2023** |              | 3.7.0           | 19.04 Disco - **20.04 Focal** / **Buster** |
-| Sep 2018 | *Dec 2019*   | 2.19.0       |                 | 18.10 Cosmic |
-| Oct 2018 |              |              |                 | 18.10 Cosmic         | 2.19.1   | 2.7.15 3.6.6 |
-| Dec 2018 | *Dec 2019*   | **2.20.0**   |                 | 19.04 Disco / **Buster** |
-| Feb 2019 | *Dec 2019*   | 2.21.0       |
-| Apr 2019 |              |              |                 | 19.04 Disco          | 2.20.1   | 2.7.16 3.7.3 |
-| Jun 2019 |              | 2.22.0       |
-| Jul 2019 | **Jul 2024** |              |                 | **Debian 10 Buster** | 2.20.1   | 2.7.16 3.7.3 |
-| Aug 2019 |              | 2.23.0       |
-| Oct 2019 | **Oct 2024** |              | 3.8.0           |
-| Oct 2019 |              |              |                 | 19.10 Eoan           | 2.20.1   | 2.7.17 3.7.5 |
-| Nov 2019 |              | 2.24.0       |
-| Jan 2020 |              | 2.25.0       |                 | **20.04 Focal** |
-| Apr 2020 | **Apr 2030** |              |                 | **20.04 Focal**      | 2.25.0   | 2.7.17 3.7.5 |
-
-
-[rel-d]: https://en.wikipedia.org/wiki/Debian_version_history
-[rel-g]: https://en.wikipedia.org/wiki/Git#Releases
-[rel-p]: https://en.wikipedia.org/wiki/History_of_Python#Table_of_versions
-[rel-u]: https://en.wikipedia.org/wiki/Ubuntu_version_history#Table_of_versions
 [example announcement]: https://groups.google.com/d/topic/repo-discuss/UGBNismWo1M/discussion
 [repo-discuss@googlegroups.com]: https://groups.google.com/forum/#!forum/repo-discuss
 [go/repo-release]: https://goto.google.com/repo-release

error.py

@@ -22,10 +22,6 @@ class ManifestInvalidRevisionError(Exception):
   """The revision value in a project is incorrect.
   """
 
-class ManifestInvalidPathError(Exception):
-  """A path used in <copyfile> or <linkfile> is incorrect.
-  """
-
 class NoManifestException(Exception):
   """The required manifest does not exist.
   """

git_command.py

@@ -28,17 +28,7 @@ from repo_trace import REPO_TRACE, IsTrace, Trace
 from wrapper import Wrapper
 
 GIT = 'git'
-# NB: These do not need to be kept in sync with the repo launcher script.
+MIN_GIT_VERSION = (1, 5, 4)
-# These may be much newer as it allows the repo launcher to roll between
-# different repo releases while source versions might require a newer git.
-#
-# The soft version is when we start warning users that the version is old and
-# we'll be dropping support for it.  We'll refuse to work with versions older
-# than the hard version.
-#
-# git-1.7 is in (EOL) Ubuntu Precise.  git-1.9 is in Ubuntu Trusty.
-MIN_GIT_VERSION_SOFT = (1, 9, 1)
-MIN_GIT_VERSION_HARD = (1, 7, 2)
 GIT_DIR = 'GIT_DIR'
 
 LAST_GITDIR = None

main.py

@@ -255,41 +255,27 @@ class _Repo(object):
     return result
 
 
-def _CheckWrapperVersion(ver_str, repo_path):
+def _CheckWrapperVersion(ver, repo_path):
-  """Verify the repo launcher is new enough for this checkout.
-
-  Args:
-    ver_str: The version string passed from the repo launcher when it ran us.
-    repo_path: The path to the repo launcher that loaded us.
-  """
-  # Refuse to work with really old wrapper versions.  We don't test these,
-  # so might as well require a somewhat recent sane version.
-  # v1.15 of the repo launcher was released in ~Mar 2012.
-  MIN_REPO_VERSION = (1, 15)
-  min_str = '.'.join(str(x) for x in MIN_REPO_VERSION)
-
   if not repo_path:
     repo_path = '~/bin/repo'
 
-  if not ver_str:
+  if not ver:
     print('no --wrapper-version argument', file=sys.stderr)
     sys.exit(1)
 
-  # Pull out the version of the repo launcher we know about to compare.
   exp = Wrapper().VERSION
-  ver = tuple(map(int, ver_str.split('.')))
+  ver = tuple(map(int, ver.split('.')))
+  if len(ver) == 1:
+    ver = (0, ver[0])
 
   exp_str = '.'.join(map(str, exp))
-  if ver < MIN_REPO_VERSION:
+  if exp[0] > ver[0] or ver < (0, 4):
     print("""
-repo: error:
+!!! A new repo command (%5s) is available.    !!!
-!!! Your version of repo %s is too old.
+!!! You must upgrade before you can continue:   !!!
-!!! We need at least version %s.
-!!! A new repo command (%s) is available.
-!!! You must upgrade before you can continue:
 
     cp %s %s
-""" % (ver_str, min_str, exp_str, WrapperPath(), repo_path), file=sys.stderr)
+""" % (exp_str, WrapperPath(), repo_path), file=sys.stderr)
     sys.exit(1)
 
   if exp > ver:

manifest_xml.py

@@ -35,8 +35,7 @@ from git_config import GitConfig
 from git_refs import R_HEADS, HEAD
 import platform_utils
 from project import RemoteSpec, Project, MetaProject
-from error import (ManifestParseError, ManifestInvalidPathError,
+from error import ManifestParseError, ManifestInvalidRevisionError
-                   ManifestInvalidRevisionError)
 
 MANIFEST_FILE_NAME = 'manifest.xml'
 LOCAL_MANIFEST_NAME = 'local_manifest.xml'
@@ -599,9 +598,6 @@ class XmlManifest(object):
         if groups:
           groups = self._ParseGroups(groups)
         revision = node.getAttribute('revision')
-        remote = node.getAttribute('remote')
-        if remote:
-          remote = self._get_remote(node)
 
         for p in self._projects[name]:
           if path and p.relpath != path:
@@ -610,8 +606,6 @@ class XmlManifest(object):
             p.groups.extend(groups)
           if revision:
             p.revisionExpr = revision
-          if remote:
-            p.remote = remote.ToRemoteSpec(name)
       if node.nodeName == 'repo-hooks':
         # Get the name of the project and the (space-separated) list of enabled.
         repo_hooks_project = self._reqatt(node, 'in-project')
@@ -949,101 +943,21 @@ class XmlManifest(object):
       worktree = os.path.join(parent.worktree, path).replace('\\', '/')
     return relpath, worktree, gitdir, objdir
 
-  @staticmethod
-  def _CheckLocalPath(path, symlink=False):
-    """Verify |path| is reasonable for use in <copyfile> & <linkfile>."""
-    if '~' in path:
-      return '~ not allowed (due to 8.3 filenames on Windows filesystems)'
-
-    # Some filesystems (like Apple's HFS+) try to normalize Unicode codepoints
-    # which means there are alternative names for ".git".  Reject paths with
-    # these in it as there shouldn't be any reasonable need for them here.
-    # The set of codepoints here was cribbed from jgit's implementation:
-    # https://eclipse.googlesource.com/jgit/jgit/+/9110037e3e9461ff4dac22fee84ef3694ed57648/org.eclipse.jgit/src/org/eclipse/jgit/lib/ObjectChecker.java#884
-    BAD_CODEPOINTS = {
-        u'\u200C',  # ZERO WIDTH NON-JOINER
-        u'\u200D',  # ZERO WIDTH JOINER
-        u'\u200E',  # LEFT-TO-RIGHT MARK
-        u'\u200F',  # RIGHT-TO-LEFT MARK
-        u'\u202A',  # LEFT-TO-RIGHT EMBEDDING
-        u'\u202B',  # RIGHT-TO-LEFT EMBEDDING
-        u'\u202C',  # POP DIRECTIONAL FORMATTING
-        u'\u202D',  # LEFT-TO-RIGHT OVERRIDE
-        u'\u202E',  # RIGHT-TO-LEFT OVERRIDE
-        u'\u206A',  # INHIBIT SYMMETRIC SWAPPING
-        u'\u206B',  # ACTIVATE SYMMETRIC SWAPPING
-        u'\u206C',  # INHIBIT ARABIC FORM SHAPING
-        u'\u206D',  # ACTIVATE ARABIC FORM SHAPING
-        u'\u206E',  # NATIONAL DIGIT SHAPES
-        u'\u206F',  # NOMINAL DIGIT SHAPES
-        u'\uFEFF',  # ZERO WIDTH NO-BREAK SPACE
-    }
-    if BAD_CODEPOINTS & set(path):
-      # This message is more expansive than reality, but should be fine.
-      return 'Unicode combining characters not allowed'
-
-    # Assume paths might be used on case-insensitive filesystems.
-    path = path.lower()
-
-    # Some people use src="." to create stable links to projects.  Lets allow
-    # that but reject all other uses of "." to keep things simple.
-    parts = path.split(os.path.sep)
-    if parts != ['.']:
-      for part in set(parts):
-        if part in {'.', '..', '.git'} or part.startswith('.repo'):
-          return 'bad component: %s' % (part,)
-
-    if not symlink and path.endswith(os.path.sep):
-      return 'dirs not allowed'
-
-    norm = os.path.normpath(path)
-    if norm == '..' or norm.startswith('../') or norm.startswith(os.path.sep):
-      return 'path cannot be outside'
-
-  @classmethod
-  def _ValidateFilePaths(cls, element, src, dest):
-    """Verify |src| & |dest| are reasonable for <copyfile> & <linkfile>.
-
-    We verify the path independent of any filesystem state as we won't have a
-    checkout available to compare to.  i.e. This is for parsing validation
-    purposes only.
-
-    We'll do full/live sanity checking before we do the actual filesystem
-    modifications in _CopyFile/_LinkFile/etc...
-    """
-    # |dest| is the file we write to or symlink we create.
-    # It is relative to the top of the repo client checkout.
-    msg = cls._CheckLocalPath(dest)
-    if msg:
-      raise ManifestInvalidPathError(
-          '<%s> invalid "dest": %s: %s' % (element, dest, msg))
-
-    # |src| is the file we read from or path we point to for symlinks.
-    # It is relative to the top of the git project checkout.
-    msg = cls._CheckLocalPath(src, symlink=element == 'linkfile')
-    if msg:
-      raise ManifestInvalidPathError(
-          '<%s> invalid "src": %s: %s' % (element, src, msg))
-
   def _ParseCopyFile(self, project, node):
     src = self._reqatt(node, 'src')
     dest = self._reqatt(node, 'dest')
     if not self.IsMirror:
       # src is project relative;
-      # dest is relative to the top of the tree.
+      # dest is relative to the top of the tree
-      # We only validate paths if we actually plan to process them.
+      project.AddCopyFile(src, dest, os.path.join(self.topdir, dest))
-      self._ValidateFilePaths('copyfile', src, dest)
-      project.AddCopyFile(src, dest, self.topdir)
 
   def _ParseLinkFile(self, project, node):
     src = self._reqatt(node, 'src')
     dest = self._reqatt(node, 'dest')
     if not self.IsMirror:
       # src is project relative;
-      # dest is relative to the top of the tree.
+      # dest is relative to the top of the tree
-      # We only validate paths if we actually plan to process them.
+      project.AddLinkFile(src, dest, os.path.join(self.topdir, dest))
-      self._ValidateFilePaths('linkfile', src, dest)
-      project.AddLinkFile(src, dest, self.topdir)
 
   def _ParseAnnotation(self, project, node):
     name = self._reqatt(node, 'name')

platform_utils_win32.py

@@ -19,18 +19,13 @@ import errno
 from pyversion import is_python3
 from ctypes import WinDLL, get_last_error, FormatError, WinError, addressof
 from ctypes import c_buffer
-from ctypes.wintypes import BOOL, BOOLEAN, LPCWSTR, DWORD, HANDLE
+from ctypes.wintypes import BOOL, BOOLEAN, LPCWSTR, DWORD, HANDLE, POINTER, c_ubyte
-from ctypes.wintypes import WCHAR, USHORT, LPVOID, ULONG
+from ctypes.wintypes import WCHAR, USHORT, LPVOID, Structure, Union, ULONG
-if is_python3():
+from ctypes.wintypes import byref
-  from ctypes import c_ubyte, Structure, Union, byref
-  from ctypes.wintypes import LPDWORD
-else:
-  # For legacy Python2 different imports are needed.
-  from ctypes.wintypes import POINTER, c_ubyte, Structure, Union, byref
-  LPDWORD = POINTER(DWORD)
 
 kernel32 = WinDLL('kernel32', use_last_error=True)
 
+LPDWORD = POINTER(DWORD)
 UCHAR = c_ubyte
 
 # Win32 error codes

project.py

@@ -36,7 +36,7 @@ from git_command import GitCommand, git_require
 from git_config import GitConfig, IsId, GetSchemeFromUrl, GetUrlCookieFile, \
     ID_RE
 from error import GitError, HookError, UploadError, DownloadError
-from error import ManifestInvalidRevisionError, ManifestInvalidPathError
+from error import ManifestInvalidRevisionError
 from error import NoManifestException
 import platform_utils
 import progress
@@ -261,70 +261,17 @@ class _Annotation(object):
     self.keep = keep
 
 
-def _SafeExpandPath(base, subpath, skipfinal=False):
-  """Make sure |subpath| is completely safe under |base|.
-
-  We make sure no intermediate symlinks are traversed, and that the final path
-  is not a special file (e.g. not a socket or fifo).
-
-  NB: We rely on a number of paths already being filtered out while parsing the
-  manifest.  See the validation logic in manifest_xml.py for more details.
-  """
-  components = subpath.split(os.path.sep)
-  if skipfinal:
-    # Whether the caller handles the final component itself.
-    finalpart = components.pop()
-
-  path = base
-  for part in components:
-    if part in {'.', '..'}:
-      raise ManifestInvalidPathError(
-          '%s: "%s" not allowed in paths' % (subpath, part))
-
-    path = os.path.join(path, part)
-    if platform_utils.islink(path):
-      raise ManifestInvalidPathError(
-          '%s: traversing symlinks not allow' % (path,))
-
-    if os.path.exists(path):
-      if not os.path.isfile(path) and not platform_utils.isdir(path):
-        raise ManifestInvalidPathError(
-            '%s: only regular files & directories allowed' % (path,))
-
-  if skipfinal:
-    path = os.path.join(path, finalpart)
-
-  return path
-
-
 class _CopyFile(object):
-  """Container for <copyfile> manifest element."""
 
-  def __init__(self, git_worktree, src, topdir, dest):
+  def __init__(self, src, dest, abssrc, absdest):
-    """Register a <copyfile> request.
-
-    Args:
-      git_worktree: Absolute path to the git project checkout.
-      src: Relative path under |git_worktree| of file to read.
-      topdir: Absolute path to the top of the repo client checkout.
-      dest: Relative path under |topdir| of file to write.
-    """
-    self.git_worktree = git_worktree
-    self.topdir = topdir
     self.src = src
     self.dest = dest
+    self.abs_src = abssrc
+    self.abs_dest = absdest
 
   def _Copy(self):
-    src = _SafeExpandPath(self.git_worktree, self.src)
+    src = self.abs_src
-    dest = _SafeExpandPath(self.topdir, self.dest)
+    dest = self.abs_dest
-
-    if platform_utils.isdir(src):
-      raise ManifestInvalidPathError(
-          '%s: copying from directory not supported' % (self.src,))
-    if platform_utils.isdir(dest):
-      raise ManifestInvalidPathError(
-          '%s: copying to directory not allowed' % (self.dest,))
-
     # copy file if it does not exist or is out of date
     if not os.path.exists(dest) or not filecmp.cmp(src, dest):
       try:
@@ -345,21 +292,13 @@ class _CopyFile(object):
 
 
 class _LinkFile(object):
-  """Container for <linkfile> manifest element."""
 
-  def __init__(self, git_worktree, src, topdir, dest):
+  def __init__(self, git_worktree, src, dest, relsrc, absdest):
-    """Register a <linkfile> request.
-
-    Args:
-      git_worktree: Absolute path to the git project checkout.
-      src: Target of symlink relative to path under |git_worktree|.
-      topdir: Absolute path to the top of the repo client checkout.
-      dest: Relative path under |topdir| of symlink to create.
-    """
     self.git_worktree = git_worktree
-    self.topdir = topdir
     self.src = src
     self.dest = dest
+    self.src_rel_to_dest = relsrc
+    self.abs_dest = absdest
 
   def __linkIt(self, relSrc, absDest):
     # link file if it does not exist or is out of date
@@ -377,42 +316,35 @@ class _LinkFile(object):
         _error('Cannot link file %s to %s', relSrc, absDest)
 
   def _Link(self):
-    """Link the self.src & self.dest paths.
+    """Link the self.rel_src_to_dest and self.abs_dest. Handles wild cards
-
+    on the src linking all of the files in the source in to the destination
-    Handles wild cards on the src linking all of the files in the source in to
+    directory.
-    the destination directory.
     """
-    # Some people use src="." to create stable links to projects.  Lets allow
+    # We use the absSrc to handle the situation where the current directory
-    # that but reject all other uses of "." to keep things simple.
+    # is not the root of the repo
-    if self.src == '.':
+    absSrc = os.path.join(self.git_worktree, self.src)
-      src = self.git_worktree
+    if os.path.exists(absSrc):
+      # Entity exists so just a simple one to one link operation
+      self.__linkIt(self.src_rel_to_dest, self.abs_dest)
     else:
-      src = _SafeExpandPath(self.git_worktree, self.src)
-
-    if os.path.exists(src):
-      # Entity exists so just a simple one to one link operation.
-      dest = _SafeExpandPath(self.topdir, self.dest, skipfinal=True)
-      # dest & src are absolute paths at this point.  Make sure the target of
-      # the symlink is relative in the context of the repo client checkout.
-      relpath = os.path.relpath(src, os.path.dirname(dest))
-      self.__linkIt(relpath, dest)
-    else:
-      dest = _SafeExpandPath(self.topdir, self.dest)
       # Entity doesn't exist assume there is a wild card
-      if os.path.exists(dest) and not platform_utils.isdir(dest):
+      absDestDir = self.abs_dest
-        _error('Link error: src with wildcard, %s must be a directory', dest)
+      if os.path.exists(absDestDir) and not platform_utils.isdir(absDestDir):
+        _error('Link error: src with wildcard, %s must be a directory',
+               absDestDir)
       else:
-        for absSrcFile in glob.glob(src):
+        absSrcFiles = glob.glob(absSrc)
+        for absSrcFile in absSrcFiles:
           # Create a releative path from source dir to destination dir
           absSrcDir = os.path.dirname(absSrcFile)
-          relSrcDir = os.path.relpath(absSrcDir, dest)
+          relSrcDir = os.path.relpath(absSrcDir, absDestDir)
 
           # Get the source file name
           srcFile = os.path.basename(absSrcFile)
 
           # Now form the final full paths to srcFile. They will be
           # absolute for the desintaiton and relative for the srouce.
-          absDest = os.path.join(dest, srcFile)
+          absDest = os.path.join(absDestDir, srcFile)
           relSrc = os.path.join(relSrcDir, srcFile)
           self.__linkIt(relSrc, absDest)
 
@@ -1780,25 +1712,18 @@ class Project(object):
       if submodules:
         syncbuf.later1(self, _dosubmodules)
 
-  def AddCopyFile(self, src, dest, topdir):
+  def AddCopyFile(self, src, dest, absdest):
-    """Mark |src| for copying to |dest| (relative to |topdir|).
+    # dest should already be an absolute path, but src is project relative
+    # make src an absolute path
+    abssrc = os.path.join(self.worktree, src)
+    self.copyfiles.append(_CopyFile(src, dest, abssrc, absdest))
 
-    No filesystem changes occur here.  Actual copying happens later on.
+  def AddLinkFile(self, src, dest, absdest):
-
+    # dest should already be an absolute path, but src is project relative
-    Paths should have basic validation run on them before being queued.
+    # make src relative path to dest
-    Further checking will be handled when the actual copy happens.
+    absdestdir = os.path.dirname(absdest)
-    """
+    relsrc = os.path.relpath(os.path.join(self.worktree, src), absdestdir)
-    self.copyfiles.append(_CopyFile(self.worktree, src, topdir, dest))
+    self.linkfiles.append(_LinkFile(self.worktree, src, dest, relsrc, absdest))
-
-  def AddLinkFile(self, src, dest, topdir):
-    """Mark |dest| to create a symlink (relative to |topdir|) pointing to |src|.
-
-    No filesystem changes occur here.  Actual linking happens later on.
-
-    Paths should have basic validation run on them before being queued.
-    Further checking will be handled when the actual link happens.
-    """
-    self.linkfiles.append(_LinkFile(self.worktree, src, topdir, dest))
 
   def AddAnnotation(self, name, value, keep):
     self.annotations.append(_Annotation(name, value, keep))
@@ -1822,18 +1747,6 @@ class Project(object):
 
 # Branch Management ##
 
-  def GetHeadPath(self):
-    """Return the full path to the HEAD ref."""
-    dotgit = os.path.join(self.worktree, '.git')
-    if os.path.isfile(dotgit):
-      # Git worktrees use a "gitdir:" syntax to point to the scratch space.
-      with open(dotgit) as fp:
-        setting = fp.read()
-        assert setting.startswith('gitdir:')
-        gitdir = setting.split(':', 1)[1].strip()
-      dotgit = os.path.join(self.worktree, gitdir)
-    return os.path.join(dotgit, HEAD)
-
   def StartBranch(self, name, branch_merge='', revision=None):
     """Create a new branch off the manifest's revision.
     """
@@ -1873,7 +1786,8 @@ class Project(object):
       except OSError:
         pass
       _lwrite(ref, '%s\n' % revid)
-      _lwrite(self.GetHeadPath(), 'ref: %s%s\n' % (R_HEADS, name))
+      _lwrite(os.path.join(self.worktree, '.git', HEAD),
+              'ref: %s%s\n' % (R_HEADS, name))
       branch.Save()
       return True
 
@@ -1920,7 +1834,8 @@ class Project(object):
       # Same revision; just update HEAD to point to the new
       # target branch, but otherwise take no other action.
       #
-      _lwrite(self.GetHeadPath(), 'ref: %s%s\n' % (R_HEADS, name))
+      _lwrite(os.path.join(self.worktree, '.git', HEAD),
+              'ref: %s%s\n' % (R_HEADS, name))
       return True
 
     return GitCommand(self,
@@ -1953,7 +1868,8 @@ class Project(object):
 
       revid = self.GetRevisionId(all_refs)
       if head == revid:
-        _lwrite(self.GetHeadPath(), '%s\n' % revid)
+        _lwrite(os.path.join(self.worktree, '.git', HEAD),
+                '%s\n' % revid)
       else:
         self._Checkout(revid, quiet=True)
 
@@ -2073,7 +1989,7 @@ class Project(object):
       gitmodules_lines = []
       fd, temp_gitmodules_path = tempfile.mkstemp()
       try:
-        os.write(fd, p.stdout.encode('utf-8'))
+        os.write(fd, p.stdout)
         os.close(fd)
         cmd = ['config', '--file', temp_gitmodules_path, '--list']
         p = GitCommand(None, cmd, capture_stdout=True, capture_stderr=True,
@@ -2482,7 +2398,7 @@ class Project(object):
         platform_utils.remove(tmpPath)
     with GetUrlCookieFile(srcUrl, quiet) as (cookiefile, proxy):
       if cookiefile:
-        cmd += ['--cookie', cookiefile]
+        cmd += ['--cookie', cookiefile, '--cookie-jar', cookiefile]
       if proxy:
         cmd += ['--proxy', proxy]
       elif 'http_proxy' in os.environ and 'darwin' == sys.platform:
@@ -3016,7 +2932,7 @@ class Project(object):
       if self._bare:
         path = os.path.join(self._project.gitdir, HEAD)
       else:
-        path = self._project.GetHeadPath()
+        path = os.path.join(self._project.worktree, '.git', HEAD)
       try:
         with open(path) as fd:
           line = fd.readline()

repo

@@ -10,93 +10,13 @@ copy of repo in the checkout.
 
 from __future__ import print_function
 
-import os
-import platform
-import subprocess
-import sys
-
-
-def exec_command(cmd):
-  """Execute |cmd| or return None on failure."""
-  try:
-    if platform.system() == 'Windows':
-      ret = subprocess.call(cmd)
-      sys.exit(ret)
-    else:
-      os.execvp(cmd[0], cmd)
-  except:
-    pass
-
-
-def check_python_version():
-  """Make sure the active Python version is recent enough."""
-  def reexec(prog):
-    exec_command([prog] + sys.argv)
-
-  MIN_PYTHON_VERSION = (3, 6)
-
-  ver = sys.version_info
-  major = ver.major
-  minor = ver.minor
-
-  # Abort on very old Python 2 versions.
-  if (major, minor) < (2, 7):
-    print('repo: error: Your Python version is too old. '
-          'Please use Python {}.{} or newer instead.'.format(
-              *MIN_PYTHON_VERSION), file=sys.stderr)
-    sys.exit(1)
-
-  # Try to re-exec the version specific Python 3 if needed.
-  if (major, minor) < MIN_PYTHON_VERSION:
-    # Python makes releases ~once a year, so try our min version +10 to help
-    # bridge the gap.  This is the fallback anyways so perf isn't critical.
-    min_major, min_minor = MIN_PYTHON_VERSION
-    for inc in range(0, 10):
-      reexec('python{}.{}'.format(min_major, min_minor + inc))
-
-    # Try the generic Python 3 wrapper, but only if it's new enough.  We don't
-    # want to go from (still supported) Python 2.7 to (unsupported) Python 3.5.
-    try:
-      proc = subprocess.Popen(
-          ['python3', '-c', 'import sys; '
-           'print(sys.version_info.major, sys.version_info.minor)'],
-          stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-      (output, _) = proc.communicate()
-      python3_ver = tuple(int(x) for x in output.decode('utf-8').split())
-    except (OSError, subprocess.CalledProcessError):
-      python3_ver = None
-
-    # The python3 version looks like it's new enough, so give it a try.
-    if python3_ver and python3_ver >= MIN_PYTHON_VERSION:
-      reexec('python3')
-
-    # We're still here, so diagnose things for the user.
-    if major < 3:
-      print('repo: warning: Python 2 is no longer supported; '
-            'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION),
-            file=sys.stderr)
-    else:
-      print('repo: error: Python 3 version is too old; '
-            'Please use Python {}.{} or newer.'.format(*MIN_PYTHON_VERSION),
-            file=sys.stderr)
-      sys.exit(1)
-
-
-if __name__ == '__main__':
-  # TODO(vapier): Enable this on Windows once we have Python 3 issues fixed.
-  if platform.system() != 'Windows':
-    check_python_version()
-
-
 # repo default configuration
 #
 import os
 REPO_URL = os.environ.get('REPO_URL', None)
 if not REPO_URL:
   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
-REPO_REV = os.environ.get('REPO_REV')
+REPO_REV = 'repo-1'
-if not REPO_REV:
-  REPO_REV = 'stable'
 
 # Copyright (C) 2008 Google Inc.
 #
@@ -113,10 +33,10 @@ if not REPO_REV:
 # limitations under the License.
 
 # increment this whenever we make important changes to this script
-VERSION = (2, 0)
+VERSION = (1, 26)
 
 # increment this if the MAINTAINER_KEYS block is modified
-KEYRING_VERSION = (2, 0)
+KEYRING_VERSION = (1, 2)
 
 # Each individual key entry is created by using:
 # gpg --armor --export keyid
@@ -162,20 +82,48 @@ HTHs37+/QLMomGEGKZMWi0dShU2J5mNRQu3Hhxl3hHDVbt5CeJBb26aQcQrFz69W
 zE3GNvmJosh6leayjtI9P2A6iEkEGBECAAkFAkj3uiACGwwACgkQFlMNXpIPXGWp
 TACbBS+Up3RpfYVfd63c1cDdlru13pQAn3NQy/SN858MkxN+zym86UBgOad2
 =CMiZ
+-----END PGP PUBLIC KEY BLOCK-----
+
+     Conley Owens <cco3@android.com>
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+Version: GnuPG v1.4.11 (GNU/Linux)
+
+mQENBFHRvc8BCADFg45Xx/y6QDC+T7Y/gGc7vx0ww7qfOwIKlAZ9xG3qKunMxo+S
+hPCnzEl3cq+6I1Ww/ndop/HB3N3toPXRCoN8Vs4/Hc7by+SnaLFnacrm+tV5/OgT
+V37Lzt8lhay1Kl+YfpFwHYYpIEBLFV9knyfRXS/428W2qhdzYfvB15/AasRmwmor
+py4NIzSs8UD/SPr1ihqNCdZM76+MQyN5HMYXW/ALZXUFG0pwluHFA7hrfPG74i8C
+zMiP7qvMWIl/r/jtzHioH1dRKgbod+LZsrDJ8mBaqsZaDmNJMhss9g76XvfMyLra
+9DI9/iFuBpGzeqBv0hwOGQspLRrEoyTeR6n1ABEBAAG0H0NvbmxleSBPd2VucyA8
+Y2NvM0BhbmRyb2lkLmNvbT6JATgEEwECACIFAlHRvc8CGwMGCwkIBwMCBhUIAgkK
+CwQWAgMBAh4BAheAAAoJEGe35EhpKzgsP6AIAJKJmNtn4l7hkYHKHFSo3egb6RjQ
+zEIP3MFTcu8HFX1kF1ZFbrp7xqurLaE53kEkKuAAvjJDAgI8mcZHP1JyplubqjQA
+xvv84gK+OGP3Xk+QK1ZjUQSbjOpjEiSZpRhWcHci3dgOUH4blJfByHw25hlgHowd
+a/2PrNKZVcJ92YienaxxGjcXEUcd0uYEG2+rwllQigFcnMFDhr9B71MfalRHjFKE
+fmdoypqLrri61YBc59P88Rw2/WUpTQjgNubSqa3A2+CKdaRyaRw+2fdF4TdR0h8W
+zbg+lbaPtJHsV+3mJC7fq26MiJDRJa5ZztpMn8su20gbLgi2ShBOaHAYDDi5AQ0E
+UdG9zwEIAMoOBq+QLNozAhxOOl5GL3StTStGRgPRXINfmViTsihrqGCWBBUfXlUE
+OytC0mYcrDUQev/8ToVoyqw+iGSwDkcSXkrEUCKFtHV/GECWtk1keyHgR10YKI1R
+mquSXoubWGqPeG1PAI74XWaRx8UrL8uCXUtmD8Q5J7mDjKR5NpxaXrwlA0bKsf2E
+Gp9tu1kKauuToZhWHMRMqYSOGikQJwWSFYKT1KdNcOXLQF6+bfoJ6sjVYdwfmNQL
+Ixn8QVhoTDedcqClSWB17VDEFDFa7MmqXZz2qtM3X1R/MUMHqPtegQzBGNhRdnI2
+V45+1Nnx/uuCxDbeI4RbHzujnxDiq70AEQEAAYkBHwQYAQIACQUCUdG9zwIbDAAK
+CRBnt+RIaSs4LNVeB/0Y2pZ8I7gAAcEM0Xw8drr4omg2fUoK1J33ozlA/RxeA/lJ
+I3KnyCDTpXuIeBKPGkdL8uMATC9Z8DnBBajRlftNDVZS3Hz4G09G9QpMojvJkFJV
+By+01Flw/X+eeN8NpqSuLV4W+AjEO8at/VvgKr1AFvBRdZ7GkpI1o6DgPe7ZqX+1
+dzQZt3e13W0rVBb/bUgx9iSLoeWP3aq/k+/GRGOR+S6F6BBSl0SQ2EF2+dIywb1x
+JuinEP+AwLAUZ1Bsx9ISC0Agpk2VeHXPL3FGhroEmoMvBzO0kTFGyoeT7PR/BfKv
++H/g3HsL2LOB9uoIm8/5p2TTU5ttYCXMHhQZ81AY
+=AUp4
 -----END PGP PUBLIC KEY BLOCK-----
 """
 
 GIT = 'git'                      # our git command
-# NB: The version of git that the repo launcher requires may be much older than
-# the version of git that the main repo source tree requires.  Keeping this at
-# an older version also makes it easier for users to upgrade/rollback as needed.
-#
-# git-1.7 is in (EOL) Ubuntu Precise.
 MIN_GIT_VERSION = (1, 7, 2)      # minimum supported git version
 repodir = '.repo'                # name of repo's private directory
 S_repo = 'repo'                  # special repo repository
 S_manifests = 'manifests'        # special manifest repository
 REPO_MAIN = S_repo + '/main.py'  # main script
+MIN_PYTHON_VERSION = (2, 7)      # minimum supported python version
 GITC_CONFIG_FILE = '/gitc/.config'
 GITC_FS_ROOT_DIR = '/gitc/manifest-rw/'
 
@@ -183,9 +131,12 @@ GITC_FS_ROOT_DIR = '/gitc/manifest-rw/'
 import collections
 import errno
 import optparse
+import platform
 import re
 import shutil
 import stat
+import subprocess
+import sys
 
 if sys.version_info[0] == 3:
   import urllib.request
@@ -198,6 +149,17 @@ else:
   urllib.error = urllib2
 
 
+# Python version check
+ver = sys.version_info
+if (ver[0], ver[1]) < MIN_PYTHON_VERSION:
+  print('error: Python version {} unsupported.\n'
+        'Please use Python {}.{} instead.'.format(
+            sys.version.split(' ')[0],
+            MIN_PYTHON_VERSION[0],
+            MIN_PYTHON_VERSION[1],
+        ), file=sys.stderr)
+  sys.exit(1)
+
 home_dot_repo = os.path.expanduser('~/.repoconfig')
 gpg_dir = os.path.join(home_dot_repo, 'gnupg')
 
@@ -273,10 +235,10 @@ group.add_option('--no-tags',
 group = init_optparse.add_option_group('repo Version options')
 group.add_option('--repo-url',
                  dest='repo_url',
-                 help='repo repository location ($REPO_URL)', metavar='URL')
+                 help='repo repository location', metavar='URL')
 group.add_option('--repo-branch',
                  dest='repo_branch',
-                 help='repo branch or revision ($REPO_REV)', metavar='REVISION')
+                 help='repo branch or revision', metavar='REVISION')
 group.add_option('--no-repo-verify',
                  dest='no_repo_verify', action='store_true',
                  help='do not verify repo source code')
@@ -964,9 +926,15 @@ def main(orig_args):
         '--']
   me.extend(orig_args)
   me.extend(extra_args)
-  exec_command(me)
+  try:
-  print("fatal: unable to start %s" % repo_main, file=sys.stderr)
+    if platform.system() == "Windows":
-  sys.exit(148)
+      sys.exit(subprocess.call(me))
+    else:
+      os.execv(sys.executable, me)
+  except OSError as e:
+    print("fatal: unable to start %s" % repo_main, file=sys.stderr)
+    print("fatal: %s" % e, file=sys.stderr)
+    sys.exit(148)
 
 
 if __name__ == '__main__':

subcmds/init.py

@@ -34,7 +34,7 @@ from command import InteractiveCommand, MirrorSafeCommand
 from error import ManifestParseError
 from project import SyncBuffer
 from git_config import GitConfig
-from git_command import git_require, MIN_GIT_VERSION_SOFT, MIN_GIT_VERSION_HARD
+from git_command import git_require, MIN_GIT_VERSION
 import platform_utils
 
 class Init(InteractiveCommand, MirrorSafeCommand):
@@ -451,12 +451,7 @@ to update the working directory files.
       self.OptionParser.error('--mirror and --archive cannot be used together.')
 
   def Execute(self, opt, args):
-    git_require(MIN_GIT_VERSION_HARD, fail=True)
+    git_require(MIN_GIT_VERSION, fail=True)
-    if not git_require(MIN_GIT_VERSION_SOFT):
-      print('repo: warning: git-%s+ will soon be required; please upgrade your '
-            'version of git to maintain support.'
-            % ('.'.join(str(x) for x in MIN_GIT_VERSION_SOFT),),
-            file=sys.stderr)
 
     self._SyncManifest(opt)
     self._LinkManifest(opt.manifest_name)

subcmds/sync.py

@@ -217,10 +217,6 @@ later is required to fix a server side protocol bug.
     p.add_option('-l', '--local-only',
                  dest='local_only', action='store_true',
                  help="only update working tree, don't fetch")
-    p.add_option('--no-manifest-update','--nmu',
-                 dest='mp_update', action='store_false', default='true',
-                 help='use the existing manifest checkout as-is. '
-                      '(do not update to the latest revision)')
     p.add_option('-n', '--network-only',
                  dest='network_only', action='store_true',
                  help="fetch only, don't update working tree")
@@ -368,7 +364,7 @@ later is required to fix a server side protocol bug.
 
     return success
 
-  def _Fetch(self, projects, opt, err_event):
+  def _Fetch(self, projects, opt):
     fetched = set()
     lock = _threading.Lock()
     pm = Progress('Fetching projects', len(projects),
@@ -380,6 +376,7 @@ later is required to fix a server side protocol bug.
 
     threads = set()
     sem = _threading.Semaphore(self.jobs)
+    err_event = _threading.Event()
     for project_list in objdir_project_map.values():
       # Check for any errors before running any more tasks.
       # ...we'll let existing threads finish, though.
@@ -408,11 +405,16 @@ later is required to fix a server side protocol bug.
     for t in threads:
       t.join()
 
+    # If we saw an error, exit with code 1 so that other scripts can check.
+    if err_event.isSet() and opt.fail_fast:
+      print('\nerror: Exited sync due to fetch errors', file=sys.stderr)
+      sys.exit(1)
+
     pm.end()
     self._fetch_times.Save()
 
     if not self.manifest.IsArchive:
-      self._GCProjects(projects, opt, err_event)
+      self._GCProjects(projects)
 
     return fetched
 
@@ -498,16 +500,12 @@ later is required to fix a server side protocol bug.
 
     return success
 
-  def _Checkout(self, all_projects, opt, err_event, err_results):
+  def _Checkout(self, all_projects, opt):
     """Checkout projects listed in all_projects
 
     Args:
       all_projects: List of all projects that should be checked out.
       opt: Program options returned from optparse.  See _Options().
-      err_event: We'll set this event in the case of an error (after printing
-          out info about the error).
-      err_results: A list of strings, paths to git repos where checkout
-          failed.
     """
 
     # Perform checkouts in multiple threads when we are using partial clone.
@@ -526,6 +524,8 @@ later is required to fix a server side protocol bug.
 
     threads = set()
     sem = _threading.Semaphore(syncjobs)
+    err_event = _threading.Event()
+    err_results = []
 
     for project in all_projects:
       # Check for any errors before running any more tasks.
@@ -556,24 +556,20 @@ later is required to fix a server side protocol bug.
       t.join()
 
     pm.end()
+    # If we saw an error, exit with code 1 so that other scripts can check.
+    if err_event.isSet():
+      print('\nerror: Exited sync due to checkout errors', file=sys.stderr)
+      if err_results:
+        print('Failing repos:\n%s' % '\n'.join(err_results),
+              file=sys.stderr)
+      sys.exit(1)
 
-  def _GCProjects(self, projects, opt, err_event):
+  def _GCProjects(self, projects):
     gc_gitdirs = {}
     for project in projects:
-      # Make sure pruning never kicks in with shared projects.
       if len(project.manifest.GetProjectsWithName(project.name)) > 1:
-        print('%s: Shared project %s found, disabling pruning.' %
+        print('Shared project %s found, disabling pruning.' % project.name)
-              (project.relpath, project.name))
+        project.bare_git.config('--replace-all', 'gc.pruneExpire', 'never')
-        if git_require((2, 7, 0)):
-          project.config.SetString('core.repositoryFormatVersion', '1')
-          project.config.SetString('extensions.preciousObjects', 'true')
-        else:
-          # This isn't perfect, but it's the best we can do with old git.
-          print('%s: WARNING: shared projects are unreliable when using old '
-                'versions of git; please upgrade to git-2.7.0+.'
-                % (project.relpath,),
-                file=sys.stderr)
-          project.config.SetString('gc.pruneExpire', 'never')
       gc_gitdirs[project.gitdir] = project.bare_git
 
     has_dash_c = git_require((1, 7, 2))
@@ -592,6 +588,7 @@ later is required to fix a server side protocol bug.
 
     threads = set()
     sem = _threading.Semaphore(jobs)
+    err_event = _threading.Event()
 
     def GC(bare_git):
       try:
@@ -606,7 +603,7 @@ later is required to fix a server side protocol bug.
         sem.release()
 
     for bare_git in gc_gitdirs.values():
-      if err_event.isSet() and opt.fail_fast:
+      if err_event.isSet():
         break
       sem.acquire()
       t = _threading.Thread(target=GC, args=(bare_git,))
@@ -617,6 +614,10 @@ later is required to fix a server side protocol bug.
     for t in threads:
       t.join()
 
+    if err_event.isSet():
+      print('\nerror: Exited sync due to gc errors', file=sys.stderr)
+      sys.exit(1)
+
   def _ReloadManifest(self, manifest_name=None):
     if manifest_name:
       # Override calls _Unload already
@@ -897,8 +898,6 @@ later is required to fix a server side protocol bug.
           print('error: failed to remove existing smart sync override manifest: %s' %
                 e, file=sys.stderr)
 
-    err_event = _threading.Event()
-
     rp = self.manifest.repoProject
     rp.PreSync()
 
@@ -908,10 +907,7 @@ later is required to fix a server side protocol bug.
     if opt.repo_upgraded:
       _PostRepoUpgrade(self.manifest, quiet=opt.quiet)
 
-    if not opt.mp_update:
+    self._UpdateManifestProject(opt, mp, manifest_name)
-      print('Skipping update of local manifest project.')
-    else:
-      self._UpdateManifestProject(opt, mp, manifest_name)
 
     if self.gitc_manifest:
       gitc_manifest_projects = self.GetProjects(args,
@@ -952,10 +948,6 @@ later is required to fix a server side protocol bug.
                                     missing_ok=True,
                                     submodules_ok=opt.fetch_submodules)
 
-    err_network_sync = False
-    err_update_projects = False
-    err_checkout = False
-
     self._fetch_times = _FetchTimes(self.manifest)
     if not opt.local_only:
       to_fetch = []
@@ -965,14 +957,10 @@ later is required to fix a server side protocol bug.
       to_fetch.extend(all_projects)
       to_fetch.sort(key=self._fetch_times.Get, reverse=True)
 
-      fetched = self._Fetch(to_fetch, opt, err_event)
+      fetched = self._Fetch(to_fetch, opt)
-
       _PostRepoFetch(rp, opt.no_repo_verify)
       if opt.network_only:
         # bail out now; the rest touches the working tree
-        if err_event.isSet():
-          print('\nerror: Exited sync due to fetch errors.\n', file=sys.stderr)
-          sys.exit(1)
         return
 
       # Iteratively fetch missing and/or nested unregistered submodules
@@ -994,56 +982,22 @@ later is required to fix a server side protocol bug.
         if previously_missing_set == missing_set:
           break
         previously_missing_set = missing_set
-        fetched.update(self._Fetch(missing, opt, err_event))
+        fetched.update(self._Fetch(missing, opt))
-
-      # If we saw an error, exit with code 1 so that other scripts can check.
-      if err_event.isSet():
-        err_network_sync = True
-        if opt.fail_fast:
-          print('\nerror: Exited sync due to fetch errors.\n'
-                'Local checkouts *not* updated. Resolve network issues & '
-                'retry.\n'
-                '`repo sync -l` will update some local checkouts.',
-                file=sys.stderr)
-          sys.exit(1)
 
     if self.manifest.IsMirror or self.manifest.IsArchive:
       # bail out now, we have no working tree
       return
 
     if self.UpdateProjectList(opt):
-      err_event.set()
+      sys.exit(1)
-      err_update_projects = True
-      if opt.fail_fast:
-        print('\nerror: Local checkouts *not* updated.', file=sys.stderr)
-        sys.exit(1)
 
-    err_results = []
+    self._Checkout(all_projects, opt)
-    self._Checkout(all_projects, opt, err_event, err_results)
-    if err_event.isSet():
-      err_checkout = True
-      # NB: We don't exit here because this is the last step.
 
     # If there's a notice that's supposed to print at the end of the sync, print
     # it now...
     if self.manifest.notice:
       print(self.manifest.notice)
 
-    # If we saw an error, exit with code 1 so that other scripts can check.
-    if err_event.isSet():
-      print('\nerror: Unable to fully sync the tree.', file=sys.stderr)
-      if err_network_sync:
-        print('error: Downloading network changes failed.', file=sys.stderr)
-      if err_update_projects:
-        print('error: Updating local project lists failed.', file=sys.stderr)
-      if err_checkout:
-        print('error: Checking out local projects failed.', file=sys.stderr)
-        if err_results:
-          print('Failing repos:\n%s' % '\n'.join(err_results), file=sys.stderr)
-      print('Try re-running with "-j1 --fail-fast" to exit at the first error.',
-            file=sys.stderr)
-      sys.exit(1)
-
 def _PostRepoUpgrade(manifest, quiet=False):
   wrapper = Wrapper()
   if wrapper.NeedSetupGnuPG():

tests/test_git_command.py

@@ -35,7 +35,7 @@ class GitCallUnitTest(unittest.TestCase):
     # We don't dive too deep into the values here to avoid having to update
     # whenever git versions change.  We do check relative to this min version
     # as this is what `repo` itself requires via MIN_GIT_VERSION.
-    MIN_GIT_VERSION = (2, 10, 2)
+    MIN_GIT_VERSION = (1, 7, 2)
     self.assertTrue(isinstance(ver.major, int))
     self.assertTrue(isinstance(ver.minor, int))
     self.assertTrue(isinstance(ver.micro, int))

tests/test_manifest_xml.py

@@ -1,85 +0,0 @@
-# -*- coding:utf-8 -*-
-#
-# Copyright (C) 2019 The Android Open Source Project
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Unittests for the manifest_xml.py module."""
-
-from __future__ import print_function
-
-import unittest
-
-import error
-import manifest_xml
-
-
-class ManifestValidateFilePaths(unittest.TestCase):
-  """Check _ValidateFilePaths helper.
-
-  This doesn't access a real filesystem.
-  """
-
-  def check_both(self, *args):
-    manifest_xml.XmlManifest._ValidateFilePaths('copyfile', *args)
-    manifest_xml.XmlManifest._ValidateFilePaths('linkfile', *args)
-
-  def test_normal_path(self):
-    """Make sure good paths are accepted."""
-    self.check_both('foo', 'bar')
-    self.check_both('foo/bar', 'bar')
-    self.check_both('foo', 'bar/bar')
-    self.check_both('foo/bar', 'bar/bar')
-
-  def test_symlink_targets(self):
-    """Some extra checks for symlinks."""
-    def check(*args):
-      manifest_xml.XmlManifest._ValidateFilePaths('linkfile', *args)
-
-    # We allow symlinks to end in a slash since we allow them to point to dirs
-    # in general.  Technically the slash isn't necessary.
-    check('foo/', 'bar')
-    # We allow a single '.' to get a reference to the project itself.
-    check('.', 'bar')
-
-  def test_bad_paths(self):
-    """Make sure bad paths (src & dest) are rejected."""
-    PATHS = (
-        '..',
-        '../',
-        './',
-        'foo/',
-        './foo',
-        '../foo',
-        'foo/./bar',
-        'foo/../../bar',
-        '/foo',
-        './../foo',
-        '.git/foo',
-        # Check case folding.
-        '.GIT/foo',
-        'blah/.git/foo',
-        '.repo/foo',
-        '.repoconfig',
-        # Block ~ due to 8.3 filenames on Windows filesystems.
-        '~',
-        'foo~',
-        'blah/foo~',
-        # Block Unicode characters that get normalized out by filesystems.
-        u'foo\u200Cbar',
-    )
-    for path in PATHS:
-      self.assertRaises(
-          error.ManifestInvalidPathError, self.check_both, path, 'a')
-      self.assertRaises(
-          error.ManifestInvalidPathError, self.check_both, 'a', path)

tests/test_project.py

@@ -25,7 +25,6 @@ import subprocess
 import tempfile
 import unittest
 
-import error
 import git_config
 import project
 
@@ -135,229 +134,3 @@ class ReviewableBranchTests(unittest.TestCase):
       self.assertFalse(rb.base_exists)
       # Hard to assert anything useful about this.
       self.assertTrue(rb.date)
-
-
-class CopyLinkTestCase(unittest.TestCase):
-  """TestCase for stub repo client checkouts.
-
-  It'll have a layout like:
-    tempdir/          # self.tempdir
-      checkout/       # self.topdir
-        git-project/  # self.worktree
-
-  Attributes:
-    tempdir: A dedicated temporary directory.
-    worktree: The top of the repo client checkout.
-    topdir: The top of a project checkout.
-  """
-
-  def setUp(self):
-    self.tempdir = tempfile.mkdtemp(prefix='repo_tests')
-    self.topdir = os.path.join(self.tempdir, 'checkout')
-    self.worktree = os.path.join(self.topdir, 'git-project')
-    os.makedirs(self.topdir)
-    os.makedirs(self.worktree)
-
-  def tearDown(self):
-    shutil.rmtree(self.tempdir, ignore_errors=True)
-
-  @staticmethod
-  def touch(path):
-    with open(path, 'w') as f:
-      pass
-
-  def assertExists(self, path, msg=None):
-    """Make sure |path| exists."""
-    if os.path.exists(path):
-      return
-
-    if msg is None:
-      msg = ['path is missing: %s' % path]
-      while path != '/':
-        path = os.path.dirname(path)
-        if not path:
-          # If we're given something like "foo", abort once we get to "".
-          break
-        result = os.path.exists(path)
-        msg.append('\tos.path.exists(%s): %s' % (path, result))
-        if result:
-          msg.append('\tcontents: %r' % os.listdir(path))
-          break
-      msg = '\n'.join(msg)
-
-    raise self.failureException(msg)
-
-
-class CopyFile(CopyLinkTestCase):
-  """Check _CopyFile handling."""
-
-  def CopyFile(self, src, dest):
-    return project._CopyFile(self.worktree, src, self.topdir, dest)
-
-  def test_basic(self):
-    """Basic test of copying a file from a project to the toplevel."""
-    src = os.path.join(self.worktree, 'foo.txt')
-    self.touch(src)
-    cf = self.CopyFile('foo.txt', 'foo')
-    cf._Copy()
-    self.assertExists(os.path.join(self.topdir, 'foo'))
-
-  def test_src_subdir(self):
-    """Copy a file from a subdir of a project."""
-    src = os.path.join(self.worktree, 'bar', 'foo.txt')
-    os.makedirs(os.path.dirname(src))
-    self.touch(src)
-    cf = self.CopyFile('bar/foo.txt', 'new.txt')
-    cf._Copy()
-    self.assertExists(os.path.join(self.topdir, 'new.txt'))
-
-  def test_dest_subdir(self):
-    """Copy a file to a subdir of a checkout."""
-    src = os.path.join(self.worktree, 'foo.txt')
-    self.touch(src)
-    cf = self.CopyFile('foo.txt', 'sub/dir/new.txt')
-    self.assertFalse(os.path.exists(os.path.join(self.topdir, 'sub')))
-    cf._Copy()
-    self.assertExists(os.path.join(self.topdir, 'sub', 'dir', 'new.txt'))
-
-  def test_update(self):
-    """Make sure changed files get copied again."""
-    src = os.path.join(self.worktree, 'foo.txt')
-    dest = os.path.join(self.topdir, 'bar')
-    with open(src, 'w') as f:
-      f.write('1st')
-    cf = self.CopyFile('foo.txt', 'bar')
-    cf._Copy()
-    self.assertExists(dest)
-    with open(dest) as f:
-      self.assertEqual(f.read(), '1st')
-
-    with open(src, 'w') as f:
-      f.write('2nd!')
-    cf._Copy()
-    with open(dest) as f:
-      self.assertEqual(f.read(), '2nd!')
-
-  def test_src_block_symlink(self):
-    """Do not allow reading from a symlinked path."""
-    src = os.path.join(self.worktree, 'foo.txt')
-    sym = os.path.join(self.worktree, 'sym')
-    self.touch(src)
-    os.symlink('foo.txt', sym)
-    self.assertExists(sym)
-    cf = self.CopyFile('sym', 'foo')
-    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
-
-  def test_src_block_symlink_traversal(self):
-    """Do not allow reading through a symlink dir."""
-    src = os.path.join(self.worktree, 'bar', 'passwd')
-    os.symlink('/etc', os.path.join(self.worktree, 'bar'))
-    self.assertExists(src)
-    cf = self.CopyFile('bar/foo.txt', 'foo')
-    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
-
-  def test_src_block_dir(self):
-    """Do not allow copying from a directory."""
-    src = os.path.join(self.worktree, 'dir')
-    os.makedirs(src)
-    cf = self.CopyFile('dir', 'foo')
-    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
-
-  def test_dest_block_symlink(self):
-    """Do not allow writing to a symlink."""
-    src = os.path.join(self.worktree, 'foo.txt')
-    self.touch(src)
-    os.symlink('dest', os.path.join(self.topdir, 'sym'))
-    cf = self.CopyFile('foo.txt', 'sym')
-    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
-
-  def test_dest_block_symlink_traversal(self):
-    """Do not allow writing through a symlink dir."""
-    src = os.path.join(self.worktree, 'foo.txt')
-    self.touch(src)
-    os.symlink('/tmp', os.path.join(self.topdir, 'sym'))
-    cf = self.CopyFile('foo.txt', 'sym/foo.txt')
-    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
-
-  def test_src_block_dir(self):
-    """Do not allow copying to a directory."""
-    src = os.path.join(self.worktree, 'foo.txt')
-    self.touch(src)
-    os.makedirs(os.path.join(self.topdir, 'dir'))
-    cf = self.CopyFile('foo.txt', 'dir')
-    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
-
-
-class LinkFile(CopyLinkTestCase):
-  """Check _LinkFile handling."""
-
-  def LinkFile(self, src, dest):
-    return project._LinkFile(self.worktree, src, self.topdir, dest)
-
-  def test_basic(self):
-    """Basic test of linking a file from a project into the toplevel."""
-    src = os.path.join(self.worktree, 'foo.txt')
-    self.touch(src)
-    lf = self.LinkFile('foo.txt', 'foo')
-    lf._Link()
-    dest = os.path.join(self.topdir, 'foo')
-    self.assertExists(dest)
-    self.assertTrue(os.path.islink(dest))
-    self.assertEqual('git-project/foo.txt', os.readlink(dest))
-
-  def test_src_subdir(self):
-    """Link to a file in a subdir of a project."""
-    src = os.path.join(self.worktree, 'bar', 'foo.txt')
-    os.makedirs(os.path.dirname(src))
-    self.touch(src)
-    lf = self.LinkFile('bar/foo.txt', 'foo')
-    lf._Link()
-    self.assertExists(os.path.join(self.topdir, 'foo'))
-
-  def test_src_self(self):
-    """Link to the project itself."""
-    dest = os.path.join(self.topdir, 'foo', 'bar')
-    lf = self.LinkFile('.', 'foo/bar')
-    lf._Link()
-    self.assertExists(dest)
-    self.assertEqual('../git-project', os.readlink(dest))
-
-  def test_dest_subdir(self):
-    """Link a file to a subdir of a checkout."""
-    src = os.path.join(self.worktree, 'foo.txt')
-    self.touch(src)
-    lf = self.LinkFile('foo.txt', 'sub/dir/foo/bar')
-    self.assertFalse(os.path.exists(os.path.join(self.topdir, 'sub')))
-    lf._Link()
-    self.assertExists(os.path.join(self.topdir, 'sub', 'dir', 'foo', 'bar'))
-
-  def test_src_block_relative(self):
-    """Do not allow relative symlinks."""
-    BAD_SOURCES = (
-        './',
-        '..',
-        '../',
-        'foo/.',
-        'foo/./bar',
-        'foo/..',
-        'foo/../foo',
-    )
-    for src in BAD_SOURCES:
-      lf = self.LinkFile(src, 'foo')
-      self.assertRaises(error.ManifestInvalidPathError, lf._Link)
-
-  def test_update(self):
-    """Make sure changed targets get updated."""
-    dest = os.path.join(self.topdir, 'sym')
-
-    src = os.path.join(self.worktree, 'foo.txt')
-    self.touch(src)
-    lf = self.LinkFile('foo.txt', 'sym')
-    lf._Link()
-    self.assertEqual('git-project/foo.txt', os.readlink(dest))
-
-    # Point the symlink somewhere else.
-    os.unlink(dest)
-    os.symlink('/', dest)
-    lf._Link()
-    self.assertEqual('git-project/foo.txt', os.readlink(dest))