main: rework launcher version checking

The code has an ad-hoc check in that it requires the launcher major
version to not be less than the source code version.  We don't really
care about that requirement, and it doesn't fit with our other version
checks.  Rework it so we explicitly declare the min launcher version
that is supported.

We'll start with requiring repo launcher 1.15 which was released back
in 2012.  Hopefully no one has anything older than that, although it's
not clear we work with even newer versions than that :).  But let's be
a little conservative with the first update to this logic.

Bug: https://crbug.com/gerrit/10418
Change-Id: I611d70c60324d313c76874e978b8499a491a5d00
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254278
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>

docs/internal-fs-layout.md

@@ -0,0 +1,145 @@
+# Repo internal filesystem layout
+
+A reference to the `.repo/` tree in repo client checkouts.
+Hopefully it's complete & up-to-date, but who knows!
+
+*** note
+**Warning**:
+This is meant for developers of the repo project itself as a quick reference.
+**Nothing** in here must be construed as ABI, or that repo itself will never
+change its internals in backwards incompatible ways.
+***
+
+[TOC]
+
+## .repo/ layout
+
+All content under `.repo/` is managed by `repo` itself with few exceptions.
+
+In general, you should not make manual changes in here.
+If a setting was initialized using an option to `repo init`, you should use that
+command to change the setting later on.
+It is always safe to re-run `repo init` in existing repo client checkouts.
+For example, if you want to change the manifest branch, you can simply run
+`repo init --manifest-branch=<new name>` and repo will take care of the rest.
+
+### repo/ state
+
+*   `repo/`: A git checkout of the repo project.  This is how `repo` re-execs
+    itself to get the latest released version.
+
+    It tracks the git repository at `REPO_URL` using the `REPO_REV` branch.
+    Those are specified at `repo init` time using the `--repo-url=<REPO_URL>`
+    and `--repo-branch=<REPO_REV>` options.
+
+    Any changes made to this directory will usually be automatically discarded
+    by repo itself when it checks for updates.  If you want to update to the
+    latest version of repo, use `repo selfupdate` instead.  If you want to
+    change the git URL/branch that this tracks, re-run `repo init` with the new
+    settings.
+
+*   `.repo_fetchtimes.json`: Used by `repo sync` to record stats when syncing
+    the various projects.
+
+### Manifests
+
+For more documentation on the manifest format, including the local_manifests
+support, see the [manifest-format.md] file.
+
+*   `manifests/`: A git checkout of the manifest project.  Its `.git/` state
+    points to the `manifest.git` bare checkout (see below).  It tracks the git
+    branch specified at `repo init` time via `--manifest-branch`.
+
+    The local branch name is always `default` regardless of the remote tracking
+    branch.  Do not get confused if the remote branch is not `default`, or if
+    there is a remote `default` that is completely different!
+
+    No manual changes should be made in here as it will just confuse repo and
+    it won't automatically recover causing no new changes to be picked up.
+
+*   `manifests.git/`: A bare checkout of the manifest project.  It tracks the
+    git repository specified at `repo init` time via `--manifest-url`.
+
+    No manual changes should be made in here as it will just confuse repo.
+    If you want to switch the tracking settings, re-run `repo init` with the
+    new settings.
+
+*   `manifest.xml -> manifests/<manifest-name>.xml`: A symlink to the manifest
+    that the user wishes to sync.  It is specified at `repo init` time via
+    `--manifest-name`.
+
+    Do not try to repoint this symlink to other files as it will confuse repo.
+    If you want to switch manifest files, re-run `repo init` with the new
+    setting.
+
+*   `manifests.git/.repo_config.json`: JSON cache of the `manifests.git/config`
+    file for repo to read/process quickly.
+
+*   `local_manifest.xml` (*Deprecated*): User-authored tweaks to the manifest
+    used to sync.  See [local manifests] for more details.
+*   `local_manifests/`: Directory of user-authored manifest fragments to tweak
+    the manifest used to sync.  See [local manifests] for more details.
+
+### Project objects
+
+*   `project.list`: Tracking file used by `repo sync` to determine when projects
+    are added or removed and need corresponding updates in the checkout.
+*   `projects/`: Bare checkouts of every project synced by the manifest.  The
+    filesystem layout matches the `<project path=...` setting in the manifest
+    (i.e. where it's checked out in the repo client source tree).  Those
+    checkouts will symlink their `.git/` state to paths under here.
+
+    Some git state is further split out under `project-objects/`.
+*   `project-objects/`: Git objects that are safe to share across multiple
+    git checkouts.  The filesystem layout matches the `<project name=...`
+    setting in the manifest (i.e. the path on the remote server).  This allows
+    for multiple checkouts of the same remote git repo to share their objects.
+    For example, you could have different branches of `foo/bar.git` checked
+    out to `foo/bar-master`, `foo/bar-release`, etc...  There will be multiple
+    trees under `projects/` for each one, but only one under `project-objects/`.
+
+    This can run into problems if different remotes use the same path on their
+    respective servers ...
+*   `subprojects/`: Like `projects/`, but for git submodules.
+*   `subproject-objects/`: Like `project-objects/`, but for git submodules.
+
+### Settings
+
+The `.repo/manifests.git/config` file is used to track settings for the entire
+repo client checkout.
+Most settings use the `[repo]` section to avoid conflicts with git.
+User controlled settings are initialized when running `repo init`.
+
+| Setting           | `repo init` Option        | Use/Meaning |
+|-------------------|---------------------------|-------------|
+| manifest.groups   | `--groups` & `--platform` | The manifest groups to sync |
+| repo.archive      | `--archive`               | Use `git archive` for checkouts |
+| repo.clonefilter  | `--clone-filter`          | Filter setting when using [partial git clones] |
+| repo.depth        | `--depth`                 | Create shallow checkouts when cloning |
+| repo.dissociate   | `--dissociate`            | Dissociate from any reference/mirrors after initial clone |
+| repo.mirror       | `--mirror`                | Checkout is a repo mirror |
+| repo.partialclone | `--partial-clone`         | Create [partial git clones] |
+| repo.reference    | `--reference`             | Reference repo client checkout |
+| repo.submodules   | `--submodules`            | Sync git submodules |
+| user.email        | `--config-name`           | User's e-mail address; Copied into `.git/config` when checking out a new project |
+| user.name         | `--config-name`           | User's name; Copied into `.git/config` when checking out a new project |
+
+[partial git clones]: https://git-scm.com/docs/gitrepository-layout#_code_partialclone_code
+
+## ~/ dotconfig layout
+
+Repo will create & maintain a few files in the user's home directory.
+
+*   `.repoconfig/`: Repo's per-user directory for all random config files/state.
+*   `.repoconfig/keyring-version`: Cache file for checking if the gnupg subdir
+    has all the same keys as the repo launcher.  Used to avoid running gpg
+    constantly as that can be quite slow.
+*   `.repoconfig/gnupg/`: GnuPG's internal state directory used when repo needs
+    to run `gpg`.  This provides isolation from the user's normal `~/.gnupg/`.
+
+*   `.repo_.gitconfig.json`: JSON cache of the `.gitconfig` file for repo to
+    read/process quickly.
+
+
+[manifest-format.md]: ./manifest-format.md
+[local manifests]: ./manifest-format.md#Local-Manifests

docs/manifest-format.md

@@ -89,6 +89,7 @@ following DTD:
   <!ATTLIST extend-project path CDATA #IMPLIED>
   <!ATTLIST extend-project groups CDATA #IMPLIED>
   <!ATTLIST extend-project revision CDATA #IMPLIED>
+  <!ATTLIST extend-project remote CDATA #IMPLIED>
 
   <!ELEMENT remove-project EMPTY>
   <!ATTLIST remove-project name  CDATA #REQUIRED>
@@ -306,6 +307,9 @@ belongs.  Same syntax as the corresponding element of `project`.
 Attribute `revision`: If specified, overrides the revision of the original
 project.  Same syntax as the corresponding element of `project`.
 
+Attribute `remote`: If specified, overrides the remote of the original
+project.  Same syntax as the corresponding element of `project`.
+
 ### Element annotation
 
 Zero or more annotation elements may be specified as children of a
@@ -338,7 +342,7 @@ It's just like copyfile and runs at the same time as copyfile but
 instead of copying it creates a symlink.
 
 The symlink is created at "dest" (relative to the top of the tree) and
-points to the path specified by "src".
+points to the path specified by "src" which is a path in the project.
 
 Parent directories of "dest" will be automatically created if missing.
 

error.py

@@ -22,6 +22,10 @@ class ManifestInvalidRevisionError(Exception):
   """The revision value in a project is incorrect.
   """
 
+class ManifestInvalidPathError(Exception):
+  """A path used in <copyfile> or <linkfile> is incorrect.
+  """
+
 class NoManifestException(Exception):
   """The required manifest does not exist.
   """

git_command.py

@@ -28,7 +28,8 @@ from repo_trace import REPO_TRACE, IsTrace, Trace
 from wrapper import Wrapper
 
 GIT = 'git'
-MIN_GIT_VERSION = (1, 5, 4)
+# Should keep in sync with the "repo" launcher file.
+MIN_GIT_VERSION = (2, 10, 2)
 GIT_DIR = 'GIT_DIR'
 
 LAST_GITDIR = None

git_config.py

@@ -528,7 +528,7 @@ def GetUrlCookieFile(url, quiet):
         cookiefile = None
         proxy = None
         for line in p.stdout:
-          line = line.strip()
+          line = line.strip().decode('utf-8')
           if line.startswith(cookieprefix):
             cookiefile = os.path.expanduser(line[len(cookieprefix):])
           if line.startswith(proxyprefix):
@@ -540,7 +540,7 @@ def GetUrlCookieFile(url, quiet):
       finally:
         p.stdin.close()
         if p.wait():
-          err_msg = p.stderr.read()
+          err_msg = p.stderr.read().decode('utf-8')
           if ' -print_config' in err_msg:
             pass  # Persistent proxy doesn't support -print_config.
           elif not quiet:

main.py

@@ -255,27 +255,41 @@ class _Repo(object):
     return result
 
 
-def _CheckWrapperVersion(ver, repo_path):
+def _CheckWrapperVersion(ver_str, repo_path):
+  """Verify the repo launcher is new enough for this checkout.
+
+  Args:
+    ver_str: The version string passed from the repo launcher when it ran us.
+    repo_path: The path to the repo launcher that loaded us.
+  """
+  # Refuse to work with really old wrapper versions.  We don't test these,
+  # so might as well require a somewhat recent sane version.
+  # v1.15 of the repo launcher was released in ~Mar 2012.
+  MIN_REPO_VERSION = (1, 15)
+  min_str = '.'.join(str(x) for x in MIN_REPO_VERSION)
+
   if not repo_path:
     repo_path = '~/bin/repo'
 
-  if not ver:
+  if not ver_str:
     print('no --wrapper-version argument', file=sys.stderr)
     sys.exit(1)
 
+  # Pull out the version of the repo launcher we know about to compare.
   exp = Wrapper().VERSION
-  ver = tuple(map(int, ver.split('.')))
-  if len(ver) == 1:
-    ver = (0, ver[0])
+  ver = tuple(map(int, ver_str.split('.')))
 
   exp_str = '.'.join(map(str, exp))
-  if exp[0] > ver[0] or ver < (0, 4):
+  if ver < MIN_REPO_VERSION:
     print("""
-!!! A new repo command (%5s) is available.    !!!
-!!! You must upgrade before you can continue:   !!!
+repo: error:
+!!! Your version of repo %s is too old.
+!!! We need at least version %s.
+!!! A new repo command (%s) is available.
+!!! You must upgrade before you can continue:
 
     cp %s %s
-""" % (exp_str, WrapperPath(), repo_path), file=sys.stderr)
+""" % (ver_str, min_str, exp_str, WrapperPath(), repo_path), file=sys.stderr)
     sys.exit(1)
 
   if exp > ver:

manifest_xml.py

@@ -35,7 +35,8 @@ from git_config import GitConfig
 from git_refs import R_HEADS, HEAD
 import platform_utils
 from project import RemoteSpec, Project, MetaProject
-from error import ManifestParseError, ManifestInvalidRevisionError
+from error import (ManifestParseError, ManifestInvalidPathError,
+                   ManifestInvalidRevisionError)
 
 MANIFEST_FILE_NAME = 'manifest.xml'
 LOCAL_MANIFEST_NAME = 'local_manifest.xml'
@@ -598,6 +599,9 @@ class XmlManifest(object):
         if groups:
           groups = self._ParseGroups(groups)
         revision = node.getAttribute('revision')
+        remote = node.getAttribute('remote')
+        if remote:
+          remote = self._get_remote(node)
 
         for p in self._projects[name]:
           if path and p.relpath != path:
@@ -606,6 +610,8 @@ class XmlManifest(object):
             p.groups.extend(groups)
           if revision:
             p.revisionExpr = revision
+          if remote:
+            p.remote = remote.ToRemoteSpec(name)
       if node.nodeName == 'repo-hooks':
         # Get the name of the project and the (space-separated) list of enabled.
         repo_hooks_project = self._reqatt(node, 'in-project')
@@ -943,21 +949,101 @@ class XmlManifest(object):
       worktree = os.path.join(parent.worktree, path).replace('\\', '/')
     return relpath, worktree, gitdir, objdir
 
+  @staticmethod
+  def _CheckLocalPath(path, symlink=False):
+    """Verify |path| is reasonable for use in <copyfile> & <linkfile>."""
+    if '~' in path:
+      return '~ not allowed (due to 8.3 filenames on Windows filesystems)'
+
+    # Some filesystems (like Apple's HFS+) try to normalize Unicode codepoints
+    # which means there are alternative names for ".git".  Reject paths with
+    # these in it as there shouldn't be any reasonable need for them here.
+    # The set of codepoints here was cribbed from jgit's implementation:
+    # https://eclipse.googlesource.com/jgit/jgit/+/9110037e3e9461ff4dac22fee84ef3694ed57648/org.eclipse.jgit/src/org/eclipse/jgit/lib/ObjectChecker.java#884
+    BAD_CODEPOINTS = {
+        u'\u200C',  # ZERO WIDTH NON-JOINER
+        u'\u200D',  # ZERO WIDTH JOINER
+        u'\u200E',  # LEFT-TO-RIGHT MARK
+        u'\u200F',  # RIGHT-TO-LEFT MARK
+        u'\u202A',  # LEFT-TO-RIGHT EMBEDDING
+        u'\u202B',  # RIGHT-TO-LEFT EMBEDDING
+        u'\u202C',  # POP DIRECTIONAL FORMATTING
+        u'\u202D',  # LEFT-TO-RIGHT OVERRIDE
+        u'\u202E',  # RIGHT-TO-LEFT OVERRIDE
+        u'\u206A',  # INHIBIT SYMMETRIC SWAPPING
+        u'\u206B',  # ACTIVATE SYMMETRIC SWAPPING
+        u'\u206C',  # INHIBIT ARABIC FORM SHAPING
+        u'\u206D',  # ACTIVATE ARABIC FORM SHAPING
+        u'\u206E',  # NATIONAL DIGIT SHAPES
+        u'\u206F',  # NOMINAL DIGIT SHAPES
+        u'\uFEFF',  # ZERO WIDTH NO-BREAK SPACE
+    }
+    if BAD_CODEPOINTS & set(path):
+      # This message is more expansive than reality, but should be fine.
+      return 'Unicode combining characters not allowed'
+
+    # Assume paths might be used on case-insensitive filesystems.
+    path = path.lower()
+
+    # Some people use src="." to create stable links to projects.  Lets allow
+    # that but reject all other uses of "." to keep things simple.
+    parts = path.split(os.path.sep)
+    if parts != ['.']:
+      for part in set(parts):
+        if part in {'.', '..', '.git'} or part.startswith('.repo'):
+          return 'bad component: %s' % (part,)
+
+    if not symlink and path.endswith(os.path.sep):
+      return 'dirs not allowed'
+
+    norm = os.path.normpath(path)
+    if norm == '..' or norm.startswith('../') or norm.startswith(os.path.sep):
+      return 'path cannot be outside'
+
+  @classmethod
+  def _ValidateFilePaths(cls, element, src, dest):
+    """Verify |src| & |dest| are reasonable for <copyfile> & <linkfile>.
+
+    We verify the path independent of any filesystem state as we won't have a
+    checkout available to compare to.  i.e. This is for parsing validation
+    purposes only.
+
+    We'll do full/live sanity checking before we do the actual filesystem
+    modifications in _CopyFile/_LinkFile/etc...
+    """
+    # |dest| is the file we write to or symlink we create.
+    # It is relative to the top of the repo client checkout.
+    msg = cls._CheckLocalPath(dest)
+    if msg:
+      raise ManifestInvalidPathError(
+          '<%s> invalid "dest": %s: %s' % (element, dest, msg))
+
+    # |src| is the file we read from or path we point to for symlinks.
+    # It is relative to the top of the git project checkout.
+    msg = cls._CheckLocalPath(src, symlink=element == 'linkfile')
+    if msg:
+      raise ManifestInvalidPathError(
+          '<%s> invalid "src": %s: %s' % (element, src, msg))
+
   def _ParseCopyFile(self, project, node):
     src = self._reqatt(node, 'src')
     dest = self._reqatt(node, 'dest')
     if not self.IsMirror:
       # src is project relative;
-      # dest is relative to the top of the tree
-      project.AddCopyFile(src, dest, os.path.join(self.topdir, dest))
+      # dest is relative to the top of the tree.
+      # We only validate paths if we actually plan to process them.
+      self._ValidateFilePaths('copyfile', src, dest)
+      project.AddCopyFile(src, dest, self.topdir)
 
   def _ParseLinkFile(self, project, node):
     src = self._reqatt(node, 'src')
     dest = self._reqatt(node, 'dest')
     if not self.IsMirror:
       # src is project relative;
-      # dest is relative to the top of the tree
-      project.AddLinkFile(src, dest, os.path.join(self.topdir, dest))
+      # dest is relative to the top of the tree.
+      # We only validate paths if we actually plan to process them.
+      self._ValidateFilePaths('linkfile', src, dest)
+      project.AddLinkFile(src, dest, self.topdir)
 
   def _ParseAnnotation(self, project, node):
     name = self._reqatt(node, 'name')

platform_utils_win32.py

@@ -19,13 +19,18 @@ import errno
 from pyversion import is_python3
 from ctypes import WinDLL, get_last_error, FormatError, WinError, addressof
 from ctypes import c_buffer
-from ctypes.wintypes import BOOL, BOOLEAN, LPCWSTR, DWORD, HANDLE, POINTER, c_ubyte
-from ctypes.wintypes import WCHAR, USHORT, LPVOID, Structure, Union, ULONG
-from ctypes.wintypes import byref
+from ctypes.wintypes import BOOL, BOOLEAN, LPCWSTR, DWORD, HANDLE
+from ctypes.wintypes import WCHAR, USHORT, LPVOID, ULONG
+if is_python3():
+  from ctypes import c_ubyte, Structure, Union, byref
+  from ctypes.wintypes import LPDWORD
+else:
+  # For legacy Python2 different imports are needed.
+  from ctypes.wintypes import POINTER, c_ubyte, Structure, Union, byref
+  LPDWORD = POINTER(DWORD)
 
 kernel32 = WinDLL('kernel32', use_last_error=True)
 
-LPDWORD = POINTER(DWORD)
 UCHAR = c_ubyte
 
 # Win32 error codes

project.py

@@ -36,7 +36,7 @@ from git_command import GitCommand, git_require
 from git_config import GitConfig, IsId, GetSchemeFromUrl, GetUrlCookieFile, \
     ID_RE
 from error import GitError, HookError, UploadError, DownloadError
-from error import ManifestInvalidRevisionError
+from error import ManifestInvalidRevisionError, ManifestInvalidPathError
 from error import NoManifestException
 import platform_utils
 import progress
@@ -261,17 +261,70 @@ class _Annotation(object):
     self.keep = keep
 
 
-class _CopyFile(object):
+def _SafeExpandPath(base, subpath, skipfinal=False):
+  """Make sure |subpath| is completely safe under |base|.
 
-  def __init__(self, src, dest, abssrc, absdest):
+  We make sure no intermediate symlinks are traversed, and that the final path
+  is not a special file (e.g. not a socket or fifo).
+
+  NB: We rely on a number of paths already being filtered out while parsing the
+  manifest.  See the validation logic in manifest_xml.py for more details.
+  """
+  components = subpath.split(os.path.sep)
+  if skipfinal:
+    # Whether the caller handles the final component itself.
+    finalpart = components.pop()
+
+  path = base
+  for part in components:
+    if part in {'.', '..'}:
+      raise ManifestInvalidPathError(
+          '%s: "%s" not allowed in paths' % (subpath, part))
+
+    path = os.path.join(path, part)
+    if platform_utils.islink(path):
+      raise ManifestInvalidPathError(
+          '%s: traversing symlinks not allow' % (path,))
+
+    if os.path.exists(path):
+      if not os.path.isfile(path) and not platform_utils.isdir(path):
+        raise ManifestInvalidPathError(
+            '%s: only regular files & directories allowed' % (path,))
+
+  if skipfinal:
+    path = os.path.join(path, finalpart)
+
+  return path
+
+
+class _CopyFile(object):
+  """Container for <copyfile> manifest element."""
+
+  def __init__(self, git_worktree, src, topdir, dest):
+    """Register a <copyfile> request.
+
+    Args:
+      git_worktree: Absolute path to the git project checkout.
+      src: Relative path under |git_worktree| of file to read.
+      topdir: Absolute path to the top of the repo client checkout.
+      dest: Relative path under |topdir| of file to write.
+    """
+    self.git_worktree = git_worktree
+    self.topdir = topdir
     self.src = src
     self.dest = dest
-    self.abs_src = abssrc
-    self.abs_dest = absdest
 
   def _Copy(self):
-    src = self.abs_src
-    dest = self.abs_dest
+    src = _SafeExpandPath(self.git_worktree, self.src)
+    dest = _SafeExpandPath(self.topdir, self.dest)
+
+    if platform_utils.isdir(src):
+      raise ManifestInvalidPathError(
+          '%s: copying from directory not supported' % (self.src,))
+    if platform_utils.isdir(dest):
+      raise ManifestInvalidPathError(
+          '%s: copying to directory not allowed' % (self.dest,))
+
     # copy file if it does not exist or is out of date
     if not os.path.exists(dest) or not filecmp.cmp(src, dest):
       try:
@@ -292,13 +345,21 @@ class _CopyFile(object):
 
 
 class _LinkFile(object):
+  """Container for <linkfile> manifest element."""
 
-  def __init__(self, git_worktree, src, dest, relsrc, absdest):
+  def __init__(self, git_worktree, src, topdir, dest):
+    """Register a <linkfile> request.
+
+    Args:
+      git_worktree: Absolute path to the git project checkout.
+      src: Target of symlink relative to path under |git_worktree|.
+      topdir: Absolute path to the top of the repo client checkout.
+      dest: Relative path under |topdir| of symlink to create.
+    """
     self.git_worktree = git_worktree
+    self.topdir = topdir
     self.src = src
     self.dest = dest
-    self.src_rel_to_dest = relsrc
-    self.abs_dest = absdest
 
   def __linkIt(self, relSrc, absDest):
     # link file if it does not exist or is out of date
@@ -316,35 +377,37 @@ class _LinkFile(object):
         _error('Cannot link file %s to %s', relSrc, absDest)
 
   def _Link(self):
-    """Link the self.rel_src_to_dest and self.abs_dest. Handles wild cards
-    on the src linking all of the files in the source in to the destination
-    directory.
+    """Link the self.src & self.dest paths.
+
+    Handles wild cards on the src linking all of the files in the source in to
+    the destination directory.
     """
-    # We use the absSrc to handle the situation where the current directory
-    # is not the root of the repo
-    absSrc = os.path.join(self.git_worktree, self.src)
-    if os.path.exists(absSrc):
-      # Entity exists so just a simple one to one link operation
-      self.__linkIt(self.src_rel_to_dest, self.abs_dest)
+    src = _SafeExpandPath(self.git_worktree, self.src)
+
+    if os.path.exists(src):
+      # Entity exists so just a simple one to one link operation.
+      dest = _SafeExpandPath(self.topdir, self.dest, skipfinal=True)
+      # dest & src are absolute paths at this point.  Make sure the target of
+      # the symlink is relative in the context of the repo client checkout.
+      relpath = os.path.relpath(src, os.path.dirname(dest))
+      self.__linkIt(relpath, dest)
     else:
+      dest = _SafeExpandPath(self.topdir, self.dest)
       # Entity doesn't exist assume there is a wild card
-      absDestDir = self.abs_dest
-      if os.path.exists(absDestDir) and not platform_utils.isdir(absDestDir):
-        _error('Link error: src with wildcard, %s must be a directory',
-               absDestDir)
+      if os.path.exists(dest) and not platform_utils.isdir(dest):
+        _error('Link error: src with wildcard, %s must be a directory', dest)
       else:
-        absSrcFiles = glob.glob(absSrc)
-        for absSrcFile in absSrcFiles:
+        for absSrcFile in glob.glob(src):
           # Create a releative path from source dir to destination dir
           absSrcDir = os.path.dirname(absSrcFile)
-          relSrcDir = os.path.relpath(absSrcDir, absDestDir)
+          relSrcDir = os.path.relpath(absSrcDir, dest)
 
           # Get the source file name
           srcFile = os.path.basename(absSrcFile)
 
           # Now form the final full paths to srcFile. They will be
           # absolute for the desintaiton and relative for the srouce.
-          absDest = os.path.join(absDestDir, srcFile)
+          absDest = os.path.join(dest, srcFile)
           relSrc = os.path.join(relSrcDir, srcFile)
           self.__linkIt(relSrc, absDest)
 
@@ -1712,18 +1775,25 @@ class Project(object):
       if submodules:
         syncbuf.later1(self, _dosubmodules)
 
-  def AddCopyFile(self, src, dest, absdest):
-    # dest should already be an absolute path, but src is project relative
-    # make src an absolute path
-    abssrc = os.path.join(self.worktree, src)
-    self.copyfiles.append(_CopyFile(src, dest, abssrc, absdest))
+  def AddCopyFile(self, src, dest, topdir):
+    """Mark |src| for copying to |dest| (relative to |topdir|).
 
-  def AddLinkFile(self, src, dest, absdest):
-    # dest should already be an absolute path, but src is project relative
-    # make src relative path to dest
-    absdestdir = os.path.dirname(absdest)
-    relsrc = os.path.relpath(os.path.join(self.worktree, src), absdestdir)
-    self.linkfiles.append(_LinkFile(self.worktree, src, dest, relsrc, absdest))
+    No filesystem changes occur here.  Actual copying happens later on.
+
+    Paths should have basic validation run on them before being queued.
+    Further checking will be handled when the actual copy happens.
+    """
+    self.copyfiles.append(_CopyFile(self.worktree, src, topdir, dest))
+
+  def AddLinkFile(self, src, dest, topdir):
+    """Mark |dest| to create a symlink (relative to |topdir|) pointing to |src|.
+
+    No filesystem changes occur here.  Actual linking happens later on.
+
+    Paths should have basic validation run on them before being queued.
+    Further checking will be handled when the actual link happens.
+    """
+    self.linkfiles.append(_LinkFile(self.worktree, src, topdir, dest))
 
   def AddAnnotation(self, name, value, keep):
     self.annotations.append(_Annotation(name, value, keep))
@@ -1747,6 +1817,18 @@ class Project(object):
 
 # Branch Management ##
 
+  def GetHeadPath(self):
+    """Return the full path to the HEAD ref."""
+    dotgit = os.path.join(self.worktree, '.git')
+    if os.path.isfile(dotgit):
+      # Git worktrees use a "gitdir:" syntax to point to the scratch space.
+      with open(dotgit) as fp:
+        setting = fp.read()
+        assert setting.startswith('gitdir:')
+        gitdir = setting.split(':', 1)[1].strip()
+      dotgit = os.path.join(self.worktree, gitdir)
+    return os.path.join(dotgit, HEAD)
+
   def StartBranch(self, name, branch_merge='', revision=None):
     """Create a new branch off the manifest's revision.
     """
@@ -1786,8 +1868,7 @@ class Project(object):
       except OSError:
         pass
       _lwrite(ref, '%s\n' % revid)
-      _lwrite(os.path.join(self.worktree, '.git', HEAD),
-              'ref: %s%s\n' % (R_HEADS, name))
+      _lwrite(self.GetHeadPath(), 'ref: %s%s\n' % (R_HEADS, name))
       branch.Save()
       return True
 
@@ -1834,8 +1915,7 @@ class Project(object):
       # Same revision; just update HEAD to point to the new
       # target branch, but otherwise take no other action.
       #
-      _lwrite(os.path.join(self.worktree, '.git', HEAD),
-              'ref: %s%s\n' % (R_HEADS, name))
+      _lwrite(self.GetHeadPath(), 'ref: %s%s\n' % (R_HEADS, name))
       return True
 
     return GitCommand(self,
@@ -1868,8 +1948,7 @@ class Project(object):
 
       revid = self.GetRevisionId(all_refs)
       if head == revid:
-        _lwrite(os.path.join(self.worktree, '.git', HEAD),
-                '%s\n' % revid)
+        _lwrite(self.GetHeadPath(), '%s\n' % revid)
       else:
         self._Checkout(revid, quiet=True)
 
@@ -2252,7 +2331,10 @@ class Project(object):
       spec.append('tag')
       spec.append(tag_name)
 
-    branch = self.revisionExpr
+    if self.manifest.IsMirror and not current_branch_only:
+      branch = None
+    else:
+      branch = self.revisionExpr
     if (not self.manifest.IsMirror and is_sha1 and depth
         and git_require((1, 8, 3))):
       # Shallow checkout of a specific commit, fetch from that commit and not
@@ -2929,7 +3011,7 @@ class Project(object):
       if self._bare:
         path = os.path.join(self._project.gitdir, HEAD)
       else:
-        path = os.path.join(self._project.worktree, '.git', HEAD)
+        path = self._project.GetHeadPath()
       try:
         with open(path) as fd:
           line = fd.readline()

repo

@@ -10,13 +10,93 @@ copy of repo in the checkout.
 
 from __future__ import print_function
 
+import os
+import platform
+import subprocess
+import sys
+
+
+def exec_command(cmd):
+  """Execute |cmd| or return None on failure."""
+  try:
+    if platform.system() == 'Windows':
+      ret = subprocess.call(cmd)
+      sys.exit(ret)
+    else:
+      os.execvp(cmd[0], cmd)
+  except:
+    pass
+
+
+def check_python_version():
+  """Make sure the active Python version is recent enough."""
+  def reexec(prog):
+    exec_command([prog] + sys.argv)
+
+  MIN_PYTHON_VERSION = (3, 6)
+
+  ver = sys.version_info
+  major = ver.major
+  minor = ver.minor
+
+  # Abort on very old Python 2 versions.
+  if (major, minor) < (2, 7):
+    print('repo: error: Your Python version is too old. '
+          'Please use Python {}.{} or newer instead.'.format(
+              *MIN_PYTHON_VERSION), file=sys.stderr)
+    sys.exit(1)
+
+  # Try to re-exec the version specific Python 3 if needed.
+  if (major, minor) < MIN_PYTHON_VERSION:
+    # Python makes releases ~once a year, so try our min version +10 to help
+    # bridge the gap.  This is the fallback anyways so perf isn't critical.
+    min_major, min_minor = MIN_PYTHON_VERSION
+    for inc in range(0, 10):
+      reexec('python{}.{}'.format(min_major, min_minor + inc))
+
+    # Try the generic Python 3 wrapper, but only if it's new enough.  We don't
+    # want to go from (still supported) Python 2.7 to (unsupported) Python 3.5.
+    try:
+      proc = subprocess.Popen(
+          ['python3', '-c', 'import sys; '
+           'print(sys.version_info.major, sys.version_info.minor)'],
+          stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+      (output, _) = proc.communicate()
+      python3_ver = tuple(int(x) for x in output.decode('utf-8').split())
+    except (OSError, subprocess.CalledProcessError):
+      python3_ver = None
+
+    # The python3 version looks like it's new enough, so give it a try.
+    if python3_ver and python3_ver >= MIN_PYTHON_VERSION:
+      reexec('python3')
+
+    # We're still here, so diagnose things for the user.
+    if major < 3:
+      print('repo: warning: Python 2 is no longer supported; '
+            'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION),
+            file=sys.stderr)
+    else:
+      print('repo: error: Python 3 version is too old; '
+            'Please use Python {}.{} or newer.'.format(*MIN_PYTHON_VERSION),
+            file=sys.stderr)
+      sys.exit(1)
+
+
+if __name__ == '__main__':
+  # TODO(vapier): Enable this on Windows once we have Python 3 issues fixed.
+  if platform.system() != 'Windows':
+    check_python_version()
+
+
 # repo default configuration
 #
 import os
 REPO_URL = os.environ.get('REPO_URL', None)
 if not REPO_URL:
   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
-REPO_REV = 'stable'
+REPO_REV = os.environ.get('REPO_REV')
+if not REPO_REV:
+  REPO_REV = 'stable'
 
 # Copyright (C) 2008 Google Inc.
 #
@@ -33,10 +113,10 @@ REPO_REV = 'stable'
 # limitations under the License.
 
 # increment this whenever we make important changes to this script
-VERSION = (1, 26)
+VERSION = (2, 0)
 
 # increment this if the MAINTAINER_KEYS block is modified
-KEYRING_VERSION = (1, 2)
+KEYRING_VERSION = (2, 0)
 
 # Each individual key entry is created by using:
 # gpg --armor --export keyid
@@ -82,48 +162,15 @@ HTHs37+/QLMomGEGKZMWi0dShU2J5mNRQu3Hhxl3hHDVbt5CeJBb26aQcQrFz69W
 zE3GNvmJosh6leayjtI9P2A6iEkEGBECAAkFAkj3uiACGwwACgkQFlMNXpIPXGWp
 TACbBS+Up3RpfYVfd63c1cDdlru13pQAn3NQy/SN858MkxN+zym86UBgOad2
 =CMiZ
------END PGP PUBLIC KEY BLOCK-----
-
-     Conley Owens <cco3@android.com>
------BEGIN PGP PUBLIC KEY BLOCK-----
-Version: GnuPG v1.4.11 (GNU/Linux)
-
-mQENBFHRvc8BCADFg45Xx/y6QDC+T7Y/gGc7vx0ww7qfOwIKlAZ9xG3qKunMxo+S
-hPCnzEl3cq+6I1Ww/ndop/HB3N3toPXRCoN8Vs4/Hc7by+SnaLFnacrm+tV5/OgT
-V37Lzt8lhay1Kl+YfpFwHYYpIEBLFV9knyfRXS/428W2qhdzYfvB15/AasRmwmor
-py4NIzSs8UD/SPr1ihqNCdZM76+MQyN5HMYXW/ALZXUFG0pwluHFA7hrfPG74i8C
-zMiP7qvMWIl/r/jtzHioH1dRKgbod+LZsrDJ8mBaqsZaDmNJMhss9g76XvfMyLra
-9DI9/iFuBpGzeqBv0hwOGQspLRrEoyTeR6n1ABEBAAG0H0NvbmxleSBPd2VucyA8
-Y2NvM0BhbmRyb2lkLmNvbT6JATgEEwECACIFAlHRvc8CGwMGCwkIBwMCBhUIAgkK
-CwQWAgMBAh4BAheAAAoJEGe35EhpKzgsP6AIAJKJmNtn4l7hkYHKHFSo3egb6RjQ
-zEIP3MFTcu8HFX1kF1ZFbrp7xqurLaE53kEkKuAAvjJDAgI8mcZHP1JyplubqjQA
-xvv84gK+OGP3Xk+QK1ZjUQSbjOpjEiSZpRhWcHci3dgOUH4blJfByHw25hlgHowd
-a/2PrNKZVcJ92YienaxxGjcXEUcd0uYEG2+rwllQigFcnMFDhr9B71MfalRHjFKE
-fmdoypqLrri61YBc59P88Rw2/WUpTQjgNubSqa3A2+CKdaRyaRw+2fdF4TdR0h8W
-zbg+lbaPtJHsV+3mJC7fq26MiJDRJa5ZztpMn8su20gbLgi2ShBOaHAYDDi5AQ0E
-UdG9zwEIAMoOBq+QLNozAhxOOl5GL3StTStGRgPRXINfmViTsihrqGCWBBUfXlUE
-OytC0mYcrDUQev/8ToVoyqw+iGSwDkcSXkrEUCKFtHV/GECWtk1keyHgR10YKI1R
-mquSXoubWGqPeG1PAI74XWaRx8UrL8uCXUtmD8Q5J7mDjKR5NpxaXrwlA0bKsf2E
-Gp9tu1kKauuToZhWHMRMqYSOGikQJwWSFYKT1KdNcOXLQF6+bfoJ6sjVYdwfmNQL
-Ixn8QVhoTDedcqClSWB17VDEFDFa7MmqXZz2qtM3X1R/MUMHqPtegQzBGNhRdnI2
-V45+1Nnx/uuCxDbeI4RbHzujnxDiq70AEQEAAYkBHwQYAQIACQUCUdG9zwIbDAAK
-CRBnt+RIaSs4LNVeB/0Y2pZ8I7gAAcEM0Xw8drr4omg2fUoK1J33ozlA/RxeA/lJ
-I3KnyCDTpXuIeBKPGkdL8uMATC9Z8DnBBajRlftNDVZS3Hz4G09G9QpMojvJkFJV
-By+01Flw/X+eeN8NpqSuLV4W+AjEO8at/VvgKr1AFvBRdZ7GkpI1o6DgPe7ZqX+1
-dzQZt3e13W0rVBb/bUgx9iSLoeWP3aq/k+/GRGOR+S6F6BBSl0SQ2EF2+dIywb1x
-JuinEP+AwLAUZ1Bsx9ISC0Agpk2VeHXPL3FGhroEmoMvBzO0kTFGyoeT7PR/BfKv
-+H/g3HsL2LOB9uoIm8/5p2TTU5ttYCXMHhQZ81AY
-=AUp4
 -----END PGP PUBLIC KEY BLOCK-----
 """
 
 GIT = 'git'                      # our git command
-MIN_GIT_VERSION = (1, 7, 2)      # minimum supported git version
+MIN_GIT_VERSION = (2, 10, 2)     # minimum supported git version
 repodir = '.repo'                # name of repo's private directory
 S_repo = 'repo'                  # special repo repository
 S_manifests = 'manifests'        # special manifest repository
 REPO_MAIN = S_repo + '/main.py'  # main script
-MIN_PYTHON_VERSION = (2, 7)      # minimum supported python version
 GITC_CONFIG_FILE = '/gitc/.config'
 GITC_FS_ROOT_DIR = '/gitc/manifest-rw/'
 
@@ -131,12 +178,9 @@ GITC_FS_ROOT_DIR = '/gitc/manifest-rw/'
 import collections
 import errno
 import optparse
-import platform
 import re
 import shutil
 import stat
-import subprocess
-import sys
 
 if sys.version_info[0] == 3:
   import urllib.request
@@ -149,17 +193,6 @@ else:
   urllib.error = urllib2
 
 
-# Python version check
-ver = sys.version_info
-if (ver[0], ver[1]) < MIN_PYTHON_VERSION:
-  print('error: Python version {} unsupported.\n'
-        'Please use Python {}.{} instead.'.format(
-            sys.version.split(' ')[0],
-            MIN_PYTHON_VERSION[0],
-            MIN_PYTHON_VERSION[1],
-        ), file=sys.stderr)
-  sys.exit(1)
-
 home_dot_repo = os.path.expanduser('~/.repoconfig')
 gpg_dir = os.path.join(home_dot_repo, 'gnupg')
 
@@ -235,10 +268,10 @@ group.add_option('--no-tags',
 group = init_optparse.add_option_group('repo Version options')
 group.add_option('--repo-url',
                  dest='repo_url',
-                 help='repo repository location', metavar='URL')
+                 help='repo repository location ($REPO_URL)', metavar='URL')
 group.add_option('--repo-branch',
                  dest='repo_branch',
-                 help='repo branch or revision', metavar='REVISION')
+                 help='repo branch or revision ($REPO_REV)', metavar='REVISION')
 group.add_option('--no-repo-verify',
                  dest='no_repo_verify', action='store_true',
                  help='do not verify repo source code')
@@ -926,15 +959,9 @@ def main(orig_args):
         '--']
   me.extend(orig_args)
   me.extend(extra_args)
-  try:
-    if platform.system() == "Windows":
-      sys.exit(subprocess.call(me))
-    else:
-      os.execv(sys.executable, me)
-  except OSError as e:
-    print("fatal: unable to start %s" % repo_main, file=sys.stderr)
-    print("fatal: %s" % e, file=sys.stderr)
-    sys.exit(148)
+  exec_command(me)
+  print("fatal: unable to start %s" % repo_main, file=sys.stderr)
+  sys.exit(148)
 
 
 if __name__ == '__main__':

run_tests

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 # -*- coding:utf-8 -*-
 # Copyright 2019 The Android Open Source Project
 #

setup.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 # -*- coding:utf-8 -*-
 # Copyright 2019 The Android Open Source Project
 #

subcmds/forall.py

@@ -366,7 +366,7 @@ def DoWork(project, mirror, opt, cmd, shell, cnt, config):
     while not s_in.is_done:
       in_ready = s_in.select()
       for s in in_ready:
-        buf = s.read()
+        buf = s.read().decode()
         if not buf:
           s.close()
           s_in.remove(s)

subcmds/gitc_init.py

@@ -50,7 +50,7 @@ use for this GITC client.
 """
 
   def _Options(self, p):
-    super(GitcInit, self)._Options(p)
+    super(GitcInit, self)._Options(p, gitc_init=True)
     g = p.add_option_group('GITC options')
     g.add_option('-f', '--manifest-file',
                  dest='manifest_file',

subcmds/init.py

@@ -81,7 +81,7 @@ manifest, a subsequent `repo sync` (or `repo sync -d`) is necessary
 to update the working directory files.
 """
 
-  def _Options(self, p):
+  def _Options(self, p, gitc_init=False):
     # Logging
     g = p.add_option_group('Logging options')
     g.add_option('-q', '--quiet',
@@ -96,7 +96,12 @@ to update the working directory files.
     g.add_option('-b', '--manifest-branch',
                  dest='manifest_branch',
                  help='manifest branch or revision', metavar='REVISION')
-    g.add_option('-c', '--current-branch',
+    cbr_opts = ['--current-branch']
+    # The gitc-init subcommand allocates -c itself, but a lot of init users
+    # want -c, so try to satisfy both as best we can.
+    if not gitc_init:
+      cbr_opts += ['-c']
+    g.add_option(*cbr_opts,
                  dest='current_branch_only', action='store_true',
                  help='fetch only current manifest branch from server')
     g.add_option('-m', '--manifest-name',

subcmds/sync.py

@@ -217,6 +217,10 @@ later is required to fix a server side protocol bug.
     p.add_option('-l', '--local-only',
                  dest='local_only', action='store_true',
                  help="only update working tree, don't fetch")
+    p.add_option('--no-manifest-update','--nmu',
+                 dest='mp_update', action='store_false', default='true',
+                 help='use the existing manifest checkout as-is. '
+                      '(do not update to the latest revision)')
     p.add_option('-n', '--network-only',
                  dest='network_only', action='store_true',
                  help="fetch only, don't update working tree")
@@ -364,7 +368,7 @@ later is required to fix a server side protocol bug.
 
     return success
 
-  def _Fetch(self, projects, opt):
+  def _Fetch(self, projects, opt, err_event):
     fetched = set()
     lock = _threading.Lock()
     pm = Progress('Fetching projects', len(projects),
@@ -376,7 +380,6 @@ later is required to fix a server side protocol bug.
 
     threads = set()
     sem = _threading.Semaphore(self.jobs)
-    err_event = _threading.Event()
     for project_list in objdir_project_map.values():
       # Check for any errors before running any more tasks.
       # ...we'll let existing threads finish, though.
@@ -405,16 +408,11 @@ later is required to fix a server side protocol bug.
     for t in threads:
       t.join()
 
-    # If we saw an error, exit with code 1 so that other scripts can check.
-    if err_event.isSet() and opt.fail_fast:
-      print('\nerror: Exited sync due to fetch errors', file=sys.stderr)
-      sys.exit(1)
-
     pm.end()
     self._fetch_times.Save()
 
     if not self.manifest.IsArchive:
-      self._GCProjects(projects)
+      self._GCProjects(projects, opt, err_event)
 
     return fetched
 
@@ -500,12 +498,16 @@ later is required to fix a server side protocol bug.
 
     return success
 
-  def _Checkout(self, all_projects, opt):
+  def _Checkout(self, all_projects, opt, err_event, err_results):
     """Checkout projects listed in all_projects
 
     Args:
       all_projects: List of all projects that should be checked out.
       opt: Program options returned from optparse.  See _Options().
+      err_event: We'll set this event in the case of an error (after printing
+          out info about the error).
+      err_results: A list of strings, paths to git repos where checkout
+          failed.
     """
 
     # Perform checkouts in multiple threads when we are using partial clone.
@@ -524,8 +526,6 @@ later is required to fix a server side protocol bug.
 
     threads = set()
     sem = _threading.Semaphore(syncjobs)
-    err_event = _threading.Event()
-    err_results = []
 
     for project in all_projects:
       # Check for any errors before running any more tasks.
@@ -556,15 +556,8 @@ later is required to fix a server side protocol bug.
       t.join()
 
     pm.end()
-    # If we saw an error, exit with code 1 so that other scripts can check.
-    if err_event.isSet():
-      print('\nerror: Exited sync due to checkout errors', file=sys.stderr)
-      if err_results:
-        print('Failing repos:\n%s' % '\n'.join(err_results),
-              file=sys.stderr)
-      sys.exit(1)
 
-  def _GCProjects(self, projects):
+  def _GCProjects(self, projects, opt, err_event):
     gc_gitdirs = {}
     for project in projects:
       if len(project.manifest.GetProjectsWithName(project.name)) > 1:
@@ -588,7 +581,6 @@ later is required to fix a server side protocol bug.
 
     threads = set()
     sem = _threading.Semaphore(jobs)
-    err_event = _threading.Event()
 
     def GC(bare_git):
       try:
@@ -603,7 +595,7 @@ later is required to fix a server side protocol bug.
         sem.release()
 
     for bare_git in gc_gitdirs.values():
-      if err_event.isSet():
+      if err_event.isSet() and opt.fail_fast:
         break
       sem.acquire()
       t = _threading.Thread(target=GC, args=(bare_git,))
@@ -614,10 +606,6 @@ later is required to fix a server side protocol bug.
     for t in threads:
       t.join()
 
-    if err_event.isSet():
-      print('\nerror: Exited sync due to gc errors', file=sys.stderr)
-      sys.exit(1)
-
   def _ReloadManifest(self, manifest_name=None):
     if manifest_name:
       # Override calls _Unload already
@@ -898,6 +886,8 @@ later is required to fix a server side protocol bug.
           print('error: failed to remove existing smart sync override manifest: %s' %
                 e, file=sys.stderr)
 
+    err_event = _threading.Event()
+
     rp = self.manifest.repoProject
     rp.PreSync()
 
@@ -907,7 +897,10 @@ later is required to fix a server side protocol bug.
     if opt.repo_upgraded:
       _PostRepoUpgrade(self.manifest, quiet=opt.quiet)
 
-    self._UpdateManifestProject(opt, mp, manifest_name)
+    if not opt.mp_update:
+      print('Skipping update of local manifest project.')
+    else:
+      self._UpdateManifestProject(opt, mp, manifest_name)
 
     if self.gitc_manifest:
       gitc_manifest_projects = self.GetProjects(args,
@@ -948,6 +941,10 @@ later is required to fix a server side protocol bug.
                                     missing_ok=True,
                                     submodules_ok=opt.fetch_submodules)
 
+    err_network_sync = False
+    err_update_projects = False
+    err_checkout = False
+
     self._fetch_times = _FetchTimes(self.manifest)
     if not opt.local_only:
       to_fetch = []
@@ -957,10 +954,14 @@ later is required to fix a server side protocol bug.
       to_fetch.extend(all_projects)
       to_fetch.sort(key=self._fetch_times.Get, reverse=True)
 
-      fetched = self._Fetch(to_fetch, opt)
+      fetched = self._Fetch(to_fetch, opt, err_event)
+
       _PostRepoFetch(rp, opt.no_repo_verify)
       if opt.network_only:
         # bail out now; the rest touches the working tree
+        if err_event.isSet():
+          print('\nerror: Exited sync due to fetch errors.\n', file=sys.stderr)
+          sys.exit(1)
         return
 
       # Iteratively fetch missing and/or nested unregistered submodules
@@ -982,22 +983,56 @@ later is required to fix a server side protocol bug.
         if previously_missing_set == missing_set:
           break
         previously_missing_set = missing_set
-        fetched.update(self._Fetch(missing, opt))
+        fetched.update(self._Fetch(missing, opt, err_event))
+
+      # If we saw an error, exit with code 1 so that other scripts can check.
+      if err_event.isSet():
+        err_network_sync = True
+        if opt.fail_fast:
+          print('\nerror: Exited sync due to fetch errors.\n'
+                'Local checkouts *not* updated. Resolve network issues & '
+                'retry.\n'
+                '`repo sync -l` will update some local checkouts.',
+                file=sys.stderr)
+          sys.exit(1)
 
     if self.manifest.IsMirror or self.manifest.IsArchive:
       # bail out now, we have no working tree
       return
 
     if self.UpdateProjectList(opt):
-      sys.exit(1)
+      err_event.set()
+      err_update_projects = True
+      if opt.fail_fast:
+        print('\nerror: Local checkouts *not* updated.', file=sys.stderr)
+        sys.exit(1)
 
-    self._Checkout(all_projects, opt)
+    err_results = []
+    self._Checkout(all_projects, opt, err_event, err_results)
+    if err_event.isSet():
+      err_checkout = True
+      # NB: We don't exit here because this is the last step.
 
     # If there's a notice that's supposed to print at the end of the sync, print
     # it now...
     if self.manifest.notice:
       print(self.manifest.notice)
 
+    # If we saw an error, exit with code 1 so that other scripts can check.
+    if err_event.isSet():
+      print('\nerror: Unable to fully sync the tree.', file=sys.stderr)
+      if err_network_sync:
+        print('error: Downloading network changes failed.', file=sys.stderr)
+      if err_update_projects:
+        print('error: Updating local project lists failed.', file=sys.stderr)
+      if err_checkout:
+        print('error: Checking out local projects failed.', file=sys.stderr)
+        if err_results:
+          print('Failing repos:\n%s' % '\n'.join(err_results), file=sys.stderr)
+      print('Try re-running with "-j1 --fail-fast" to exit at the first error.',
+            file=sys.stderr)
+      sys.exit(1)
+
 def _PostRepoUpgrade(manifest, quiet=False):
   wrapper = Wrapper()
   if wrapper.NeedSetupGnuPG():

tests/test_git_command.py

@@ -35,7 +35,7 @@ class GitCallUnitTest(unittest.TestCase):
     # We don't dive too deep into the values here to avoid having to update
     # whenever git versions change.  We do check relative to this min version
     # as this is what `repo` itself requires via MIN_GIT_VERSION.
-    MIN_GIT_VERSION = (1, 7, 2)
+    MIN_GIT_VERSION = (2, 10, 2)
     self.assertTrue(isinstance(ver.major, int))
     self.assertTrue(isinstance(ver.minor, int))
     self.assertTrue(isinstance(ver.micro, int))

tests/test_manifest_xml.py

@@ -0,0 +1,85 @@
+# -*- coding:utf-8 -*-
+#
+# Copyright (C) 2019 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unittests for the manifest_xml.py module."""
+
+from __future__ import print_function
+
+import unittest
+
+import error
+import manifest_xml
+
+
+class ManifestValidateFilePaths(unittest.TestCase):
+  """Check _ValidateFilePaths helper.
+
+  This doesn't access a real filesystem.
+  """
+
+  def check_both(self, *args):
+    manifest_xml.XmlManifest._ValidateFilePaths('copyfile', *args)
+    manifest_xml.XmlManifest._ValidateFilePaths('linkfile', *args)
+
+  def test_normal_path(self):
+    """Make sure good paths are accepted."""
+    self.check_both('foo', 'bar')
+    self.check_both('foo/bar', 'bar')
+    self.check_both('foo', 'bar/bar')
+    self.check_both('foo/bar', 'bar/bar')
+
+  def test_symlink_targets(self):
+    """Some extra checks for symlinks."""
+    def check(*args):
+      manifest_xml.XmlManifest._ValidateFilePaths('linkfile', *args)
+
+    # We allow symlinks to end in a slash since we allow them to point to dirs
+    # in general.  Technically the slash isn't necessary.
+    check('foo/', 'bar')
+    # We allow a single '.' to get a reference to the project itself.
+    check('.', 'bar')
+
+  def test_bad_paths(self):
+    """Make sure bad paths (src & dest) are rejected."""
+    PATHS = (
+        '..',
+        '../',
+        './',
+        'foo/',
+        './foo',
+        '../foo',
+        'foo/./bar',
+        'foo/../../bar',
+        '/foo',
+        './../foo',
+        '.git/foo',
+        # Check case folding.
+        '.GIT/foo',
+        'blah/.git/foo',
+        '.repo/foo',
+        '.repoconfig',
+        # Block ~ due to 8.3 filenames on Windows filesystems.
+        '~',
+        'foo~',
+        'blah/foo~',
+        # Block Unicode characters that get normalized out by filesystems.
+        u'foo\u200Cbar',
+    )
+    for path in PATHS:
+      self.assertRaises(
+          error.ManifestInvalidPathError, self.check_both, path, 'a')
+      self.assertRaises(
+          error.ManifestInvalidPathError, self.check_both, 'a', path)

tests/test_project.py

@@ -25,6 +25,7 @@ import subprocess
 import tempfile
 import unittest
 
+import error
 import git_config
 import project
 
@@ -134,3 +135,206 @@ class ReviewableBranchTests(unittest.TestCase):
       self.assertFalse(rb.base_exists)
       # Hard to assert anything useful about this.
       self.assertTrue(rb.date)
+
+
+class CopyLinkTestCase(unittest.TestCase):
+  """TestCase for stub repo client checkouts.
+
+  It'll have a layout like:
+    tempdir/          # self.tempdir
+      checkout/       # self.topdir
+        git-project/  # self.worktree
+
+  Attributes:
+    tempdir: A dedicated temporary directory.
+    worktree: The top of the repo client checkout.
+    topdir: The top of a project checkout.
+  """
+
+  def setUp(self):
+    self.tempdir = tempfile.mkdtemp(prefix='repo_tests')
+    self.topdir = os.path.join(self.tempdir, 'checkout')
+    self.worktree = os.path.join(self.topdir, 'git-project')
+    os.makedirs(self.topdir)
+    os.makedirs(self.worktree)
+
+  def tearDown(self):
+    shutil.rmtree(self.tempdir, ignore_errors=True)
+
+  @staticmethod
+  def touch(path):
+    with open(path, 'w') as f:
+      pass
+
+  def assertExists(self, path, msg=None):
+    """Make sure |path| exists."""
+    if os.path.exists(path):
+      return
+
+    if msg is None:
+      msg = ['path is missing: %s' % path]
+      while path != '/':
+        path = os.path.dirname(path)
+        if not path:
+          # If we're given something like "foo", abort once we get to "".
+          break
+        result = os.path.exists(path)
+        msg.append('\tos.path.exists(%s): %s' % (path, result))
+        if result:
+          msg.append('\tcontents: %r' % os.listdir(path))
+          break
+      msg = '\n'.join(msg)
+
+    raise self.failureException(msg)
+
+
+class CopyFile(CopyLinkTestCase):
+  """Check _CopyFile handling."""
+
+  def CopyFile(self, src, dest):
+    return project._CopyFile(self.worktree, src, self.topdir, dest)
+
+  def test_basic(self):
+    """Basic test of copying a file from a project to the toplevel."""
+    src = os.path.join(self.worktree, 'foo.txt')
+    self.touch(src)
+    cf = self.CopyFile('foo.txt', 'foo')
+    cf._Copy()
+    self.assertExists(os.path.join(self.topdir, 'foo'))
+
+  def test_src_subdir(self):
+    """Copy a file from a subdir of a project."""
+    src = os.path.join(self.worktree, 'bar', 'foo.txt')
+    os.makedirs(os.path.dirname(src))
+    self.touch(src)
+    cf = self.CopyFile('bar/foo.txt', 'new.txt')
+    cf._Copy()
+    self.assertExists(os.path.join(self.topdir, 'new.txt'))
+
+  def test_dest_subdir(self):
+    """Copy a file to a subdir of a checkout."""
+    src = os.path.join(self.worktree, 'foo.txt')
+    self.touch(src)
+    cf = self.CopyFile('foo.txt', 'sub/dir/new.txt')
+    self.assertFalse(os.path.exists(os.path.join(self.topdir, 'sub')))
+    cf._Copy()
+    self.assertExists(os.path.join(self.topdir, 'sub', 'dir', 'new.txt'))
+
+  def test_update(self):
+    """Make sure changed files get copied again."""
+    src = os.path.join(self.worktree, 'foo.txt')
+    dest = os.path.join(self.topdir, 'bar')
+    with open(src, 'w') as f:
+      f.write('1st')
+    cf = self.CopyFile('foo.txt', 'bar')
+    cf._Copy()
+    self.assertExists(dest)
+    with open(dest) as f:
+      self.assertEqual(f.read(), '1st')
+
+    with open(src, 'w') as f:
+      f.write('2nd!')
+    cf._Copy()
+    with open(dest) as f:
+      self.assertEqual(f.read(), '2nd!')
+
+  def test_src_block_symlink(self):
+    """Do not allow reading from a symlinked path."""
+    src = os.path.join(self.worktree, 'foo.txt')
+    sym = os.path.join(self.worktree, 'sym')
+    self.touch(src)
+    os.symlink('foo.txt', sym)
+    self.assertExists(sym)
+    cf = self.CopyFile('sym', 'foo')
+    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
+
+  def test_src_block_symlink_traversal(self):
+    """Do not allow reading through a symlink dir."""
+    src = os.path.join(self.worktree, 'bar', 'passwd')
+    os.symlink('/etc', os.path.join(self.worktree, 'bar'))
+    self.assertExists(src)
+    cf = self.CopyFile('bar/foo.txt', 'foo')
+    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
+
+  def test_src_block_dir(self):
+    """Do not allow copying from a directory."""
+    src = os.path.join(self.worktree, 'dir')
+    os.makedirs(src)
+    cf = self.CopyFile('dir', 'foo')
+    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
+
+  def test_dest_block_symlink(self):
+    """Do not allow writing to a symlink."""
+    src = os.path.join(self.worktree, 'foo.txt')
+    self.touch(src)
+    os.symlink('dest', os.path.join(self.topdir, 'sym'))
+    cf = self.CopyFile('foo.txt', 'sym')
+    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
+
+  def test_dest_block_symlink_traversal(self):
+    """Do not allow writing through a symlink dir."""
+    src = os.path.join(self.worktree, 'foo.txt')
+    self.touch(src)
+    os.symlink('/tmp', os.path.join(self.topdir, 'sym'))
+    cf = self.CopyFile('foo.txt', 'sym/foo.txt')
+    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
+
+  def test_src_block_dir(self):
+    """Do not allow copying to a directory."""
+    src = os.path.join(self.worktree, 'foo.txt')
+    self.touch(src)
+    os.makedirs(os.path.join(self.topdir, 'dir'))
+    cf = self.CopyFile('foo.txt', 'dir')
+    self.assertRaises(error.ManifestInvalidPathError, cf._Copy)
+
+
+class LinkFile(CopyLinkTestCase):
+  """Check _LinkFile handling."""
+
+  def LinkFile(self, src, dest):
+    return project._LinkFile(self.worktree, src, self.topdir, dest)
+
+  def test_basic(self):
+    """Basic test of linking a file from a project into the toplevel."""
+    src = os.path.join(self.worktree, 'foo.txt')
+    self.touch(src)
+    lf = self.LinkFile('foo.txt', 'foo')
+    lf._Link()
+    dest = os.path.join(self.topdir, 'foo')
+    self.assertExists(dest)
+    self.assertTrue(os.path.islink(dest))
+    self.assertEqual('git-project/foo.txt', os.readlink(dest))
+
+  def test_src_subdir(self):
+    """Link to a file in a subdir of a project."""
+    src = os.path.join(self.worktree, 'bar', 'foo.txt')
+    os.makedirs(os.path.dirname(src))
+    self.touch(src)
+    lf = self.LinkFile('bar/foo.txt', 'foo')
+    lf._Link()
+    self.assertExists(os.path.join(self.topdir, 'foo'))
+
+  def test_dest_subdir(self):
+    """Link a file to a subdir of a checkout."""
+    src = os.path.join(self.worktree, 'foo.txt')
+    self.touch(src)
+    lf = self.LinkFile('foo.txt', 'sub/dir/foo/bar')
+    self.assertFalse(os.path.exists(os.path.join(self.topdir, 'sub')))
+    lf._Link()
+    self.assertExists(os.path.join(self.topdir, 'sub', 'dir', 'foo', 'bar'))
+
+  def test_update(self):
+    """Make sure changed targets get updated."""
+    dest = os.path.join(self.topdir, 'sym')
+
+    src = os.path.join(self.worktree, 'foo.txt')
+    self.touch(src)
+    lf = self.LinkFile('foo.txt', 'sym')
+    lf._Link()
+    self.assertEqual('git-project/foo.txt', os.readlink(dest))
+
+    # Point the symlink somewhere else.
+    os.unlink(dest)
+    os.symlink('/', dest)
+    lf._Link()
+    self.assertEqual('git-project/foo.txt', os.readlink(dest))