Revert "Save cookies back to jar when fetching clone.bundle"

This reverts commit 4abf8e6ef8.

The curl process for updating the cookie file is not atomic.  When
fetching many bundles in parallel, we can sometimes corrupt the file
causing it to be cleared.  Since users should manage gitcookies on
their own, leave it read-only.

Bug: https://crbug.com/gerrit/12300
Change-Id: Id472c99b197bc4cf8533c649f8881509f38643c1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254092
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
(cherry picked from commit dc1d0e0c7f)

.gitignore

@@ -1,3 +1,11 @@
+*.egg-info/
+*.log
 *.pyc
+__pycache__
+/dist
 .repopickle_*
 /repoc
+/.tox
+
+# PyCharm related
+/.idea/

MANIFEST.in

@@ -0,0 +1,6 @@
+graft docs hooks tests
+include *.py
+include LICENSE
+include git_ssh
+include repo
+include run_tests

README.md

@@ -14,3 +14,23 @@ that you can put anywhere in your path.
 * [repo Manifest Format](./docs/manifest-format.md)
 * [repo Hooks](./docs/repo-hooks.md)
 * [Submitting patches](./SUBMITTING_PATCHES.md)
+* Running Repo in [Microsoft Windows](./docs/windows.md)
+
+## Install
+
+Many distros include repo, so you might be able to install from there.
+```sh
+# Debian/Ubuntu.
+$ sudo apt-get install repo
+
+# Gentoo.
+$ sudo emerge dev-vcs/repo
+```
+
+You can install it manually as well as it's a single script.
+```sh
+$ mkdir -p ~/.bin
+$ PATH="${HOME}/.bin:${PATH}"
+$ curl https://storage.googleapis.com/git-repo-downloads/repo > ~/.bin/repo
+$ chmod a+rx ~/.bin/repo
+```

SUBMITTING_PATCHES.md

@@ -69,10 +69,38 @@ suppressed in the included `.flake8` file.
 
 ## Running tests
 
-There is a [`./run_tests`](./run_tests) helper script for quickly invoking all
-of our unittests.  The coverage isn't great currently, but it should still be
-run for all commits.
+We use [pytest](https://pytest.org/) and [tox](https://tox.readthedocs.io/) for
+running tests.  You should make sure to install those first.
 
+To run the full suite against all supported Python versions, simply execute:
+```sh
+$ tox -p auto
+```
+
+We have [`./run_tests`](./run_tests) which is a simple wrapper around `pytest`:
+```sh
+# Run the full suite against the default Python version.
+$ ./run_tests
+# List each test as it runs.
+$ ./run_tests -v
+
+# Run a specific unittest module (and all tests in it).
+$ ./run_tests tests/test_git_command.py
+
+# Run a specific testsuite in a specific unittest module.
+$ ./run_tests tests/test_editor.py::EditString
+
+# Run a single test.
+$ ./run_tests tests/test_editor.py::EditString::test_cat_editor
+
+# List all available tests.
+$ ./run_tests --collect-only
+
+# Run a single test using substring match.
+$ ./run_tests -k test_cat_editor
+```
+
+The coverage isn't great currently, but it should still be run for all commits.
 Adding more unittests for changes you make would be greatly appreciated :).
 Check out the [tests/](./tests/) subdirectory for more details.
 

command.py

@@ -175,7 +175,10 @@ class Command(object):
       self._ResetPathToProjectMap(all_projects_list)
 
       for arg in args:
-        projects = manifest.GetProjectsWithName(arg)
+        # We have to filter by manifest groups in case the requested project is
+        # checked out multiple times or differently based on them.
+        projects = [project for project in manifest.GetProjectsWithName(arg)
+                    if project.MatchesGroups(groups)]
 
         if not projects:
           path = os.path.abspath(arg).replace('\\', '/')
@@ -200,7 +203,7 @@ class Command(object):
 
         for project in projects:
           if not missing_ok and not project.Exists:
-            raise NoSuchProjectError(arg)
+            raise NoSuchProjectError('%s (%s)' % (arg, project.relpath))
           if not project.MatchesGroups(groups):
             raise InvalidProjectGroupsError(arg)
 

docs/python-support.md

@@ -7,9 +7,9 @@ their old LTS/corp systems and have little power to change the system.
 
 ## Summary
 
-* Python 3.6 (released Dec 2016) is required by default starting with repo-1.14.
+* Python 3.6 (released Dec 2016) is required by default starting with repo-2.x.
 * Older versions of Python (e.g. v2.7) may use the legacy feature-frozen branch
-  based on repo-1.13.
+  based on repo-1.x.
 
 ## Overview
 

docs/windows.md

@@ -0,0 +1,144 @@
+# Microsoft Windows Details
+
+Repo is primarily developed on Linux with a lot of users on macOS.
+Windows is, unfortunately, not a common platform.
+There is support in repo for Windows, but there might be some rough edges.
+
+Keep in mind that Windows in general is "best effort" and "community supported".
+That means we don't actively test or verify behavior, but rely heavily on users
+to report problems back to us, and to contribute fixes as needed.
+
+[TOC]
+
+## Windows
+
+We only support Windows 10 or newer.
+This is largely due to symlinks not being available in older versions, but it's
+also due to most developers not using Windows.
+
+We will never add code specific to older versions of Windows.
+It might work, but it most likely won't, so please don't bother asking.
+
+## Symlinks
+
+Repo will use symlinks heavily internally.
+On *NIX platforms, this isn't an issue, but Windows makes it a bit difficult.
+
+There are some documents out there for how to do this, but usually the easiest
+answer is to run your shell as an Administrator and invoke repo/git in that.
+
+This isn't a great solution, but Windows doesn't make this easy, so here we are.
+
+### Launch Git Bash
+
+If you install Git Bash (see below), you can launch that with appropriate
+permissions so that all programs "just work".
+
+* Open the Start Menu (i.e. press the ⊞ key).
+* Find/search for "Git Bash".
+* Right click it and select "Run as administrator".
+
+*** note
+**NB**: This environment is only needed when running `repo`, or any specific `git`
+command that might involve symlinks (e.g. `pull` or `checkout`).
+You do not need to run all your commands in here such as your editor.
+***
+
+### Symlinks with GNU tools
+
+If you want to use `ln -s` inside of the default Git/bash shell, you might need
+to export this environment variable:
+```sh
+$ export MSYS="winsymlinks:nativestrict"
+```
+
+Otherwise `ln -s` will copy files and not actually create a symlink.
+This also helps `tar` unpack symlinks, so that's nice.
+
+### References
+
+* https://github.com/git-for-windows/git/wiki/Symbolic-Links
+* https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/
+
+## Python
+
+You should make sure to be running Python 3.6 or newer under Windows.
+Python 2 might work, but due to already limited platform testing, you should
+only run newer Python versions.
+See our [Python Support](./python-support.md) document for more details.
+
+You can grab the latest Windows installer here:<br>
+https://www.python.org/downloads/release/python-3
+
+## Git
+
+You should install the most recent version of Git for Windows:<br>
+https://git-scm.com/download/win
+
+When installing, make sure to turn on "Enable symbolic links" when prompted.
+
+If you've already installed Git for Windows, you can simply download the latest
+installer from above and run it again.
+It should safely upgrade things in situ for you.
+This is useful if you want to switch the symbolic link option after the fact.
+
+## Shell
+
+We don't have a specific requirement for shell environments when running repo.
+Most developers use MinTTY/bash that's included with the Git for Windows install
+(so see above for installing Git).
+
+Command & Powershell & the Windows Terminal probably work.
+Who knows!
+
+## FAQ
+
+### repo upload always complains about allowing hooks or using --no-verify!
+
+When using `repo upload` in projects that have custom repohooks, you might get
+an error like the following:
+```sh
+$ repo upload
+ERROR: You must allow the pre-upload hook or use --no-verify.
+```
+
+This can be confusing as you never get prompted.
+[MinTTY has a bug][mintty] that breaks isatty checking inside of repo which
+causes repo to never interactively prompt the user which means the upload check
+always fails.
+
+You can workaround this by manually granting consent when uploading.
+Simply add the `--verify` option whenever uploading:
+```sh
+$ repo upload --verify
+```
+
+You will have to specify this flag every time you upload.
+
+[mintty]: https://github.com/mintty/mintty/issues/56
+
+### repohooks always fail with an close_fds error.
+
+When using the [reference repohooks project][repohooks] included in AOSP,
+you might see errors like this when running `repo upload`:
+```sh
+$ repo upload
+ERROR: Traceback (most recent call last):
+  ...
+  File "C:\...\lib\subprocess.py", line 351, in __init__
+    raise ValueError("close_fds is not supported on Windows "
+ValueError: close_fds is not supported on Windows platforms if you redirect stdin/stderr/stdout
+
+Failed to run main() for pre-upload hook; see traceback above.
+```
+
+This error shows up when using Python 2.
+You should upgrade to Python 3 instead (see above).
+
+If you already have Python 3 installed, make sure it's the default version.
+Running `python --version` should say `Python 3`, not `Python 2`.
+If you didn't install the Python versions, or don't have permission to change
+the default version, you can probably workaround this by changing `$PATH` in
+your shell so the Python 3 version is found first.
+
+[repohooks]: https://android.googlesource.com/platform/tools/repohooks

editor.py

@@ -68,11 +68,14 @@ least one of these before using this command.""", file=sys.stderr)
   def EditString(cls, data):
     """Opens an editor to edit the given content.
 
-       Args:
-         data        : the text to edit
+    Args:
+      data: The text to edit.
 
-      Returns:
-        new value of edited text; None if editing did not succeed
+    Returns:
+      New value of edited text.
+
+    Raises:
+      EditorError: The editor failed to run.
     """
     editor = cls._GetEditor()
     if editor == ':':
@@ -80,7 +83,7 @@ least one of these before using this command.""", file=sys.stderr)
 
     fd, path = tempfile.mkstemp()
     try:
-      os.write(fd, data)
+      os.write(fd, data.encode('utf-8'))
       os.close(fd)
       fd = None
 
@@ -106,11 +109,8 @@ least one of these before using this command.""", file=sys.stderr)
         raise EditorError('editor failed with exit status %d: %s %s'
           % (rc, editor, path))
 
-      fd2 = open(path)
-      try:
-        return fd2.read()
-      finally:
-        fd2.close()
+      with open(path, mode='rb') as fd2:
+        return fd2.read().decode('utf-8')
     finally:
       if fd:
         os.close(fd)

git_command.py

@@ -22,6 +22,7 @@ import tempfile
 from signal import SIGTERM
 
 from error import GitError
+from git_refs import HEAD
 import platform_utils
 from repo_trace import REPO_TRACE, IsTrace, Trace
 from wrapper import Wrapper
@@ -98,6 +99,86 @@ class _GitCall(object):
     return fun
 git = _GitCall()
 
+
+def RepoSourceVersion():
+  """Return the version of the repo.git tree."""
+  ver = getattr(RepoSourceVersion, 'version', None)
+
+  # We avoid GitCommand so we don't run into circular deps -- GitCommand needs
+  # to initialize version info we provide.
+  if ver is None:
+    env = GitCommand._GetBasicEnv()
+
+    proj = os.path.dirname(os.path.abspath(__file__))
+    env[GIT_DIR] = os.path.join(proj, '.git')
+
+    p = subprocess.Popen([GIT, 'describe', HEAD], stdout=subprocess.PIPE,
+                         env=env)
+    if p.wait() == 0:
+      ver = p.stdout.read().strip().decode('utf-8')
+      if ver.startswith('v'):
+        ver = ver[1:]
+    else:
+      ver = 'unknown'
+    setattr(RepoSourceVersion, 'version', ver)
+
+  return ver
+
+
+class UserAgent(object):
+  """Mange User-Agent settings when talking to external services
+
+  We follow the style as documented here:
+  https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent
+  """
+
+  _os = None
+  _repo_ua = None
+  _git_ua = None
+
+  @property
+  def os(self):
+    """The operating system name."""
+    if self._os is None:
+      os_name = sys.platform
+      if os_name.lower().startswith('linux'):
+        os_name = 'Linux'
+      elif os_name == 'win32':
+        os_name = 'Win32'
+      elif os_name == 'cygwin':
+        os_name = 'Cygwin'
+      elif os_name == 'darwin':
+        os_name = 'Darwin'
+      self._os = os_name
+
+    return self._os
+
+  @property
+  def repo(self):
+    """The UA when connecting directly from repo."""
+    if self._repo_ua is None:
+      py_version = sys.version_info
+      self._repo_ua = 'git-repo/%s (%s) git/%s Python/%d.%d.%d' % (
+          RepoSourceVersion(),
+          self.os,
+          git.version_tuple().full,
+          py_version.major, py_version.minor, py_version.micro)
+
+    return self._repo_ua
+
+  @property
+  def git(self):
+    """The UA when running git."""
+    if self._git_ua is None:
+      self._git_ua = 'git/%s (%s) git-repo/%s' % (
+          git.version_tuple().full,
+          self.os,
+          RepoSourceVersion())
+
+    return self._git_ua
+
+user_agent = UserAgent()
+
 def git_require(min_version, fail=False, msg=''):
   git_version = git.version_tuple()
   if min_version <= git_version:
@@ -125,17 +206,7 @@ class GitCommand(object):
                ssh_proxy = False,
                cwd = None,
                gitdir = None):
-    env = os.environ.copy()
-
-    for key in [REPO_TRACE,
-              GIT_DIR,
-              'GIT_ALTERNATE_OBJECT_DIRECTORIES',
-              'GIT_OBJECT_DIRECTORY',
-              'GIT_WORK_TREE',
-              'GIT_GRAFT_FILE',
-              'GIT_INDEX_FILE']:
-      if key in env:
-        del env[key]
+    env = self._GetBasicEnv()
 
     # If we are not capturing std* then need to print it.
     self.tee = {'stdout': not capture_stdout, 'stderr': not capture_stderr}
@@ -155,6 +226,7 @@ class GitCommand(object):
     if 'GIT_ALLOW_PROTOCOL' not in env:
       _setenv(env, 'GIT_ALLOW_PROTOCOL',
               'file:git:http:https:ssh:persistent-http:persistent-https:sso:rpc')
+    _setenv(env, 'GIT_HTTP_USER_AGENT', user_agent.git)
 
     if project:
       if not cwd:
@@ -227,6 +299,23 @@ class GitCommand(object):
     self.process = p
     self.stdin = p.stdin
 
+  @staticmethod
+  def _GetBasicEnv():
+    """Return a basic env for running git under.
+
+    This is guaranteed to be side-effect free.
+    """
+    env = os.environ.copy()
+    for key in (REPO_TRACE,
+                GIT_DIR,
+                'GIT_ALTERNATE_OBJECT_DIRECTORIES',
+                'GIT_OBJECT_DIRECTORY',
+                'GIT_WORK_TREE',
+                'GIT_GRAFT_FILE',
+                'GIT_INDEX_FILE'):
+      env.pop(key, None)
+    return env
+
   def Wait(self):
     try:
       p = self.process

git_config.py

@@ -276,22 +276,16 @@ class GitConfig(object):
       return None
     try:
       Trace(': parsing %s', self.file)
-      fd = open(self._json)
-      try:
+      with open(self._json) as fd:
         return json.load(fd)
-      finally:
-        fd.close()
     except (IOError, ValueError):
       platform_utils.remove(self._json)
       return None
 
   def _SaveJson(self, cache):
     try:
-      fd = open(self._json, 'w')
-      try:
+      with open(self._json, 'w') as fd:
         json.dump(cache, fd, indent=2)
-      finally:
-        fd.close()
     except (IOError, TypeError):
       if os.path.exists(self._json):
         platform_utils.remove(self._json)
@@ -534,7 +528,7 @@ def GetUrlCookieFile(url, quiet):
         cookiefile = None
         proxy = None
         for line in p.stdout:
-          line = line.strip()
+          line = line.strip().decode('utf-8')
           if line.startswith(cookieprefix):
             cookiefile = os.path.expanduser(line[len(cookieprefix):])
           if line.startswith(proxyprefix):
@@ -546,7 +540,7 @@ def GetUrlCookieFile(url, quiet):
       finally:
         p.stdin.close()
         if p.wait():
-          err_msg = p.stderr.read()
+          err_msg = p.stderr.read().decode('utf-8')
           if ' -print_config' in err_msg:
             pass  # Persistent proxy doesn't support -print_config.
           elif not quiet:
@@ -773,15 +767,12 @@ class Branch(object):
       self._Set('merge', self.merge)
 
     else:
-      fd = open(self._config.file, 'a')
-      try:
+      with open(self._config.file, 'a') as fd:
         fd.write('[branch "%s"]\n' % self.name)
         if self.remote:
           fd.write('\tremote = %s\n' % self.remote.name)
         if self.merge:
           fd.write('\tmerge = %s\n' % self.merge)
-      finally:
-        fd.close()
 
   def _Set(self, key, value):
     key = 'branch.%s.%s' % (self.name, key)

git_refs.py

@@ -141,18 +141,11 @@ class GitRefs(object):
 
   def _ReadLoose1(self, path, name):
     try:
-      fd = open(path)
-    except IOError:
-      return
-
-    try:
-      try:
+      with open(path) as fd:
         mtime = os.path.getmtime(path)
         ref_id = fd.readline()
-      except (IOError, OSError):
-        return
-    finally:
-      fd.close()
+    except (IOError, OSError):
+      return
 
     try:
       ref_id = ref_id.decode()

main.py

@@ -23,17 +23,18 @@ which takes care of execing this entry point.
 
 from __future__ import print_function
 import getpass
-import imp
 import netrc
 import optparse
 import os
 import sys
+import textwrap
 import time
 
 from pyversion import is_python3
 if is_python3():
   import urllib.request
 else:
+  import imp
   import urllib2
   urllib = imp.new_module('urllib')
   urllib.request = urllib2
@@ -46,7 +47,7 @@ except ImportError:
 from color import SetDefaultColoring
 import event_log
 from repo_trace import SetTrace
-from git_command import git, GitCommand
+from git_command import git, GitCommand, user_agent
 from git_config import init_ssh, close_ssh
 from command import InteractiveCommand
 from command import MirrorSafeCommand
@@ -71,8 +72,10 @@ if not is_python3():
   input = raw_input
 
 global_options = optparse.OptionParser(
-                 usage="repo [-p|--paginate|--no-pager] COMMAND [ARGS]"
-                 )
+    usage='repo [-p|--paginate|--no-pager] COMMAND [ARGS]',
+    add_help_option=False)
+global_options.add_option('-h', '--help', action='store_true',
+                          help='show this help message and exit')
 global_options.add_option('-p', '--paginate',
                           dest='pager', action='store_true',
                           help='display command output in the pager')
@@ -123,6 +126,14 @@ class _Repo(object):
       argv = []
     gopts, _gargs = global_options.parse_args(glob)
 
+    if gopts.help:
+      global_options.print_help()
+      commands = ' '.join(sorted(self.commands))
+      wrapped_commands = textwrap.wrap(commands, width=77)
+      print('\nAvailable commands:\n  %s' % ('\n  '.join(wrapped_commands),))
+      print('\nRun `repo help <command>` for command-specific details.')
+      global_options.exit()
+
     return (name, gopts, argv)
 
   def _Run(self, name, gopts, argv):
@@ -244,10 +255,6 @@ class _Repo(object):
     return result
 
 
-def _MyRepoPath():
-  return os.path.dirname(__file__)
-
-
 def _CheckWrapperVersion(ver, repo_path):
   if not repo_path:
     repo_path = '~/bin/repo'
@@ -299,51 +306,13 @@ def _PruneOptions(argv, opt):
       continue
     i += 1
 
-_user_agent = None
-
-def _UserAgent():
-  global _user_agent
-
-  if _user_agent is None:
-    py_version = sys.version_info
-
-    os_name = sys.platform
-    if os_name == 'linux2':
-      os_name = 'Linux'
-    elif os_name == 'win32':
-      os_name = 'Win32'
-    elif os_name == 'cygwin':
-      os_name = 'Cygwin'
-    elif os_name == 'darwin':
-      os_name = 'Darwin'
-
-    p = GitCommand(
-      None, ['describe', 'HEAD'],
-      cwd = _MyRepoPath(),
-      capture_stdout = True)
-    if p.Wait() == 0:
-      repo_version = p.stdout
-      if len(repo_version) > 0 and repo_version[-1] == '\n':
-        repo_version = repo_version[0:-1]
-      if len(repo_version) > 0 and repo_version[0] == 'v':
-        repo_version = repo_version[1:]
-    else:
-      repo_version = 'unknown'
-
-    _user_agent = 'git-repo/%s (%s) git/%s Python/%d.%d.%d' % (
-      repo_version,
-      os_name,
-      git.version_tuple().full,
-      py_version[0], py_version[1], py_version[2])
-  return _user_agent
-
 class _UserAgentHandler(urllib.request.BaseHandler):
   def http_request(self, req):
-    req.add_header('User-Agent', _UserAgent())
+    req.add_header('User-Agent', user_agent.repo)
     return req
 
   def https_request(self, req):
-    req.add_header('User-Agent', _UserAgent())
+    req.add_header('User-Agent', user_agent.repo)
     return req
 
 def _AddPasswordFromUserInput(handler, msg, req):

platform_utils.py

@@ -80,7 +80,7 @@ class FileDescriptorStreams(object):
     """
     raise NotImplementedError
 
-  def _create_stream(fd, dest, std_name):
+  def _create_stream(self, fd, dest, std_name):
     """ Creates a new stream wrapping an existing file descriptor.
     """
     raise NotImplementedError
@@ -241,14 +241,15 @@ def _makelongpath(path):
     return path
 
 
-def rmtree(path):
+def rmtree(path, ignore_errors=False):
   """shutil.rmtree(path) wrapper with support for long paths on Windows.
 
   Availability: Unix, Windows."""
+  onerror = None
   if isWindows():
-    shutil.rmtree(_makelongpath(path), onerror=handle_rmtree_error)
-  else:
-    shutil.rmtree(path)
+    path = _makelongpath(path)
+    onerror = handle_rmtree_error
+  shutil.rmtree(path, ignore_errors=ignore_errors, onerror=onerror)
 
 
 def handle_rmtree_error(function, path, excinfo):

platform_utils_win32.py

@@ -16,6 +16,7 @@
 
 import errno
 
+from pyversion import is_python3
 from ctypes import WinDLL, get_last_error, FormatError, WinError, addressof
 from ctypes import c_buffer
 from ctypes.wintypes import BOOL, BOOLEAN, LPCWSTR, DWORD, HANDLE, POINTER, c_ubyte
@@ -179,7 +180,7 @@ def readlink(path):
   if reparse_point_handle == INVALID_HANDLE_VALUE:
     _raise_winerror(
         get_last_error(),
-        'Error opening symblic link \"%s\"'.format(path))
+        'Error opening symbolic link \"%s\"'.format(path))
   target_buffer = c_buffer(MAXIMUM_REPARSE_DATA_BUFFER_SIZE)
   n_bytes_returned = DWORD()
   io_result = DeviceIoControl(reparse_point_handle,
@@ -194,7 +195,7 @@ def readlink(path):
   if not io_result:
     _raise_winerror(
         get_last_error(),
-        'Error reading symblic link \"%s\"'.format(path))
+        'Error reading symbolic link \"%s\"'.format(path))
   rdb = REPARSE_DATA_BUFFER.from_buffer(target_buffer)
   if rdb.ReparseTag == IO_REPARSE_TAG_SYMLINK:
     return _preserve_encoding(path, rdb.SymbolicLinkReparseBuffer.PrintName)
@@ -203,11 +204,15 @@ def readlink(path):
   # Unsupported reparse point type
   _raise_winerror(
       ERROR_NOT_SUPPORTED,
-      'Error reading symblic link \"%s\"'.format(path))
+      'Error reading symbolic link \"%s\"'.format(path))
 
 
 def _preserve_encoding(source, target):
   """Ensures target is the same string type (i.e. unicode or str) as source."""
+
+  if is_python3():
+    return target
+
   if isinstance(source, unicode):
     return unicode(target)
   return str(target)

progress.py

@@ -39,7 +39,7 @@ class Progress(object):
     self._print_newline = print_newline
     self._always_print_percentage = always_print_percentage
 
-  def update(self, inc=1):
+  def update(self, inc=1, msg=''):
     self._done += inc
 
     if _NOT_TTY or IsTrace():
@@ -62,12 +62,13 @@ class Progress(object):
 
       if self._lastp != p or self._always_print_percentage:
         self._lastp = p
-        sys.stderr.write('%s\r%s: %3d%% (%d%s/%d%s)%s' % (
+        sys.stderr.write('%s\r%s: %3d%% (%d%s/%d%s)%s%s%s' % (
           CSI_ERASE_LINE,
           self._title,
           p,
           self._done, self._units,
           self._total, self._units,
+          ' ' if msg else '', msg,
           "\n" if self._print_newline else ""))
         sys.stderr.flush()
 

project.py

@@ -58,11 +58,8 @@ else:
 def _lwrite(path, content):
   lock = '%s.lock' % path
 
-  fd = open(lock, 'w')
-  try:
+  with open(lock, 'w') as fd:
     fd.write(content)
-  finally:
-    fd.close()
 
   try:
     platform_utils.rename(lock, path)
@@ -137,6 +134,7 @@ class DownloadedChange(object):
 
 class ReviewableBranch(object):
   _commit_cache = None
+  _base_exists = None
 
   def __init__(self, project, branch, base):
     self.project = project
@@ -150,14 +148,19 @@ class ReviewableBranch(object):
   @property
   def commits(self):
     if self._commit_cache is None:
-      self._commit_cache = self.project.bare_git.rev_list('--abbrev=8',
-                                                          '--abbrev-commit',
-                                                          '--pretty=oneline',
-                                                          '--reverse',
-                                                          '--date-order',
-                                                          not_rev(self.base),
-                                                          R_HEADS + self.name,
-                                                          '--')
+      args = ('--abbrev=8', '--abbrev-commit', '--pretty=oneline', '--reverse',
+              '--date-order', not_rev(self.base), R_HEADS + self.name, '--')
+      try:
+        self._commit_cache = self.project.bare_git.rev_list(*args)
+      except GitError:
+        # We weren't able to probe the commits for this branch.  Was it tracking
+        # a branch that no longer exists?  If so, return no commits.  Otherwise,
+        # rethrow the error as we don't know what's going on.
+        if self.base_exists:
+          raise
+
+        self._commit_cache = []
+
     return self._commit_cache
 
   @property
@@ -176,6 +179,23 @@ class ReviewableBranch(object):
                                      R_HEADS + self.name,
                                      '--')
 
+  @property
+  def base_exists(self):
+    """Whether the branch we're tracking exists.
+
+    Normally it should, but sometimes branches we track can get deleted.
+    """
+    if self._base_exists is None:
+      try:
+        self.project.bare_git.rev_parse('--verify', not_rev(self.base))
+        # If we're still here, the base branch exists.
+        self._base_exists = True
+      except GitError:
+        # If we failed to verify, the base branch doesn't exist.
+        self._base_exists = False
+
+    return self._base_exists
+
   def UploadForReview(self, people,
                       auto_topic=False,
                       draft=False,
@@ -230,6 +250,7 @@ class DiffColoring(Coloring):
   def __init__(self, config):
     Coloring.__init__(self, config, 'diff')
     self.project = self.printer('header', attr='bold')
+    self.fail = self.printer('fail', fg='red')
 
 
 class _Annotation(object):
@@ -865,10 +886,17 @@ class Project(object):
   @property
   def CurrentBranch(self):
     """Obtain the name of the currently checked out branch.
-       The branch name omits the 'refs/heads/' prefix.
-       None is returned if the project is on a detached HEAD.
+
+    The branch name omits the 'refs/heads/' prefix.
+    None is returned if the project is on a detached HEAD, or if the work_git is
+    otheriwse inaccessible (e.g. an incomplete sync).
     """
-    b = self.work_git.GetHead()
+    try:
+      b = self.work_git.GetHead()
+    except NoManifestException:
+      # If the local checkout is in a bad state, don't barf.  Let the callers
+      # process this like the head is unreadable.
+      return None
     if b.startswith(R_HEADS):
       return b[len(R_HEADS):]
     return None
@@ -1037,7 +1065,7 @@ class Project(object):
     """Prints the status of the repository to stdout.
 
     Args:
-      output: If specified, redirect the output to this object.
+      output_redir: If specified, redirect the output to this object.
       quiet:  If True then only print the project name.  Do not print
               the modified files, branch name, etc.
     """
@@ -1136,10 +1164,18 @@ class Project(object):
       cmd.append('--src-prefix=a/%s/' % self.relpath)
       cmd.append('--dst-prefix=b/%s/' % self.relpath)
     cmd.append('--')
-    p = GitCommand(self,
-                   cmd,
-                   capture_stdout=True,
-                   capture_stderr=True)
+    try:
+      p = GitCommand(self,
+                     cmd,
+                     capture_stdout=True,
+                     capture_stderr=True)
+    except GitError as e:
+      out.nl()
+      out.project('project %s/' % self.relpath)
+      out.nl()
+      out.fail('%s', str(e))
+      out.nl()
+      return False
     has_diff = False
     for line in p.process.stdout:
       if not hasattr(line, 'encode'):
@@ -1150,7 +1186,7 @@ class Project(object):
         out.nl()
         has_diff = True
       print(line[:-1])
-    p.Wait()
+    return p.Wait() == 0
 
 
 # Publish / Upload ##
@@ -1377,12 +1413,9 @@ class Project(object):
     if is_new:
       alt = os.path.join(self.gitdir, 'objects/info/alternates')
       try:
-        fd = open(alt)
-        try:
+        with open(alt) as fd:
           # This works for both absolute and relative alternate directories.
           alt_dir = os.path.join(self.objdir, 'objects', fd.readline().rstrip())
-        finally:
-          fd.close()
       except IOError:
         alt_dir = None
     else:
@@ -1489,6 +1522,13 @@ class Project(object):
     """Perform only the local IO portion of the sync process.
        Network access is not required.
     """
+    if not os.path.exists(self.gitdir):
+      syncbuf.fail(self,
+                   'Cannot checkout %s due to missing network sync; Run '
+                   '`repo sync -n %s` first.' %
+                   (self.name, self.name))
+      return
+
     self._InitWorkTree(force_sync=force_sync, submodules=submodules)
     all_refs = self.bare_ref.all
     self.CleanPublishedCache(all_refs)
@@ -1569,7 +1609,16 @@ class Project(object):
       return
 
     upstream_gain = self._revlist(not_rev(HEAD), revid)
-    pub = self.WasPublished(branch.name, all_refs)
+
+    # See if we can perform a fast forward merge.  This can happen if our
+    # branch isn't in the exact same state as we last published.
+    try:
+      self.work_git.merge_base('--is-ancestor', HEAD, revid)
+      # Skip the published logic.
+      pub = False
+    except GitError:
+      pub = self.WasPublished(branch.name, all_refs)
+
     if pub:
       not_merged = self._revlist(not_rev(revid), pub)
       if not_merged:
@@ -1940,7 +1989,7 @@ class Project(object):
       gitmodules_lines = []
       fd, temp_gitmodules_path = tempfile.mkstemp()
       try:
-        os.write(fd, p.stdout)
+        os.write(fd, p.stdout.encode('utf-8'))
         os.close(fd)
         cmd = ['config', '--file', temp_gitmodules_path, '--list']
         p = GitCommand(None, cmd, capture_stdout=True, capture_stderr=True,
@@ -2186,16 +2235,6 @@ class Project(object):
       cmd.append('--update-head-ok')
     cmd.append(name)
 
-    spec = []
-
-    # If using depth then we should not get all the tags since they may
-    # be outside of the depth.
-    if no_tags or depth:
-      cmd.append('--no-tags')
-    else:
-      cmd.append('--tags')
-      spec.append(str((u'+refs/tags/*:') + remote.ToLocal('refs/tags/*')))
-
     if force_sync:
       cmd.append('--force')
 
@@ -2205,6 +2244,7 @@ class Project(object):
     if submodules:
       cmd.append('--recurse-submodules=on-demand')
 
+    spec = []
     if not current_branch_only:
       # Fetch whole repo
       spec.append(str((u'+refs/heads/*:') + remote.ToLocal('refs/heads/*')))
@@ -2212,19 +2252,36 @@ class Project(object):
       spec.append('tag')
       spec.append(tag_name)
 
-    if not self.manifest.IsMirror:
+    if self.manifest.IsMirror and not current_branch_only:
+      branch = None
+    else:
       branch = self.revisionExpr
-      if is_sha1 and depth and git_require((1, 8, 3)):
-        # Shallow checkout of a specific commit, fetch from that commit and not
-        # the heads only as the commit might be deeper in the history.
-        spec.append(branch)
-      else:
-        if is_sha1:
-          branch = self.upstream
-        if branch is not None and branch.strip():
-          if not branch.startswith('refs/'):
-            branch = R_HEADS + branch
-          spec.append(str((u'+%s:' % branch) + remote.ToLocal(branch)))
+    if (not self.manifest.IsMirror and is_sha1 and depth
+        and git_require((1, 8, 3))):
+      # Shallow checkout of a specific commit, fetch from that commit and not
+      # the heads only as the commit might be deeper in the history.
+      spec.append(branch)
+    else:
+      if is_sha1:
+        branch = self.upstream
+      if branch is not None and branch.strip():
+        if not branch.startswith('refs/'):
+          branch = R_HEADS + branch
+        spec.append(str((u'+%s:' % branch) + remote.ToLocal(branch)))
+
+    # If mirroring repo and we cannot deduce the tag or branch to fetch, fetch
+    # whole repo.
+    if self.manifest.IsMirror and not spec:
+      spec.append(str((u'+refs/heads/*:') + remote.ToLocal('refs/heads/*')))
+
+    # If using depth then we should not get all the tags since they may
+    # be outside of the depth.
+    if no_tags or depth:
+      cmd.append('--no-tags')
+    else:
+      cmd.append('--tags')
+      spec.append(str((u'+refs/tags/*:') + remote.ToLocal('refs/tags/*')))
+
     cmd.extend(spec)
 
     ok = False
@@ -2341,7 +2398,7 @@ class Project(object):
         platform_utils.remove(tmpPath)
     with GetUrlCookieFile(srcUrl, quiet) as (cookiefile, proxy):
       if cookiefile:
-        cmd += ['--cookie', cookiefile, '--cookie-jar', cookiefile]
+        cmd += ['--cookie', cookiefile]
       if proxy:
         cmd += ['--proxy', proxy]
       elif 'http_proxy' in os.environ and 'darwin' == sys.platform:
@@ -2690,41 +2747,45 @@ class Project(object):
           raise
 
   def _InitWorkTree(self, force_sync=False, submodules=False):
-    dotgit = os.path.join(self.worktree, '.git')
-    init_dotgit = not os.path.exists(dotgit)
+    realdotgit = os.path.join(self.worktree, '.git')
+    tmpdotgit = realdotgit + '.tmp'
+    init_dotgit = not os.path.exists(realdotgit)
+    if init_dotgit:
+      dotgit = tmpdotgit
+      platform_utils.rmtree(tmpdotgit, ignore_errors=True)
+      os.makedirs(tmpdotgit)
+      self._ReferenceGitDir(self.gitdir, tmpdotgit, share_refs=True,
+                            copy_all=False)
+    else:
+      dotgit = realdotgit
+
     try:
-      if init_dotgit:
-        os.makedirs(dotgit)
-        self._ReferenceGitDir(self.gitdir, dotgit, share_refs=True,
-                              copy_all=False)
+      self._CheckDirReference(self.gitdir, dotgit, share_refs=True)
+    except GitError as e:
+      if force_sync and not init_dotgit:
+        try:
+          platform_utils.rmtree(dotgit)
+          return self._InitWorkTree(force_sync=False, submodules=submodules)
+        except:
+          raise e
+      raise e
 
-      try:
-        self._CheckDirReference(self.gitdir, dotgit, share_refs=True)
-      except GitError as e:
-        if force_sync:
-          try:
-            platform_utils.rmtree(dotgit)
-            return self._InitWorkTree(force_sync=False, submodules=submodules)
-          except:
-            raise e
-        raise e
+    if init_dotgit:
+      _lwrite(os.path.join(tmpdotgit, HEAD), '%s\n' % self.GetRevisionId())
 
-      if init_dotgit:
-        _lwrite(os.path.join(dotgit, HEAD), '%s\n' % self.GetRevisionId())
+      # Now that the .git dir is fully set up, move it to its final home.
+      platform_utils.rename(tmpdotgit, realdotgit)
 
-        cmd = ['read-tree', '--reset', '-u']
-        cmd.append('-v')
-        cmd.append(HEAD)
-        if GitCommand(self, cmd).Wait() != 0:
-          raise GitError("cannot initialize work tree for " + self.name)
+      # Finish checking out the worktree.
+      cmd = ['read-tree', '--reset', '-u']
+      cmd.append('-v')
+      cmd.append(HEAD)
+      if GitCommand(self, cmd).Wait() != 0:
+        raise GitError('Cannot initialize work tree for ' + self.name)
 
-        if submodules:
-          self._SyncSubmodules(quiet=True)
-        self._CopyAndLinkFiles()
-    except Exception:
-      if init_dotgit:
-        platform_utils.rmtree(dotgit)
-      raise
+      if submodules:
+        self._SyncSubmodules(quiet=True)
+      self._CopyAndLinkFiles()
 
   def _get_symlink_error_message(self):
     if platform_utils.isWindows():
@@ -2873,13 +2934,10 @@ class Project(object):
       else:
         path = os.path.join(self._project.worktree, '.git', HEAD)
       try:
-        fd = open(path)
+        with open(path) as fd:
+          line = fd.readline()
       except IOError as e:
         raise NoManifestException(path, str(e))
-      try:
-        line = fd.readline()
-      finally:
-        fd.close()
       try:
         line = line.decode()
       except AttributeError:

repo

@@ -16,7 +16,9 @@ import os
 REPO_URL = os.environ.get('REPO_URL', None)
 if not REPO_URL:
   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
-REPO_REV = 'stable'
+REPO_REV = os.environ.get('REPO_REV')
+if not REPO_REV:
+  REPO_REV = 'repo-1'
 
 # Copyright (C) 2008 Google Inc.
 #
@@ -33,7 +35,7 @@ REPO_REV = 'stable'
 # limitations under the License.
 
 # increment this whenever we make important changes to this script
-VERSION = (1, 25)
+VERSION = (1, 27)
 
 # increment this if the MAINTAINER_KEYS block is modified
 KEYRING_VERSION = (1, 2)
@@ -235,10 +237,10 @@ group.add_option('--no-tags',
 group = init_optparse.add_option_group('repo Version options')
 group.add_option('--repo-url',
                  dest='repo_url',
-                 help='repo repository location', metavar='URL')
+                 help='repo repository location ($REPO_URL)', metavar='URL')
 group.add_option('--repo-branch',
                  dest='repo_branch',
-                 help='repo branch or revision', metavar='REVISION')
+                 help='repo branch or revision ($REPO_REV)', metavar='REVISION')
 group.add_option('--no-repo-verify',
                  dest='no_repo_verify', action='store_true',
                  help='do not verify repo source code')
@@ -366,15 +368,18 @@ def _Init(args, gitc_init=False):
 
   _CheckGitVersion()
   try:
-    if NeedSetupGnuPG():
-      can_verify = SetupGnuPG(opt.quiet)
+    if opt.no_repo_verify:
+      do_verify = False
     else:
-      can_verify = True
+      if NeedSetupGnuPG():
+        do_verify = SetupGnuPG(opt.quiet)
+      else:
+        do_verify = True
 
     dst = os.path.abspath(os.path.join(repodir, S_repo))
     _Clone(url, dst, opt.quiet, not opt.no_clone_bundle)
 
-    if can_verify and not opt.no_repo_verify:
+    if do_verify:
       rev = _Verify(dst, branch, opt.quiet)
     else:
       rev = 'refs/remotes/origin/%s^0' % branch
@@ -513,9 +518,8 @@ def SetupGnuPG(quiet):
     sys.exit(1)
   print()
 
-  fd = open(os.path.join(home_dot_repo, 'keyring-version'), 'w')
-  fd.write('.'.join(map(str, KEYRING_VERSION)) + '\n')
-  fd.close()
+  with open(os.path.join(home_dot_repo, 'keyring-version'), 'w') as fd:
+    fd.write('.'.join(map(str, KEYRING_VERSION)) + '\n')
   return True
 
 

run_tests

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 # -*- coding:utf-8 -*-
 # Copyright 2019 The Android Open Source Project
 #
@@ -27,14 +27,13 @@ import sys
 def run_pytest(cmd, argv):
   """Run the unittests via |cmd|."""
   try:
-    subprocess.check_call([cmd] + argv)
-    return 0
+    return subprocess.call([cmd] + argv)
   except OSError as e:
     if e.errno == errno.ENOENT:
       print('%s: unable to run `%s`: %s' % (__file__, cmd, e), file=sys.stderr)
       print('%s: Try installing pytest: sudo apt-get install python-pytest' %
             (__file__,), file=sys.stderr)
-      return 1
+      return 127
     else:
       raise
 

setup.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+# -*- coding:utf-8 -*-
+# Copyright 2019 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the 'License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Python packaging for repo."""
+
+from __future__ import print_function
+
+import os
+import setuptools
+
+
+TOPDIR = os.path.dirname(os.path.abspath(__file__))
+
+
+# Rip out the first intro paragraph.
+with open(os.path.join(TOPDIR, 'README.md')) as fp:
+    lines = fp.read().splitlines()[2:]
+    end = lines.index('')
+    long_description = ' '.join(lines[0:end])
+
+
+# https://packaging.python.org/tutorials/packaging-projects/
+setuptools.setup(
+    name='repo',
+    version='1.13.8',
+    maintainer='Various',
+    maintainer_email='repo-discuss@googlegroups.com',
+    description='Repo helps manage many Git repositories',
+    long_description=long_description,
+    long_description_content_type='text/plain',
+    url='https://gerrit.googlesource.com/git-repo/',
+    project_urls={
+        'Bug Tracker': 'https://bugs.chromium.org/p/gerrit/issues/list?q=component:repo',
+    },
+    # https://pypi.org/classifiers/
+    classifiers=[
+        'Development Status :: 6 - Mature',
+        'Environment :: Console',
+        'Intended Audience :: Developers',
+        'License :: OSI Approved :: Apache Software License',
+        'Natural Language :: English',
+        'Operating System :: MacOS :: MacOS X',
+        'Operating System :: Microsoft :: Windows :: Windows 10',
+        'Operating System :: POSIX :: Linux',
+        'Topic :: Software Development :: Version Control :: Git',
+    ],
+    # We support Python 2.7 and Python 3.6+.
+    python_requires='>=2.7, ' + ', '.join('!=3.%i.*' % x for x in range(0, 6)),
+    packages=['subcmds'],
+)

subcmds/diff.py

@@ -37,5 +37,8 @@ to the Unix 'patch' command.
                  help='Paths are relative to the repository root')
 
   def Execute(self, opt, args):
+    ret = 0
     for project in self.GetProjects(args):
-      project.PrintWorkTreeDiff(opt.absolute)
+      if not project.PrintWorkTreeDiff(opt.absolute):
+        ret = 1
+    return ret

subcmds/forall.py

@@ -139,6 +139,9 @@ without iterating through the remaining projects.
     p.add_option('-e', '--abort-on-errors',
                  dest='abort_on_errors', action='store_true',
                  help='Abort if a command exits unsuccessfully')
+    p.add_option('--ignore-missing', action='store_true',
+                 help='Silently skip & do not exit non-zero due missing '
+                      'checkouts')
 
     g = p.add_option_group('Output')
     g.add_option('-p',
@@ -323,10 +326,14 @@ def DoWork(project, mirror, opt, cmd, shell, cnt, config):
     cwd = project['worktree']
 
   if not os.path.exists(cwd):
-    if (opt.project_header and opt.verbose) \
-    or not opt.project_header:
+    # Allow the user to silently ignore missing checkouts so they can run on
+    # partial checkouts (good for infra recovery tools).
+    if opt.ignore_missing:
+      return 0
+    if ((opt.project_header and opt.verbose)
+        or not opt.project_header):
       print('skipping %s/' % project['relpath'], file=sys.stderr)
-    return
+    return 1
 
   if opt.project_header:
     stdin = subprocess.PIPE
@@ -359,7 +366,7 @@ def DoWork(project, mirror, opt, cmd, shell, cnt, config):
     while not s_in.is_done:
       in_ready = s_in.select()
       for s in in_ready:
-        buf = s.read()
+        buf = s.read().decode()
         if not buf:
           s.close()
           s_in.remove(s)

subcmds/gitc_init.py

@@ -50,7 +50,7 @@ use for this GITC client.
 """
 
   def _Options(self, p):
-    super(GitcInit, self)._Options(p)
+    super(GitcInit, self)._Options(p, gitc_init=True)
     g = p.add_option_group('GITC options')
     g.add_option('-f', '--manifest-file',
                  dest='manifest_file',

subcmds/grep.py

@@ -15,15 +15,19 @@
 # limitations under the License.
 
 from __future__ import print_function
+
 import sys
+
 from color import Coloring
 from command import PagedCommand
+from error import GitError
 from git_command import git_require, GitCommand
 
 class GrepColoring(Coloring):
   def __init__(self, config):
     Coloring.__init__(self, config, 'grep')
     self.project = self.printer('project', attr='bold')
+    self.fail = self.printer('fail', fg='red')
 
 class Grep(PagedCommand):
   common = True
@@ -184,15 +188,25 @@ contain a line that matches both expressions:
       cmd_argv.extend(opt.revision)
     cmd_argv.append('--')
 
+    git_failed = False
     bad_rev = False
     have_match = False
 
     for project in projects:
-      p = GitCommand(project,
-                     cmd_argv,
-                     bare = False,
-                     capture_stdout = True,
-                     capture_stderr = True)
+      try:
+        p = GitCommand(project,
+                       cmd_argv,
+                       bare=False,
+                       capture_stdout=True,
+                       capture_stderr=True)
+      except GitError as e:
+        git_failed = True
+        out.project('--- project %s ---' % project.relpath)
+        out.nl()
+        out.fail('%s', str(e))
+        out.nl()
+        continue
+
       if p.Wait() != 0:
         # no results
         #
@@ -202,7 +216,7 @@ contain a line that matches both expressions:
           else:
             out.project('--- project %s ---' % project.relpath)
             out.nl()
-            out.write("%s", p.stderr)
+            out.fail('%s', p.stderr.strip())
             out.nl()
         continue
       have_match = True
@@ -231,7 +245,9 @@ contain a line that matches both expressions:
         for line in r:
           print(line)
 
-    if have_match:
+    if git_failed:
+      sys.exit(1)
+    elif have_match:
       sys.exit(0)
     elif have_rev and bad_rev:
       for r in opt.revision:

subcmds/help.py

@@ -33,11 +33,8 @@ class Help(PagedCommand, MirrorSafeCommand):
 Displays detailed usage information about a command.
 """
 
-  def _PrintAllCommands(self):
-    print('usage: repo COMMAND [ARGS]')
-    print('The complete list of recognized repo commands are:')
-    commandNames = list(sorted(self.commands))
-
+  def _PrintCommands(self, commandNames):
+    """Helper to display |commandNames| summaries."""
     maxlen = 0
     for name in commandNames:
       maxlen = max(maxlen, len(name))
@@ -50,6 +47,12 @@ Displays detailed usage information about a command.
       except AttributeError:
         summary = ''
       print(fmt % (name, summary))
+
+  def _PrintAllCommands(self):
+    print('usage: repo COMMAND [ARGS]')
+    print('The complete list of recognized repo commands are:')
+    commandNames = list(sorted(self.commands))
+    self._PrintCommands(commandNames)
     print("See 'repo help <command>' for more information on a "
           'specific command.')
 
@@ -71,19 +74,8 @@ Displays detailed usage information about a command.
     commandNames = list(sorted([name
                     for name, command in self.commands.items()
                     if command.common and gitc_supported(command)]))
+    self._PrintCommands(commandNames)
 
-    maxlen = 0
-    for name in commandNames:
-      maxlen = max(maxlen, len(name))
-    fmt = '  %%-%ds  %%s' % maxlen
-
-    for name in commandNames:
-      command = self.commands[name]
-      try:
-        summary = command.helpSummary.strip()
-      except AttributeError:
-        summary = ''
-      print(fmt % (name, summary))
     print(
 "See 'repo help <command>' for more information on a specific command.\n"
 "See 'repo help --all' for a complete list of recognized commands.")

subcmds/info.py

@@ -16,7 +16,6 @@
 
 from command import PagedCommand
 from color import Coloring
-from error import NoSuchProjectError
 from git_refs import R_M
 
 class _Coloring(Coloring):
@@ -82,10 +81,8 @@ class Info(PagedCommand):
     self.out.nl()
 
   def printDiffInfo(self, args):
-    try:
-      projs = self.GetProjects(args)
-    except NoSuchProjectError:
-      return
+    # We let exceptions bubble up to main as they'll be well structured.
+    projs = self.GetProjects(args)
 
     for p in projs:
       self.heading("Project: ")
@@ -97,13 +94,23 @@ class Info(PagedCommand):
       self.out.nl()
 
       self.heading("Current revision: ")
+      self.headtext(p.GetRevisionId())
+      self.out.nl()
+
+      currentBranch = p.CurrentBranch
+      if currentBranch:
+        self.heading('Current branch: ')
+        self.headtext(currentBranch)
+        self.out.nl()
+
+      self.heading("Manifest revision: ")
       self.headtext(p.revisionExpr)
       self.out.nl()
 
       localBranches = list(p.GetBranches().keys())
       self.heading("Local Branches: ")
       self.redtext(str(len(localBranches)))
-      if len(localBranches) > 0:
+      if localBranches:
         self.text(" [")
         self.text(", ".join(localBranches))
         self.text("]")

subcmds/init.py

@@ -81,7 +81,7 @@ manifest, a subsequent `repo sync` (or `repo sync -d`) is necessary
 to update the working directory files.
 """
 
-  def _Options(self, p):
+  def _Options(self, p, gitc_init=False):
     # Logging
     g = p.add_option_group('Logging options')
     g.add_option('-q', '--quiet',
@@ -96,7 +96,12 @@ to update the working directory files.
     g.add_option('-b', '--manifest-branch',
                  dest='manifest_branch',
                  help='manifest branch or revision', metavar='REVISION')
-    g.add_option('--current-branch',
+    cbr_opts = ['--current-branch']
+    # The gitc-init subcommand allocates -c itself, but a lot of init users
+    # want -c, so try to satisfy both as best we can.
+    if not gitc_init:
+      cbr_opts += ['-c']
+    g.add_option(*cbr_opts,
                  dest='current_branch_only', action='store_true',
                  help='fetch only current manifest branch from server')
     g.add_option('-m', '--manifest-name',

subcmds/manifest.py

@@ -40,10 +40,9 @@ in a Git repository for use during future 'repo init' invocations.
     helptext = self._helpDescription + '\n'
     r = os.path.dirname(__file__)
     r = os.path.dirname(r)
-    fd = open(os.path.join(r, 'docs', 'manifest-format.md'))
-    for line in fd:
-      helptext += line
-    fd.close()
+    with open(os.path.join(r, 'docs', 'manifest-format.md')) as fd:
+      for line in fd:
+        helptext += line
     return helptext
 
   def _Options(self, p):

subcmds/prune.py

@@ -51,11 +51,16 @@ class Prune(PagedCommand):
         out.project('project %s/' % project.relpath)
         out.nl()
 
-      commits = branch.commits
-      date = branch.date
-      print('%s %-33s (%2d commit%s, %s)' % (
+      print('%s %-33s ' % (
             branch.name == project.CurrentBranch and '*' or ' ',
-            branch.name,
+            branch.name), end='')
+
+      if not branch.base_exists:
+        print('(ignoring: tracking branch is gone: %s)' % (branch.base,))
+      else:
+        commits = branch.commits
+        date = branch.date
+        print('(%2d commit%s, %s)' % (
             len(commits),
             len(commits) != 1 and 's' or ' ',
             date))

subcmds/rebase.py

@@ -17,9 +17,18 @@
 from __future__ import print_function
 import sys
 
+from color import Coloring
 from command import Command
 from git_command import GitCommand
 
+
+class RebaseColoring(Coloring):
+  def __init__(self, config):
+    Coloring.__init__(self, config, 'rebase')
+    self.project = self.printer('project', attr='bold')
+    self.fail = self.printer('fail', fg='red')
+
+
 class Rebase(Command):
   common = True
   helpSummary = "Rebase local branches on upstream branch"
@@ -37,6 +46,9 @@ branch but need to incorporate new upstream changes "underneath" them.
                 dest="interactive", action="store_true",
                 help="interactive rebase (single project only)")
 
+    p.add_option('--fail-fast',
+                 dest='fail_fast', action='store_true',
+                 help='Stop rebasing after first error is hit')
     p.add_option('-f', '--force-rebase',
                  dest='force_rebase', action='store_true',
                  help='Pass --force-rebase to git rebase')
@@ -88,7 +100,15 @@ branch but need to incorporate new upstream changes "underneath" them.
     if opt.interactive:
       common_args.append('-i')
 
+    config = self.manifest.manifestProject.config
+    out = RebaseColoring(config)
+    out.redirect(sys.stdout)
+
+    ret = 0
     for project in all_projects:
+      if ret and opt.fail_fast:
+        break
+
       cb = project.CurrentBranch
       if not cb:
         if one_project:
@@ -114,8 +134,10 @@ branch but need to incorporate new upstream changes "underneath" them.
 
       args.append(upbranch.LocalMerge)
 
-      print('# %s: rebasing %s -> %s'
-            % (project.relpath, cb, upbranch.LocalMerge), file=sys.stderr)
+      out.project('project %s: rebasing %s -> %s',
+                  project.relpath, cb, upbranch.LocalMerge)
+      out.nl()
+      out.flush()
 
       needs_stash = False
       if opt.auto_stash:
@@ -127,13 +149,21 @@ branch but need to incorporate new upstream changes "underneath" them.
           stash_args = ["stash"]
 
           if GitCommand(project, stash_args).Wait() != 0:
-            return 1
+            ret += 1
+            continue
 
       if GitCommand(project, args).Wait() != 0:
-        return 1
+        ret += 1
+        continue
 
       if needs_stash:
         stash_args.append('pop')
         stash_args.append('--quiet')
         if GitCommand(project, stash_args).Wait() != 0:
-          return 1
+          ret += 1
+
+    if ret:
+      out.fail('%i projects had errors', ret)
+      out.nl()
+
+    return ret

subcmds/sync.py

@@ -315,9 +315,6 @@ later is required to fix a server side protocol bug.
     # We'll set to true once we've locked the lock.
     did_lock = False
 
-    if not opt.quiet:
-      print('Fetching project %s' % project.name)
-
     # Encapsulate everything in a try/except/finally so that:
     # - We always set err_event in the case of an exception.
     # - We always make sure we unlock the lock if we locked it.
@@ -350,7 +347,7 @@ later is required to fix a server side protocol bug.
             raise _FetchError()
 
         fetched.add(project.gitdir)
-        pm.update()
+        pm.update(msg=project.name)
       except _FetchError:
         pass
       except Exception as e:
@@ -371,7 +368,6 @@ later is required to fix a server side protocol bug.
     fetched = set()
     lock = _threading.Lock()
     pm = Progress('Fetching projects', len(projects),
-                  print_newline=not(opt.quiet),
                   always_print_percentage=opt.quiet)
 
     objdir_project_map = dict()
@@ -440,7 +436,7 @@ later is required to fix a server side protocol bug.
     finally:
       sem.release()
 
-  def _CheckoutOne(self, opt, project, lock, pm, err_event):
+  def _CheckoutOne(self, opt, project, lock, pm, err_event, err_results):
     """Checkout work tree for one project
 
     Args:
@@ -452,6 +448,8 @@ later is required to fix a server side protocol bug.
           lock held).
       err_event: We'll set this event in the case of an error (after printing
           out info about the error).
+      err_results: A list of strings, paths to git repos where checkout
+          failed.
 
     Returns:
       Whether the fetch was successful.
@@ -459,9 +457,6 @@ later is required to fix a server side protocol bug.
     # We'll set to true once we've locked the lock.
     did_lock = False
 
-    if not opt.quiet:
-      print('Checking out project %s' % project.name)
-
     # Encapsulate everything in a try/except/finally so that:
     # - We always set err_event in the case of an exception.
     # - We always make sure we unlock the lock if we locked it.
@@ -472,11 +467,11 @@ later is required to fix a server side protocol bug.
     try:
       try:
         project.Sync_LocalHalf(syncbuf, force_sync=opt.force_sync)
-        success = syncbuf.Finish()
 
         # Lock around all the rest of the code, since printing, updating a set
         # and Progress.update() are not thread safe.
         lock.acquire()
+        success = syncbuf.Finish()
         did_lock = True
 
         if not success:
@@ -485,7 +480,7 @@ later is required to fix a server side protocol bug.
                 file=sys.stderr)
           raise _CheckoutError()
 
-        pm.update()
+        pm.update(msg=project.name)
       except _CheckoutError:
         pass
       except Exception as e:
@@ -496,6 +491,8 @@ later is required to fix a server side protocol bug.
         raise
     finally:
       if did_lock:
+        if not success:
+          err_results.append(project.relpath)
         lock.release()
       finish = time.time()
       self.event_log.AddSync(project, event_log.TASK_SYNC_LOCAL,
@@ -523,11 +520,12 @@ later is required to fix a server side protocol bug.
       syncjobs = 1
 
     lock = _threading.Lock()
-    pm = Progress('Syncing work tree', len(all_projects))
+    pm = Progress('Checking out projects', len(all_projects))
 
     threads = set()
     sem = _threading.Semaphore(syncjobs)
     err_event = _threading.Event()
+    err_results = []
 
     for project in all_projects:
       # Check for any errors before running any more tasks.
@@ -542,7 +540,8 @@ later is required to fix a server side protocol bug.
                       project=project,
                       lock=lock,
                       pm=pm,
-                      err_event=err_event)
+                      err_event=err_event,
+                      err_results=err_results)
         if syncjobs > 1:
           t = _threading.Thread(target=self._CheckoutWorker,
                                 kwargs=kwargs)
@@ -560,6 +559,9 @@ later is required to fix a server side protocol bug.
     # If we saw an error, exit with code 1 so that other scripts can check.
     if err_event.isSet():
       print('\nerror: Exited sync due to checkout errors', file=sys.stderr)
+      if err_results:
+        print('Failing repos:\n%s' % '\n'.join(err_results),
+              file=sys.stderr)
       sys.exit(1)
 
   def _GCProjects(self, projects):
@@ -692,11 +694,8 @@ later is required to fix a server side protocol bug.
     old_project_paths = []
 
     if os.path.exists(file_path):
-      fd = open(file_path, 'r')
-      try:
+      with open(file_path, 'r') as fd:
         old_project_paths = fd.read().split('\n')
-      finally:
-        fd.close()
       # In reversed order, so subfolders are deleted before parent folder.
       for path in sorted(old_project_paths, reverse=True):
         if not path:
@@ -731,12 +730,9 @@ later is required to fix a server side protocol bug.
               return 1
 
     new_project_paths.sort()
-    fd = open(file_path, 'w')
-    try:
+    with open(file_path, 'w') as fd:
       fd.write('\n'.join(new_project_paths))
       fd.write('\n')
-    finally:
-      fd.close()
     return 0
 
   def _SmartSyncSetup(self, opt, smart_sync_manifest_path):
@@ -809,11 +805,8 @@ later is required to fix a server side protocol bug.
       if success:
         manifest_name = os.path.basename(smart_sync_manifest_path)
         try:
-          f = open(smart_sync_manifest_path, 'w')
-          try:
+          with open(smart_sync_manifest_path, 'w') as f:
             f.write(manifest_str)
-          finally:
-            f.close()
         except IOError as e:
           print('error: cannot write manifest to %s:\n%s'
                 % (smart_sync_manifest_path, e),
@@ -836,6 +829,33 @@ later is required to fix a server side protocol bug.
 
     return manifest_name
 
+  def _UpdateManifestProject(self, opt, mp, manifest_name):
+    """Fetch & update the local manifest project."""
+    if not opt.local_only:
+      start = time.time()
+      success = mp.Sync_NetworkHalf(quiet=opt.quiet,
+                                    current_branch_only=opt.current_branch_only,
+                                    no_tags=opt.no_tags,
+                                    optimized_fetch=opt.optimized_fetch,
+                                    submodules=self.manifest.HasSubmodules,
+                                    clone_filter=self.manifest.CloneFilter)
+      finish = time.time()
+      self.event_log.AddSync(mp, event_log.TASK_SYNC_NETWORK,
+                             start, finish, success)
+
+    if mp.HasChanges:
+      syncbuf = SyncBuffer(mp.config)
+      start = time.time()
+      mp.Sync_LocalHalf(syncbuf, submodules=self.manifest.HasSubmodules)
+      clean = syncbuf.Finish()
+      self.event_log.AddSync(mp, event_log.TASK_SYNC_LOCAL,
+                             start, time.time(), clean)
+      if not clean:
+        sys.exit(1)
+      self._ReloadManifest(opt.manifest_name)
+      if opt.jobs is None:
+        self.jobs = self.manifest.default.sync_j
+
   def ValidateOptions(self, opt, args):
     if opt.force_broken:
       print('warning: -f/--force-broken is now the default behavior, and the '
@@ -887,30 +907,7 @@ later is required to fix a server side protocol bug.
     if opt.repo_upgraded:
       _PostRepoUpgrade(self.manifest, quiet=opt.quiet)
 
-    if not opt.local_only:
-      start = time.time()
-      success = mp.Sync_NetworkHalf(quiet=opt.quiet,
-                                    current_branch_only=opt.current_branch_only,
-                                    no_tags=opt.no_tags,
-                                    optimized_fetch=opt.optimized_fetch,
-                                    submodules=self.manifest.HasSubmodules,
-                                    clone_filter=self.manifest.CloneFilter)
-      finish = time.time()
-      self.event_log.AddSync(mp, event_log.TASK_SYNC_NETWORK,
-                             start, finish, success)
-
-    if mp.HasChanges:
-      syncbuf = SyncBuffer(mp.config)
-      start = time.time()
-      mp.Sync_LocalHalf(syncbuf, submodules=self.manifest.HasSubmodules)
-      clean = syncbuf.Finish()
-      self.event_log.AddSync(mp, event_log.TASK_SYNC_LOCAL,
-                             start, time.time(), clean)
-      if not clean:
-        sys.exit(1)
-      self._ReloadManifest(manifest_name)
-      if opt.jobs is None:
-        self.jobs = self.manifest.default.sync_j
+    self._UpdateManifestProject(opt, mp, manifest_name)
 
     if self.gitc_manifest:
       gitc_manifest_projects = self.GetProjects(args,
@@ -1098,11 +1095,8 @@ class _FetchTimes(object):
   def _Load(self):
     if self._times is None:
       try:
-        f = open(self._path)
-        try:
+        with open(self._path) as f:
           self._times = json.load(f)
-        finally:
-          f.close()
       except (IOError, ValueError):
         try:
           platform_utils.remove(self._path)
@@ -1122,11 +1116,8 @@ class _FetchTimes(object):
       del self._times[name]
 
     try:
-      f = open(self._path, 'w')
-      try:
+      with open(self._path, 'w') as f:
         json.dump(self._times, f, indent=2)
-      finally:
-        f.close()
     except (IOError, TypeError):
       try:
         platform_utils.remove(self._path)

subcmds/upload.py

@@ -271,11 +271,6 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
       branches[project.name] = b
     script.append('')
 
-    script = [ x.encode('utf-8')
-             if issubclass(type(x), unicode)
-             else x
-             for x in script ]
-
     script = Editor.EditString("\n".join(script)).split("\n")
 
     project_re = re.compile(r'^#?\s*project\s*([^\s]+)/:$')

subcmds/version.py

@@ -17,7 +17,7 @@
 from __future__ import print_function
 import sys
 from command import Command, MirrorSafeCommand
-from git_command import git
+from git_command import git, RepoSourceVersion, user_agent
 from git_refs import HEAD
 
 class Version(Command, MirrorSafeCommand):
@@ -34,12 +34,20 @@ class Version(Command, MirrorSafeCommand):
     rp = self.manifest.repoProject
     rem = rp.GetRemote(rp.remote.name)
 
-    print('repo version %s' % rp.work_git.describe(HEAD))
+    # These might not be the same.  Report them both.
+    src_ver = RepoSourceVersion()
+    rp_ver = rp.bare_git.describe(HEAD)
+    print('repo version %s' % rp_ver)
     print('       (from %s)' % rem.url)
 
     if Version.wrapper_path is not None:
       print('repo launcher version %s' % Version.wrapper_version)
       print('       (from %s)' % Version.wrapper_path)
 
+      if src_ver != rp_ver:
+        print('       (currently at %s)' % src_ver)
+
+    print('repo User-Agent %s' % user_agent.repo)
     print('git %s' % git.version_tuple().full)
+    print('git User-Agent %s' % user_agent.git)
     print('Python %s' % sys.version)

tests/test_editor.py

@@ -0,0 +1,60 @@
+# -*- coding:utf-8 -*-
+#
+# Copyright (C) 2019 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unittests for the editor.py module."""
+
+from __future__ import print_function
+
+import unittest
+
+from editor import Editor
+
+
+class EditorTestCase(unittest.TestCase):
+  """Take care of resetting Editor state across tests."""
+
+  def setUp(self):
+    self.setEditor(None)
+
+  def tearDown(self):
+    self.setEditor(None)
+
+  @staticmethod
+  def setEditor(editor):
+    Editor._editor = editor
+
+
+class GetEditor(EditorTestCase):
+  """Check GetEditor behavior."""
+
+  def test_basic(self):
+    """Basic checking of _GetEditor."""
+    self.setEditor(':')
+    self.assertEqual(':', Editor._GetEditor())
+
+
+class EditString(EditorTestCase):
+  """Check EditString behavior."""
+
+  def test_no_editor(self):
+    """Check behavior when no editor is available."""
+    self.setEditor(':')
+    self.assertEqual('foo', Editor.EditString('foo'))
+
+  def test_cat_editor(self):
+    """Check behavior when editor is `cat`."""
+    self.setEditor('cat')
+    self.assertEqual('foo', Editor.EditString('foo'))

tests/test_git_command.py

@@ -18,6 +18,7 @@
 
 from __future__ import print_function
 
+import re
 import unittest
 
 import git_command
@@ -47,3 +48,31 @@ class GitCallUnitTest(unittest.TestCase):
     self.assertLess(ver, (9999, 9999, 9999))
 
     self.assertNotEqual('', ver.full)
+
+
+class UserAgentUnitTest(unittest.TestCase):
+  """Tests the UserAgent function."""
+
+  def test_smoke_os(self):
+    """Make sure UA OS setting returns something useful."""
+    os_name = git_command.user_agent.os
+    # We can't dive too deep because of OS/tool differences, but we can check
+    # the general form.
+    m = re.match(r'^[^ ]+$', os_name)
+    self.assertIsNotNone(m)
+
+  def test_smoke_repo(self):
+    """Make sure repo UA returns something useful."""
+    ua = git_command.user_agent.repo
+    # We can't dive too deep because of OS/tool differences, but we can check
+    # the general form.
+    m = re.match(r'^git-repo/[^ ]+ ([^ ]+) git/[^ ]+ Python/[0-9.]+', ua)
+    self.assertIsNotNone(m)
+
+  def test_smoke_git(self):
+    """Make sure git UA returns something useful."""
+    ua = git_command.user_agent.git
+    # We can't dive too deep because of OS/tool differences, but we can check
+    # the general form.
+    m = re.match(r'^git/[^ ]+ ([^ ]+) git-repo/[^ ]+', ua)
+    self.assertIsNotNone(m)

tests/test_project.py

@@ -18,11 +18,30 @@
 
 from __future__ import print_function
 
+import contextlib
+import os
+import shutil
+import subprocess
+import tempfile
 import unittest
 
+import git_config
 import project
 
 
+@contextlib.contextmanager
+def TempGitTree():
+  """Create a new empty git checkout for testing."""
+  # TODO(vapier): Convert this to tempfile.TemporaryDirectory once we drop
+  # Python 2 support entirely.
+  try:
+    tempdir = tempfile.mkdtemp(prefix='repo-tests')
+    subprocess.check_call(['git', 'init'], cwd=tempdir)
+    yield tempdir
+  finally:
+    shutil.rmtree(tempdir)
+
+
 class RepoHookShebang(unittest.TestCase):
   """Check shebang parsing in RepoHook."""
 
@@ -60,3 +79,58 @@ class RepoHookShebang(unittest.TestCase):
     for shebang, interp in DATA:
       self.assertEqual(project.RepoHook._ExtractInterpFromShebang(shebang),
                        interp)
+
+
+class FakeProject(object):
+  """A fake for Project for basic functionality."""
+
+  def __init__(self, worktree):
+    self.worktree = worktree
+    self.gitdir = os.path.join(worktree, '.git')
+    self.name = 'fakeproject'
+    self.work_git = project.Project._GitGetByExec(
+        self, bare=False, gitdir=self.gitdir)
+    self.bare_git = project.Project._GitGetByExec(
+        self, bare=True, gitdir=self.gitdir)
+    self.config = git_config.GitConfig.ForRepository(gitdir=self.gitdir)
+
+
+class ReviewableBranchTests(unittest.TestCase):
+  """Check ReviewableBranch behavior."""
+
+  def test_smoke(self):
+    """A quick run through everything."""
+    with TempGitTree() as tempdir:
+      fakeproj = FakeProject(tempdir)
+
+      # Generate some commits.
+      with open(os.path.join(tempdir, 'readme'), 'w') as fp:
+        fp.write('txt')
+      fakeproj.work_git.add('readme')
+      fakeproj.work_git.commit('-mAdd file')
+      fakeproj.work_git.checkout('-b', 'work')
+      fakeproj.work_git.rm('-f', 'readme')
+      fakeproj.work_git.commit('-mDel file')
+
+      # Start off with the normal details.
+      rb = project.ReviewableBranch(
+          fakeproj, fakeproj.config.GetBranch('work'), 'master')
+      self.assertEqual('work', rb.name)
+      self.assertEqual(1, len(rb.commits))
+      self.assertIn('Del file', rb.commits[0])
+      d = rb.unabbrev_commits
+      self.assertEqual(1, len(d))
+      short, long = next(iter(d.items()))
+      self.assertTrue(long.startswith(short))
+      self.assertTrue(rb.base_exists)
+      # Hard to assert anything useful about this.
+      self.assertTrue(rb.date)
+
+      # Now delete the tracking branch!
+      fakeproj.work_git.branch('-D', 'master')
+      rb = project.ReviewableBranch(
+          fakeproj, fakeproj.config.GetBranch('work'), 'master')
+      self.assertEqual(0, len(rb.commits))
+      self.assertFalse(rb.base_exists)
+      # Hard to assert anything useful about this.
+      self.assertTrue(rb.date)

tox.ini

@@ -0,0 +1,22 @@
+# Copyright 2019 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# https://tox.readthedocs.io/
+
+[tox]
+envlist = py27, py36, py37, py38
+
+[testenv]
+deps = pytest
+commands = {toxinidir}/run_tests