project: Check if dotgit exists w/out symlink check

os.path.exists returns false on a broken symlink. This is not what repo
needs when checking if a project is setup properly.

For example, if src/foo/.git can't be resolved, repo tries to create
symlink and that results in FileExistsError.

Use lexists which returns True even if symlink is broken. That will
force path where repo checks where symlink is pointing to and will fix
it to the correct location.

Bug: b/281746795
Change-Id: Id3f7dc3a3cb6499d02ce7335eca992ddc7deb645
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415197
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: George Engelbrecht <engeg@google.com>
Reviewed-by: Greg Edelston <gredelston@google.com>

.github/workflows/test-ci.yml

@@ -13,8 +13,9 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-latest, macos-latest, windows-latest]
+        # ubuntu-20.04 is the last version that supports python 3.6
-        python-version: ['3.6', '3.7', '3.8', '3.9', '3.10']
+        os: [ubuntu-20.04, macos-latest, windows-latest]
+        python-version: ['3.6', '3.7', '3.8', '3.9', '3.10', '3.11', '3.12']
     runs-on: ${{ matrix.os }}
 
     steps:

color.py

@@ -103,7 +103,7 @@ def SetDefaultColoring(state):
         DEFAULT = "never"
 
 
-class Coloring(object):
+class Coloring:
     def __init__(self, config, section_type):
         self._section = "color.%s" % section_type
         self._config = config
@@ -194,7 +194,7 @@ class Coloring(object):
         if not opt:
             return _Color(fg, bg, attr)
 
-        v = self._config.GetString("%s.%s" % (self._section, opt))
+        v = self._config.GetString(f"{self._section}.{opt}")
         if v is None:
             return _Color(fg, bg, attr)
 

command.py

@@ -46,7 +46,7 @@ class UsageError(RepoExitError):
     """Exception thrown with invalid command usage."""
 
 
-class Command(object):
+class Command:
     """Base class for any command line action in repo."""
 
     # Singleton for all commands to track overall repo command execution and
@@ -290,7 +290,7 @@ class Command(object):
                 output.end()
 
     def _ResetPathToProjectMap(self, projects):
-        self._by_path = dict((p.worktree, p) for p in projects)
+        self._by_path = {p.worktree: p for p in projects}
 
     def _UpdatePathToProjectMap(self, project):
         self._by_path[project.worktree] = project
@@ -476,8 +476,7 @@ class Command(object):
             top = self.manifest
         yield top
         if not opt.this_manifest_only:
-            for child in top.all_children:
+            yield from top.all_children
-                yield child
 
 
 class InteractiveCommand(Command):
@@ -498,11 +497,11 @@ class PagedCommand(Command):
         return True
 
 
-class MirrorSafeCommand(object):
+class MirrorSafeCommand:
     """Command permits itself to run within a mirror, and does not require a
     working directory.
     """
 
 
-class GitcClientCommand(object):
+class GitcClientCommand:
     """Command that requires the local client to be a GITC client."""

docs/python-support.md

@@ -1,47 +1,92 @@
 # Supported Python Versions
 
-With Python 2.7 officially going EOL on [01 Jan 2020](https://pythonclock.org/),
+This documents the current supported Python versions, and tries to provide
-we need a support plan for the repo project itself.
+guidance for when we decide to drop support for older versions.
-Inevitably, there will be a long tail of users who still want to use Python 2 on
-their old LTS/corp systems and have little power to change the system.
 
 ## Summary
 
-* Python 3.6 (released Dec 2016) is required by default starting with repo-2.x.
+*   Python 3.6 (released Dec 2016) is required starting with repo-2.0.
-* Older versions of Python (e.g. v2.7) may use the legacy feature-frozen branch
+*   Older versions of Python (e.g. v2.7) may use old releases via the repo-1.x
-  based on repo-1.x.
+    branch, but no support is provided.
 
-## Overview
+## repo hooks
-
-We provide a branch for Python 2 users that is feature-frozen.
-Bugfixes may be added on a best-effort basis or from the community, but largely
-no new features will be added, nor is support guaranteed.
-
-Users can select this during `repo init` time via the [repo launcher].
-Otherwise the default branches (e.g. stable & main) will be used which will
-require Python 3.
-
-This means the [repo launcher] needs to support both Python 2 & Python 3, but
-since it doesn't import any other repo code, this shouldn't be too problematic.
-
-The main branch will require Python 3.6 at a minimum.
-If the system has an older version of Python 3, then users will have to select
-the legacy Python 2 branch instead.
-
-### repo hooks
 
 Projects that use [repo hooks] run on independent schedules.
-They might migrate to Python 3 earlier or later than us.
+Since it's not possible to detect what version of Python the hooks were written
-To support them, we'll probe the shebang of the hook script and if we find an
+or tested against, we always import & exec them with the active Python version.
-interpreter in there that indicates a different version than repo is currently
-running under, we'll attempt to reexec ourselves under that.
 
-For example, a hook with a header like `#!/usr/bin/python2` will have repo
+If the user's Python is too new for the [repo hooks], then it is up to the hooks
-execute `/usr/bin/python2` to execute the hook code specifically if repo is
+maintainer to update.
-currently running Python 3.
 
-For more details, consult the [repo hooks] documentation.
+## Repo launcher
+
+The [repo launcher] is an independent script that can support older versions of
+Python without holding back the rest of the codebase.
+If it detects the current version of Python is too old, it will try to reexec
+via a newer version of Python via standard `pythonX.Y` interpreter names.
+
+However, this is provided as a nicety when it is not onerous, and there is no
+official support for older versions of Python than the rest of the codebase.
+
+If your default python interpreters are too old to run the launcher even though
+you have newer versions installed, your choices are:
+
+*   Modify the [repo launcher]'s shebang to suite your environment.
+*   Download an older version of the [repo launcher] and don't upgrade it.
+    Be aware that we do not guarantee old repo launchers will work with current
+    versions of repo.  Bug reports using old launchers will not be accepted.
+
+## When to drop support
+
+So far, Python 3.6 has provided most of the interesting features that we want
+(e.g. typing & f-strings), and there haven't been features in newer versions
+that are critical to us.
+
+That said, let's assume we need functionality that only exists in Python 3.7.
+How do we decide when it's acceptable to drop Python 3.6?
+
+1.  Review the [Project References](./release-process.md#project-references) to
+    see what major distros are using the previous version of Python, and when
+    they go EOL.  Generally we care about Ubuntu LTS & current/previous Debian
+    stable versions.
+    *   If they're all EOL already, then go for it, drop support.
+    *   If they aren't EOL, start a thread on [repo-discuss] to see how the user
+        base feels about the proposal.
+1.  Update the "soft" versions in the codebase.  This will start warning users
+    that the older version is deprecated.
+    *   Update [repo](/repo) if the launcher needs updating.
+        This only helps with people who download newer launchers.
+    *   Update [main.py](/main.py) for the main codebase.
+        This warns for everyone regardless of [repo launcher] version.
+    *   Update [requirements.json](/requirements.json).
+        This allows [repo launcher] to display warnings/errors without having
+        to execute the new codebase.  This helps in case of syntax or module
+        changes where older versions won't even be able to import the new code.
+1.  After some grace period (ideally at least 2 quarters after the first release
+    with the updated soft requirements), update the "hard" versions, and then
+    start using the new functionality.
+
+## Python 2.7 & 3.0-3.5
+
+> **There is no support for these versions.**
+> **Do not file bugs if you are using old Python versions.**
+> **Any such reports will be marked invalid and ignored.**
+> **Upgrade your distro and/or runtime instead.**
+
+Fetch an old version of the [repo launcher]:
+
+```sh
+$ curl https://storage.googleapis.com/git-repo-downloads/repo-2.32 > ~/.bin/repo-2.32
+$ chmod a+rx ~/.bin/repo-2.32
+```
+
+Then initialize an old version of repo:
+
+```sh
+$ repo-2.32 init --repo-rev=repo-1 ...
+```
 
 
+[repo-discuss]: https://groups.google.com/forum/#!forum/repo-discuss
 [repo hooks]: ./repo-hooks.md
 [repo launcher]: ../repo

docs/release-process.md

@@ -202,7 +202,7 @@ still support them.
 Things in italics are things we used to care about but probably don't anymore.
 
 |   Date   |     EOL      | [Git][rel-g] | [Python][rel-p] | [SSH][rel-o] | [Ubuntu][rel-u] / [Debian][rel-d] | Git | Python | SSH |
-|:--------:|:------------:|:------------:|:---------------:|:------------:|-----------------------------------|-----|--------|-----|
+|:--------:|:------------:|:------------:|:---------------:|:------------:|-----------------------------------|:---:|:------:|:---:|
 | Apr 2008 |              |              |                 | 5.0          |
 | Jun 2008 |              |              |                 | 5.1          |
 | Oct 2008 | *Oct 2013*   |              | 2.6.0           |              | *10.04 Lucid* - 10.10 Maverick / *Squeeze* |
@@ -241,7 +241,7 @@ Things in italics are things we used to care about but probably don't anymore.
 | Feb 2014 | *Dec 2014*   | **1.9.0**    |                 |              | *14.04 Trusty* |
 | Mar 2014 | *Mar 2019*   |              | *3.4.0*         |              | *14.04 Trusty* - 15.10 Wily / *Jessie* |
 | Mar 2014 |              |              |                 | 6.6          | *14.04 Trusty* - 14.10 Utopic |
-| Apr 2014 | *Apr 2022*   |              |                 |              | *14.04 Trusty*       | 1.9.1    | 2.7.5 3.4.0  | 6.6 |
+| Apr 2014 | *Apr 2024*   |              |                 |              | *14.04 Trusty*       | 1.9.1    | 2.7.5 3.4.0  | 6.6 |
 | May 2014 | *Dec 2014*   | 2.0.0        |
 | Aug 2014 | *Dec 2014*   | *2.1.0*      |                 |              | 14.10 Utopic - 15.04 Vivid / *Jessie* |
 | Oct 2014 |              |              |                 | 6.7          | 15.04 Vivid |
@@ -262,7 +262,7 @@ Things in italics are things we used to care about but probably don't anymore.
 | Jan 2016 | *Jul 2017*   | *2.7.0*      |                 |              | *16.04 Xenial* |
 | Feb 2016 |              |              |                 | 7.2          | *16.04 Xenial* |
 | Mar 2016 | *Jul 2017*   | 2.8.0        |
-| Apr 2016 | *Apr 2024*   |              |                 |              | *16.04 Xenial*       | 2.7.4    | 2.7.11 3.5.1 | 7.2 |
+| Apr 2016 | *Apr 2026*   |              |                 |              | *16.04 Xenial*       | 2.7.4    | 2.7.11 3.5.1 | 7.2 |
 | Jun 2016 | *Jul 2017*   | 2.9.0        |                 |              | 16.10 Yakkety |
 | Jul 2016 |              |              |                 | 7.3          | 16.10 Yakkety |
 | Sep 2016 | *Sep 2017*   | 2.10.0       |
@@ -312,14 +312,33 @@ Things in italics are things we used to care about but probably don't anymore.
 | Oct 2020 |              |              |                 |              | 20.10 Groovy         | 2.27.0   | 2.7.18 3.8.6 | 8.3 |
 | Oct 2020 | **Oct 2025** |              | 3.9.0           |              | 21.04 Hirsute / **Bullseye** |
 | Dec 2020 | *Mar 2021*   | 2.30.0       |                 |              | 21.04 Hirsute / **Bullseye** |
-| Mar 2021 |              | 2.31.0       |
+| Mar 2021 |              | 2.31.0       |                 | 8.5          |
-| Mar 2021 |              |              |                 | 8.5          |
 | Apr 2021 |              |              |                 | 8.6          |
 | Apr 2021 | *Jan 2022*   |              |                 |              | 21.04 Hirsute        | 2.30.2   | 2.7.18 3.9.4 | 8.4 |
 | Jun 2021 |              | 2.32.0       |
-| Aug 2021 |              | 2.33.0       |
+| Aug 2021 |              | 2.33.0       |                 | 8.7          |
-| Aug 2021 |              |              |                 | 8.7          |
 | Aug 2021 | **Aug 2026** |              |                 |              | **Debian 11 Bullseye** | 2.30.2 | 2.7.18 3.9.2 | 8.4 |
+| Sep 2021 |              |              |                 | 8.8          |
+| Oct 2021 |              | 2.34.0       | 3.10.0          |              | **22.04 Jammy** |
+| Jan 2022 |              | 2.35.0       |
+| Feb 2022 |              |              |                 | 8.9          | **22.04 Jammy** |
+| Apr 2022 |              | 2.36.0       |                 | 9.0          |
+| Apr 2022 | **Apr 2032** |              |                 |              | **22.04 Jammy**      | 2.34.1 | 2.7.18 3.10.6 | 8.9 |
+| Jun 2022 |              | 2.37.0       |
+| Oct 2022 |              | 2.38.0       |                 | 9.1          |
+| Oct 2022 |              |              | 3.11.0          |              | **Bookworm** |
+| Dec 2022 |              | 2.39.0       |                 |              | **Bookworm** |
+| Feb 2023 |              |              |                 | 9.2          | **Bookworm** |
+| Mar 2023 |              | 2.40.0       |                 | 9.3          |
+| Jun 2023 |              | 2.41.0       |
+| Jun 2023 | **Jun 2028** |              |                 |              | **Debian 12 Bookworm** | 2.39.2 | 3.11.2 | 9.2 |
+| Aug 2023 |              | 2.42.0       |                 | 9.4          |
+| Oct 2023 |              |              | 3.12.0          | 9.5          |
+| Nov 2022 |              | 2.43.0       |
+| Dec 2023 |              |              |                 | 9.6          |
+| Feb 2024 |              | 2.44.0       |
+| Mar 2024 |              |              |                 | 9.7          |
+| Oct 2024 |              |              | 3.13.0          |
 | **Date** |   **EOL**    | **[Git][rel-g]** | **[Python][rel-p]** | **[SSH][rel-o]** | **[Ubuntu][rel-u] / [Debian][rel-d]** | **Git** | **Python** | **SSH** |
 
 
@@ -328,7 +347,7 @@ Things in italics are things we used to care about but probably don't anymore.
 [rel-g]: https://en.wikipedia.org/wiki/Git#Releases
 [rel-o]: https://www.openssh.com/releasenotes.html
 [rel-p]: https://en.wikipedia.org/wiki/History_of_Python#Table_of_versions
-[rel-u]: https://en.wikipedia.org/wiki/Ubuntu_version_history#Table_of_versions
+[rel-u]: https://wiki.ubuntu.com/Releases
 [example announcement]: https://groups.google.com/d/topic/repo-discuss/UGBNismWo1M/discussion
 [repo-discuss@googlegroups.com]: https://groups.google.com/forum/#!forum/repo-discuss
 [go/repo-release]: https://goto.google.com/repo-release

editor.py

@@ -22,7 +22,7 @@ from error import EditorError
 import platform_utils
 
 
-class Editor(object):
+class Editor:
     """Manages the user's preferred text editor."""
 
     _editor = None
@@ -104,9 +104,7 @@ least one of these before using this command.""",  # noqa: E501
             try:
                 rc = subprocess.Popen(args, shell=shell).wait()
             except OSError as e:
-                raise EditorError(
+                raise EditorError(f"editor failed, {str(e)}: {editor} {path}")
-                    "editor failed, %s: %s %s" % (str(e), editor, path)
-                )
             if rc != 0:
                 raise EditorError(
                     "editor failed with exit status %d: %s %s"

event_log.py

@@ -21,7 +21,7 @@ TASK_SYNC_NETWORK = "sync-network"
 TASK_SYNC_LOCAL = "sync-local"
 
 
-class EventLog(object):
+class EventLog:
     """Event log that records events that occurred during a repo invocation.
 
     Events are written to the log as a consecutive JSON entries, one per line.

git_command.py

@@ -15,6 +15,7 @@
 import functools
 import json
 import os
+import re
 import subprocess
 import sys
 from typing import Any, Optional
@@ -24,6 +25,7 @@ from error import RepoExitError
 from git_refs import HEAD
 from git_trace2_event_log_base import BaseEventLog
 import platform_utils
+from repo_logging import RepoLogger
 from repo_trace import IsTrace
 from repo_trace import REPO_TRACE
 from repo_trace import Trace
@@ -50,17 +52,19 @@ DEFAULT_GIT_FAIL_MESSAGE = "git command failure"
 ERROR_EVENT_LOGGING_PREFIX = "RepoGitCommandError"
 # Common line length limit
 GIT_ERROR_STDOUT_LINES = 1
-GIT_ERROR_STDERR_LINES = 1
+GIT_ERROR_STDERR_LINES = 10
 INVALID_GIT_EXIT_CODE = 126
 
+logger = RepoLogger(__file__)
 
-class _GitCall(object):
+
+class _GitCall:
     @functools.lru_cache(maxsize=None)
     def version_tuple(self):
         ret = Wrapper().ParseGitVersion()
         if ret is None:
             msg = "fatal: unable to detect git version"
-            print(msg, file=sys.stderr)
+            logger.error(msg)
             raise GitRequireError(msg)
         return ret
 
@@ -131,19 +135,22 @@ def GetEventTargetPath():
     if retval == 0:
         # Strip trailing carriage-return in path.
         path = p.stdout.rstrip("\n")
+        if path == "":
+            return None
     elif retval != 1:
         # `git config --get` is documented to produce an exit status of `1`
         # if the requested variable is not present in the configuration.
         # Report any other return value as an error.
-        print(
+        logger.error(
             "repo: error: 'git config --get' call failed with return code: "
-            "%r, stderr: %r" % (retval, p.stderr),
+            "%r, stderr: %r",
-            file=sys.stderr,
+            retval,
+            p.stderr,
         )
     return path
 
 
-class UserAgent(object):
+class UserAgent:
     """Mange User-Agent settings when talking to external services
 
     We follow the style as documented here:
@@ -191,12 +198,10 @@ class UserAgent(object):
     def git(self):
         """The UA when running git."""
         if self._git_ua is None:
-            self._git_ua = "git/%s (%s) git-repo/%s" % (
+            self._git_ua = (
-                git.version_tuple().full,
+                f"git/{git.version_tuple().full} ({self.os}) "
-                self.os,
+                f"git-repo/{RepoSourceVersion()}"
-                RepoSourceVersion(),
             )
-
         return self._git_ua
 
 
@@ -211,8 +216,8 @@ def git_require(min_version, fail=False, msg=""):
         need = ".".join(map(str, min_version))
         if msg:
             msg = " for " + msg
-        error_msg = "fatal: git %s or later required%s" % (need, msg)
+        error_msg = f"fatal: git {need} or later required{msg}"
-        print(error_msg, file=sys.stderr)
+        logger.error(error_msg)
         raise GitRequireError(error_msg)
     return False
 
@@ -238,7 +243,7 @@ def _build_env(
         env["GIT_SSH"] = ssh_proxy.proxy
         env["GIT_SSH_VARIANT"] = "ssh"
     if "http_proxy" in env and "darwin" == sys.platform:
-        s = "'http.proxy=%s'" % (env["http_proxy"],)
+        s = f"'http.proxy={env['http_proxy']}'"
         p = env.get("GIT_CONFIG_PARAMETERS")
         if p is not None:
             s = p + " " + s
@@ -267,7 +272,7 @@ def _build_env(
     return env
 
 
-class GitCommand(object):
+class GitCommand:
     """Wrapper around a single git invocation."""
 
     def __init__(
@@ -297,6 +302,7 @@ class GitCommand(object):
         self.project = project
         self.cmdv = cmdv
         self.verify_command = verify_command
+        self.stdout, self.stderr = None, None
 
         # Git on Windows wants its paths only using / for reliability.
         if platform_utils.isWindows():
@@ -326,14 +332,6 @@ class GitCommand(object):
                 command.append("--progress")
         command.extend(cmdv[1:])
 
-        stdin = subprocess.PIPE if input else None
-        stdout = subprocess.PIPE if capture_stdout else None
-        stderr = (
-            subprocess.STDOUT
-            if merge_output
-            else (subprocess.PIPE if capture_stderr else None)
-        )
-
         event_log = (
             BaseEventLog(env=env, add_init_count=True)
             if add_event_log
@@ -344,9 +342,9 @@ class GitCommand(object):
             self._RunCommand(
                 command,
                 env,
-                stdin=stdin,
+                capture_stdout=capture_stdout,
-                stdout=stdout,
+                capture_stderr=capture_stderr,
-                stderr=stderr,
+                merge_output=merge_output,
                 ssh_proxy=ssh_proxy,
                 cwd=cwd,
                 input=input,
@@ -377,13 +375,46 @@ class GitCommand(object):
         self,
         command,
         env,
-        stdin=None,
+        capture_stdout=False,
-        stdout=None,
+        capture_stderr=False,
-        stderr=None,
+        merge_output=False,
         ssh_proxy=None,
         cwd=None,
         input=None,
     ):
+        # Set subprocess.PIPE for streams that need to be captured.
+        stdin = subprocess.PIPE if input else None
+        stdout = subprocess.PIPE if capture_stdout else None
+        stderr = (
+            subprocess.STDOUT
+            if merge_output
+            else (subprocess.PIPE if capture_stderr else None)
+        )
+
+        # tee_stderr acts like a tee command for stderr, in that, it captures
+        # stderr from the subprocess and streams it back to sys.stderr, while
+        # keeping a copy in-memory.
+        # This allows us to store stderr logs from the subprocess into
+        # GitCommandError.
+        # Certain git operations, such as `git push`, writes diagnostic logs,
+        # such as, progress bar for pushing, into stderr. To ensure we don't
+        # break git's UX, we need to write to sys.stderr as we read from the
+        # subprocess. Setting encoding or errors makes subprocess return
+        # io.TextIOWrapper, which is line buffered. To avoid line-buffering
+        # while tee-ing stderr, we unset these kwargs. See GitCommand._Tee
+        # for tee-ing between the streams.
+        # We tee stderr iff the caller doesn't want to capture any stream to
+        # not disrupt the existing flow.
+        # See go/tee-repo-stderr for more context.
+        tee_stderr = False
+        kwargs = {"encoding": "utf-8", "errors": "backslashreplace"}
+        if not (stdin or stdout or stderr):
+            tee_stderr = True
+            # stderr will be written back to sys.stderr even though it is
+            # piped here.
+            stderr = subprocess.PIPE
+            kwargs = {}
+
         dbg = ""
         if IsTrace():
             global LAST_CWD
@@ -430,15 +461,14 @@ class GitCommand(object):
                     command,
                     cwd=cwd,
                     env=env,
-                    encoding="utf-8",
-                    errors="backslashreplace",
                     stdin=stdin,
                     stdout=stdout,
                     stderr=stderr,
+                    **kwargs,
                 )
             except Exception as e:
                 raise GitPopenCommandError(
-                    message="%s: %s" % (command[1], e),
+                    message=f"{command[1]}: {e}",
                     project=self.project.name if self.project else None,
                     command_args=self.cmdv,
                 )
@@ -449,12 +479,45 @@ class GitCommand(object):
             self.process = p
 
             try:
-                self.stdout, self.stderr = p.communicate(input=input)
+                if tee_stderr:
+                    # tee_stderr streams stderr to sys.stderr while capturing
+                    # a copy within self.stderr. tee_stderr is only enabled
+                    # when the caller wants to pipe no stream.
+                    self.stderr = self._Tee(p.stderr, sys.stderr)
+                else:
+                    self.stdout, self.stderr = p.communicate(input=input)
             finally:
                 if ssh_proxy:
                     ssh_proxy.remove_client(p)
             self.rc = p.wait()
 
+    @staticmethod
+    def _Tee(in_stream, out_stream):
+        """Writes text from in_stream to out_stream while recording in buffer.
+
+        Args:
+            in_stream: I/O stream to be read from.
+            out_stream: I/O stream to write to.
+
+        Returns:
+            A str containing everything read from the in_stream.
+        """
+        buffer = ""
+        read_size = 1024 if sys.version_info < (3, 7) else -1
+        chunk = in_stream.read1(read_size)
+        while chunk:
+            # Convert to str.
+            if not hasattr(chunk, "encode"):
+                chunk = chunk.decode("utf-8", "backslashreplace")
+
+            buffer += chunk
+            out_stream.write(chunk)
+            out_stream.flush()
+
+            chunk = in_stream.read1(read_size)
+
+        return buffer
+
     @staticmethod
     def _GetBasicEnv():
         """Return a basic env for running git under.
@@ -517,6 +580,29 @@ class GitCommandError(GitError):
     raised exclusively from non-zero exit codes returned from git commands.
     """
 
+    # Tuples with error formats and suggestions for those errors.
+    _ERROR_TO_SUGGESTION = [
+        (
+            re.compile("couldn't find remote ref .*"),
+            "Check if the provided ref exists in the remote.",
+        ),
+        (
+            re.compile("unable to access '.*': .*"),
+            (
+                "Please make sure you have the correct access rights and the "
+                "repository exists."
+            ),
+        ),
+        (
+            re.compile("'.*' does not appear to be a git repository"),
+            "Are you running this repo command outside of a repo workspace?",
+        ),
+        (
+            re.compile("not a git repository"),
+            "Are you running this repo command outside of a repo workspace?",
+        ),
+    ]
+
     def __init__(
         self,
         message: str = DEFAULT_GIT_FAIL_MESSAGE,
@@ -533,16 +619,37 @@ class GitCommandError(GitError):
         self.git_stdout = git_stdout
         self.git_stderr = git_stderr
 
+    @property
+    @functools.lru_cache(maxsize=None)
+    def suggestion(self):
+        """Returns helpful next steps for the given stderr."""
+        if not self.git_stderr:
+            return self.git_stderr
+
+        for err, suggestion in self._ERROR_TO_SUGGESTION:
+            if err.search(self.git_stderr):
+                return suggestion
+
+        return None
+
     def __str__(self):
         args = "[]" if not self.command_args else " ".join(self.command_args)
         error_type = type(self).__name__
-        return f"""{error_type}: {self.message}
+        string = f"{error_type}: '{args}' on {self.project} failed"
-    Project: {self.project}
+
-    Args: {args}
+        if self.message != DEFAULT_GIT_FAIL_MESSAGE:
-    Stdout:
+            string += f": {self.message}"
-{self.git_stdout}
+
-    Stderr:
+        if self.git_stdout:
-{self.git_stderr}"""
+            string += f"\nstdout: {self.git_stdout}"
+
+        if self.git_stderr:
+            string += f"\nstderr: {self.git_stderr}"
+
+        if self.suggestion:
+            string += f"\nsuggestion: {self.suggestion}"
+
+        return string
 
 
 class GitPopenCommandError(GitError):

git_config.py

@@ -70,7 +70,7 @@ def _key(name):
     return ".".join(parts)
 
 
-class GitConfig(object):
+class GitConfig:
     _ForUser = None
 
     _ForSystem = None
@@ -180,7 +180,7 @@ class GitConfig(object):
             config_dict[key] = self.GetString(key)
         return config_dict
 
-    def GetBoolean(self, name: str) -> Union[str, None]:
+    def GetBoolean(self, name: str) -> Union[bool, None]:
         """Returns a boolean from the configuration file.
 
         Returns:
@@ -370,7 +370,7 @@ class GitConfig(object):
             with Trace(": parsing %s", self.file):
                 with open(self._json) as fd:
                     return json.load(fd)
-        except (IOError, ValueError):
+        except (OSError, ValueError):
             platform_utils.remove(self._json, missing_ok=True)
             return None
 
@@ -378,7 +378,7 @@ class GitConfig(object):
         try:
             with open(self._json, "w") as fd:
                 json.dump(cache, fd, indent=2)
-        except (IOError, TypeError):
+        except (OSError, TypeError):
             platform_utils.remove(self._json, missing_ok=True)
 
     def _ReadGit(self):
@@ -418,7 +418,7 @@ class GitConfig(object):
         if p.Wait() == 0:
             return p.stdout
         else:
-            raise GitError("git config %s: %s" % (str(args), p.stderr))
+            raise GitError(f"git config {str(args)}: {p.stderr}")
 
 
 class RepoConfig(GitConfig):
@@ -430,7 +430,7 @@ class RepoConfig(GitConfig):
         return os.path.join(repo_config_dir, ".repoconfig/config")
 
 
-class RefSpec(object):
+class RefSpec:
     """A Git refspec line, split into its components:
 
     forced:  True if the line starts with '+'
@@ -541,7 +541,7 @@ def GetUrlCookieFile(url, quiet):
     yield cookiefile, None
 
 
-class Remote(object):
+class Remote:
     """Configuration options related to a remote."""
 
     def __init__(self, config, name):
@@ -651,13 +651,11 @@ class Remote(object):
                             userEmail, host, port
                         )
                 except urllib.error.HTTPError as e:
-                    raise UploadError("%s: %s" % (self.review, str(e)))
+                    raise UploadError(f"{self.review}: {str(e)}")
                 except urllib.error.URLError as e:
-                    raise UploadError("%s: %s" % (self.review, str(e)))
+                    raise UploadError(f"{self.review}: {str(e)}")
                 except http.client.HTTPException as e:
-                    raise UploadError(
+                    raise UploadError(f"{self.review}: {e.__class__.__name__}")
-                        "%s: %s" % (self.review, e.__class__.__name__)
-                    )
 
                 REVIEW_CACHE[u] = self._review_url
         return self._review_url + self.projectname
@@ -666,7 +664,7 @@ class Remote(object):
         username = self._config.GetString("review.%s.username" % self.review)
         if username is None:
             username = userEmail.split("@")[0]
-        return "ssh://%s@%s:%s/" % (username, host, port)
+        return f"ssh://{username}@{host}:{port}/"
 
     def ToLocal(self, rev):
         """Convert a remote revision string to something we have locally."""
@@ -715,15 +713,15 @@ class Remote(object):
         self._Set("fetch", list(map(str, self.fetch)))
 
     def _Set(self, key, value):
-        key = "remote.%s.%s" % (self.name, key)
+        key = f"remote.{self.name}.{key}"
         return self._config.SetString(key, value)
 
     def _Get(self, key, all_keys=False):
-        key = "remote.%s.%s" % (self.name, key)
+        key = f"remote.{self.name}.{key}"
         return self._config.GetString(key, all_keys=all_keys)
 
 
-class Branch(object):
+class Branch:
     """Configuration options related to a single branch."""
 
     def __init__(self, config, name):
@@ -762,11 +760,11 @@ class Branch(object):
                     fd.write("\tmerge = %s\n" % self.merge)
 
     def _Set(self, key, value):
-        key = "branch.%s.%s" % (self.name, key)
+        key = f"branch.{self.name}.{key}"
         return self._config.SetString(key, value)
 
     def _Get(self, key, all_keys=False):
-        key = "branch.%s.%s" % (self.name, key)
+        key = f"branch.{self.name}.{key}"
         return self._config.GetString(key, all_keys=all_keys)
 
 

git_refs.py

@@ -28,7 +28,7 @@ R_WORKTREE_M = R_WORKTREE + "m/"
 R_M = "refs/remotes/m/"
 
 
-class GitRefs(object):
+class GitRefs:
     def __init__(self, gitdir):
         self._gitdir = gitdir
         self._phyref = None
@@ -105,10 +105,8 @@ class GitRefs(object):
     def _ReadPackedRefs(self):
         path = os.path.join(self._gitdir, "packed-refs")
         try:
-            fd = open(path, "r")
+            fd = open(path)
             mtime = os.path.getmtime(path)
-        except IOError:
-            return
         except OSError:
             return
         try:

git_superproject.py

@@ -66,12 +66,12 @@ class UpdateProjectsResult(NamedTuple):
     fatal: bool
 
 
-class Superproject(object):
+class Superproject:
     """Get commit ids from superproject.
 
-    Initializes a local copy of a superproject for the manifest. This allows
+    Initializes a bare local copy of a superproject for the manifest. This
-    lookup of commit ids for all projects. It contains _project_commit_ids which
+    allows lookup of commit ids for all projects. It contains
-    is a dictionary with project/commit id entries.
+    _project_commit_ids which is a dictionary with project/commit id entries.
     """
 
     def __init__(
@@ -235,7 +235,8 @@ class Superproject(object):
         p = GitCommand(
             None,
             cmd,
-            cwd=self._work_git,
+            gitdir=self._work_git,
+            bare=True,
             capture_stdout=True,
             capture_stderr=True,
         )
@@ -271,7 +272,8 @@ class Superproject(object):
         p = GitCommand(
             None,
             cmd,
-            cwd=self._work_git,
+            gitdir=self._work_git,
+            bare=True,
             capture_stdout=True,
             capture_stderr=True,
         )
@@ -381,7 +383,7 @@ class Superproject(object):
         try:
             with open(manifest_path, "w", encoding="utf-8") as fp:
                 fp.write(manifest_str)
-        except IOError as e:
+        except OSError as e:
             self._LogError("cannot write manifest to : {} {}", manifest_path, e)
             return None
         return manifest_path

git_trace2_event_log_base.py

@@ -38,11 +38,13 @@ import tempfile
 import threading
 
 
+# Timeout when sending events via socket (applies to connect, send)
+SOCK_TIMEOUT = 0.5  # in seconds
 # BaseEventLog __init__ Counter that is consistent within the same process
 p_init_count = 0
 
 
-class BaseEventLog(object):
+class BaseEventLog:
     """Event log that records events that occurred during a repo invocation.
 
     Events are written to the log as a consecutive JSON entries, one per line.
@@ -76,9 +78,8 @@ class BaseEventLog(object):
         # Save both our sid component and the complete sid.
         # We use our sid component (self._sid) as the unique filename prefix and
         # the full sid (self._full_sid) in the log itself.
-        self._sid = "repo-%s-P%08x" % (
+        self._sid = (
-            self.start.strftime("%Y%m%dT%H%M%SZ"),
+            f"repo-{self.start.strftime('%Y%m%dT%H%M%SZ')}-P{os.getpid():08x}"
-            os.getpid(),
         )
 
         if add_init_count:
@@ -297,6 +298,7 @@ class BaseEventLog(object):
                     with socket.socket(
                         socket.AF_UNIX, socket.SOCK_STREAM
                     ) as sock:
+                        sock.settimeout(SOCK_TIMEOUT)
                         sock.connect(path)
                         self._WriteLog(sock.sendall)
                     return f"af_unix:stream:{path}"

hooks.py

@@ -12,11 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import errno
-import json
 import os
 import re
-import subprocess
 import sys
 import traceback
 import urllib.parse
@@ -25,7 +22,7 @@ from error import HookError
 from git_refs import HEAD
 
 
-class RepoHook(object):
+class RepoHook:
     """A RepoHook contains information about a script to run as a hook.
 
     Hooks are used to run a python script before running an upload (for
@@ -183,7 +180,7 @@ class RepoHook(object):
                 abort_if_user_denies was passed to the consturctor.
         """
         hooks_config = self._hooks_project.config
-        git_approval_key = "repo.hooks.%s.%s" % (self._hook_type, subkey)
+        git_approval_key = f"repo.hooks.{self._hook_type}.{subkey}"
 
         # Get the last value that the user approved for this hook; may be None.
         old_val = hooks_config.GetString(git_approval_key)
@@ -196,7 +193,7 @@ class RepoHook(object):
             else:
                 # Give the user a reason why we're prompting, since they last
                 # told us to "never ask again".
-                prompt = "WARNING: %s\n\n" % (changed_prompt,)
+                prompt = f"WARNING: {changed_prompt}\n\n"
         else:
             prompt = ""
 
@@ -244,9 +241,8 @@ class RepoHook(object):
         return self._CheckForHookApprovalHelper(
             "approvedmanifest",
             self._manifest_url,
-            "Run hook scripts from %s" % (self._manifest_url,),
+            f"Run hook scripts from {self._manifest_url}",
-            "Manifest URL has changed since %s was allowed."
+            f"Manifest URL has changed since {self._hook_type} was allowed.",
-            % (self._hook_type,),
         )
 
     def _CheckForHookApprovalHash(self):
@@ -265,7 +261,7 @@ class RepoHook(object):
             "approvedhash",
             self._GetHash(),
             prompt % (self._GetMustVerb(), self._script_fullpath),
-            "Scripts have changed since %s was allowed." % (self._hook_type,),
+            f"Scripts have changed since {self._hook_type} was allowed.",
         )
 
     @staticmethod
@@ -298,43 +294,6 @@ class RepoHook(object):
 
         return interp
 
-    def _ExecuteHookViaReexec(self, interp, context, **kwargs):
-        """Execute the hook script through |interp|.
-
-        Note: Support for this feature should be dropped ~Jun 2021.
-
-        Args:
-            interp: The Python program to run.
-            context: Basic Python context to execute the hook inside.
-            kwargs: Arbitrary arguments to pass to the hook script.
-
-        Raises:
-            HookError: When the hooks failed for any reason.
-        """
-        # This logic needs to be kept in sync with _ExecuteHookViaImport below.
-        script = """
-import json, os, sys
-path = '''%(path)s'''
-kwargs = json.loads('''%(kwargs)s''')
-context = json.loads('''%(context)s''')
-sys.path.insert(0, os.path.dirname(path))
-data = open(path).read()
-exec(compile(data, path, 'exec'), context)
-context['main'](**kwargs)
-""" % {
-            "path": self._script_fullpath,
-            "kwargs": json.dumps(kwargs),
-            "context": json.dumps(context),
-        }
-
-        # We pass the script via stdin to avoid OS argv limits.  It also makes
-        # unhandled exception tracebacks less verbose/confusing for users.
-        cmd = [interp, "-c", "import sys; exec(sys.stdin.read())"]
-        proc = subprocess.Popen(cmd, stdin=subprocess.PIPE)
-        proc.communicate(input=script.encode("utf-8"))
-        if proc.returncode:
-            raise HookError("Failed to run %s hook." % (self._hook_type,))
-
     def _ExecuteHookViaImport(self, data, context, **kwargs):
         """Execute the hook code in |data| directly.
 
@@ -412,30 +371,13 @@ context['main'](**kwargs)
             # See what version of python the hook has been written against.
             data = open(self._script_fullpath).read()
             interp = self._ExtractInterpFromShebang(data)
-            reexec = False
             if interp:
                 prog = os.path.basename(interp)
-                if prog.startswith("python2") and sys.version_info.major != 2:
+                if prog.startswith("python2"):
-                    reexec = True
+                    raise HookError("Python 2 is not supported")
-                elif prog.startswith("python3") and sys.version_info.major == 2:
-                    reexec = True
-
-            # Attempt to execute the hooks through the requested version of
-            # Python.
-            if reexec:
-                try:
-                    self._ExecuteHookViaReexec(interp, context, **kwargs)
-                except OSError as e:
-                    if e.errno == errno.ENOENT:
-                        # We couldn't find the interpreter, so fallback to
-                        # importing.
-                        reexec = False
-                    else:
-                        raise
 
             # Run the hook by importing directly.
-            if not reexec:
+            self._ExecuteHookViaImport(data, context, **kwargs)
-                self._ExecuteHookViaImport(data, context, **kwargs)
         finally:
             # Restore sys.path and CWD.
             sys.path = orig_syspath

hooks/commit-msg

@@ -1,5 +1,5 @@
 #!/bin/sh
-# From Gerrit Code Review 3.6.1 c67916dbdc07555c44e32a68f92ffc484b9b34f0
+# From Gerrit Code Review 3.10.0 d5403dbf335ba7d48977fc95170c3f7027c34659
 #
 # Part of Gerrit Code Review (https://www.gerritcodereview.com/)
 #
@@ -31,14 +31,21 @@ if test ! -f "$1" ; then
 fi
 
 # Do not create a change id if requested
-if test "false" = "$(git config --bool --get gerrit.createChangeId)" ; then
+create_setting=$(git config --get gerrit.createChangeId)
-  exit 0
+case "$create_setting" in
-fi
+  false)
+    exit 0
+    ;;
+  always)
+    ;;
+  *)
+    # Do not create a change id for squash/fixup commits.
+    if head -n1 "$1" | LC_ALL=C grep -q '^[a-z][a-z]*! '; then
+      exit 0
+    fi
+    ;;
+esac
 
-# Do not create a change id for squash commits.
-if head -n1 "$1" | grep -q '^squash! '; then
-  exit 0
-fi
 
 if git rev-parse --verify HEAD >/dev/null 2>&1; then
   refhash="$(git rev-parse HEAD)"
@@ -51,7 +58,7 @@ dest="$1.tmp.${random}"
 
 trap 'rm -f "$dest" "$dest-2"' EXIT
 
-if ! git stripspace --strip-comments < "$1" > "${dest}" ; then
+if ! cat "$1" | sed -e '/>8/q' | git stripspace --strip-comments > "${dest}" ; then
    echo "cannot strip comments from $1"
    exit 1
 fi
@@ -65,7 +72,7 @@ reviewurl="$(git config --get gerrit.reviewUrl)"
 if test -n "${reviewurl}" ; then
   token="Link"
   value="${reviewurl%/}/id/I$random"
-  pattern=".*/id/I[0-9a-f]\{40\}$"
+  pattern=".*/id/I[0-9a-f]\{40\}"
 else
   token="Change-Id"
   value="I$random"
@@ -92,7 +99,7 @@ fi
 # Avoid the --where option which only appeared in Git 2.15
 if ! git -c trailer.where=before interpret-trailers \
          --trailer "Signed-off-by: $token: $value" < "$dest-2" |
-     sed -re "s/^Signed-off-by: ($token: )/\1/" \
+     sed -e "s/^Signed-off-by: \($token: \)/\1/" \
          -e "/^Signed-off-by: SENTINEL/d" > "$dest" ; then
   echo "cannot insert $token line in $1"
   exit 1

main.py

@@ -48,6 +48,7 @@ from error import DownloadError
 from error import GitcUnsupportedError
 from error import InvalidProjectGroupsError
 from error import ManifestInvalidRevisionError
+from error import ManifestParseError
 from error import NoManifestException
 from error import NoSuchProjectError
 from error import RepoChangedException
@@ -86,27 +87,19 @@ logger = RepoLogger(__file__)
 MIN_PYTHON_VERSION_SOFT = (3, 6)
 MIN_PYTHON_VERSION_HARD = (3, 6)
 
-if sys.version_info.major < 3:
+if sys.version_info < MIN_PYTHON_VERSION_HARD:
     logger.error(
-        "repo: error: Python 2 is no longer supported; "
+        "repo: error: Python version is too old; "
         "Please upgrade to Python %d.%d+.",
         *MIN_PYTHON_VERSION_SOFT,
     )
     sys.exit(1)
-else:
+elif sys.version_info < MIN_PYTHON_VERSION_SOFT:
-    if sys.version_info < MIN_PYTHON_VERSION_HARD:
+    logger.error(
-        logger.error(
+        "repo: warning: your Python version is no longer supported; "
-            "repo: error: Python 3 version is too old; "
+        "Please upgrade to Python %d.%d+.",
-            "Please upgrade to Python %d.%d+.",
+        *MIN_PYTHON_VERSION_SOFT,
-            *MIN_PYTHON_VERSION_SOFT,
+    )
-        )
-        sys.exit(1)
-    elif sys.version_info < MIN_PYTHON_VERSION_SOFT:
-        logger.error(
-            "repo: warning: your Python 3 version is no longer supported; "
-            "Please upgrade to Python %d.%d+.",
-            *MIN_PYTHON_VERSION_SOFT,
-        )
 
 KEYBOARD_INTERRUPT_EXIT = 128 + signal.SIGINT
 MAX_PRINT_ERRORS = 5
@@ -194,7 +187,7 @@ global_options.add_option(
 )
 
 
-class _Repo(object):
+class _Repo:
     def __init__(self, repodir):
         self.repodir = repodir
         self.commands = all_commands
@@ -206,9 +199,8 @@ class _Repo(object):
         if short:
             commands = " ".join(sorted(self.commands))
             wrapped_commands = textwrap.wrap(commands, width=77)
-            print(
+            help_commands = "".join(f"\n  {x}" for x in wrapped_commands)
-                "Available commands:\n  %s" % ("\n  ".join(wrapped_commands),)
+            print(f"Available commands:{help_commands}")
-            )
             print("\nRun `repo help <command>` for command-specific details.")
             print("Bug reports:", Wrapper().BUG_URL)
         else:
@@ -244,7 +236,7 @@ class _Repo(object):
         if name in self.commands:
             return name, []
 
-        key = "alias.%s" % (name,)
+        key = f"alias.{name}"
         alias = RepoConfig.ForRepository(self.repodir).GetString(key)
         if alias is None:
             alias = RepoConfig.ForUser().GetString(key)
@@ -278,10 +270,14 @@ class _Repo(object):
             self._PrintHelp(short=True)
             return 1
 
-        run = lambda: self._RunLong(name, gopts, argv) or 0
+        git_trace2_event_log = EventLog()
+        run = (
+            lambda: self._RunLong(name, gopts, argv, git_trace2_event_log) or 0
+        )
         with Trace(
-            "starting new command: %s",
+            "starting new command: %s [sid=%s]",
             ", ".join([name] + argv),
+            git_trace2_event_log.full_sid,
             first_trace=True,
         ):
             if gopts.trace_python:
@@ -298,12 +294,11 @@ class _Repo(object):
                 result = run()
         return result
 
-    def _RunLong(self, name, gopts, argv):
+    def _RunLong(self, name, gopts, argv, git_trace2_event_log):
         """Execute the (longer running) requested subcommand."""
         result = 0
         SetDefaultColoring(gopts.color)
 
-        git_trace2_event_log = EventLog()
         outer_client = RepoClient(self.repodir)
         repo_client = outer_client
         if gopts.submanifest_path:
@@ -430,7 +425,7 @@ class _Repo(object):
                         error_info = json.dumps(
                             {
                                 "ErrorType": type(error).__name__,
-                                "Project": project,
+                                "Project": str(project),
                                 "Message": str(error),
                             }
                         )
@@ -448,6 +443,7 @@ class _Repo(object):
         except (
             DownloadError,
             ManifestInvalidRevisionError,
+            ManifestParseError,
             NoManifestException,
         ) as e:
             logger.error("error: in `%s`: %s", " ".join([name] + argv), e)
@@ -566,9 +562,11 @@ repo: error:
         sys.exit(1)
 
     if exp > ver:
-        logger.warn("\n... A new version of repo (%s) is available.", exp_str)
+        logger.warning(
+            "\n... A new version of repo (%s) is available.", exp_str
+        )
         if os.access(repo_path, os.W_OK):
-            logger.warn(
+            logger.warning(
                 """\
 ... You should upgrade soon:
     cp %s %s
@@ -577,7 +575,7 @@ repo: error:
                 repo_path,
             )
         else:
-            logger.warn(
+            logger.warning(
                 """\
 ... New version is available at: %s
 ... The launcher is run from: %s
@@ -795,7 +793,7 @@ def init_http():
             mgr.add_password(p[1], "https://%s/" % host, p[0], p[2])
     except netrc.NetrcParseError:
         pass
-    except IOError:
+    except OSError:
         pass
     handlers.append(_BasicAuthHandler(mgr))
     handlers.append(_DigestAuthHandler(mgr))

manifest_xml.py

@@ -114,12 +114,40 @@ def XmlInt(node, attr, default=None):
     try:
         return int(value)
     except ValueError:
-        raise ManifestParseError(
+        raise ManifestParseError(f'manifest: invalid {attr}="{value}" integer')
-            'manifest: invalid %s="%s" integer' % (attr, value)
-        )
 
 
-class _Default(object):
+def normalize_url(url: str) -> str:
+    """Mutate input 'url' into normalized form:
+
+    * remove trailing slashes
+    * convert SCP-like syntax to SSH URL
+
+    Args:
+        url: URL to modify
+
+    Returns:
+        The normalized URL.
+    """
+
+    url = url.rstrip("/")
+    parsed_url = urllib.parse.urlparse(url)
+
+    # This matches patterns like "git@github.com:foo".
+    scp_like_url_re = r"^[^/:]+@[^/:]+:[^/]+"
+
+    # If our URL is missing a schema and matches git's
+    # SCP-like syntax we should convert it to a proper
+    # SSH URL instead to make urljoin() happier.
+    #
+    # See: https://git-scm.com/docs/git-clone#URLS
+    if not parsed_url.scheme and re.match(scp_like_url_re, url):
+        return "ssh://" + url.replace(":", "/", 1)
+
+    return url
+
+
+class _Default:
     """Project defaults within the manifest."""
 
     revisionExpr = None
@@ -142,7 +170,7 @@ class _Default(object):
         return self.__dict__ != other.__dict__
 
 
-class _XmlRemote(object):
+class _XmlRemote:
     def __init__(
         self,
         name,
@@ -182,20 +210,22 @@ class _XmlRemote(object):
     def _resolveFetchUrl(self):
         if self.fetchUrl is None:
             return ""
-        url = self.fetchUrl.rstrip("/")
-        manifestUrl = self.manifestUrl.rstrip("/")
-        # urljoin will gets confused over quite a few things.  The ones we care
-        # about here are:
-        # * no scheme in the base url, like <hostname:port>
-        # We handle no scheme by replacing it with an obscure protocol, gopher
-        # and then replacing it with the original when we are done.
 
-        if manifestUrl.find(":") != manifestUrl.find("/") - 1:
+        fetch_url = normalize_url(self.fetchUrl)
-            url = urllib.parse.urljoin("gopher://" + manifestUrl, url)
+        manifest_url = normalize_url(self.manifestUrl)
-            url = re.sub(r"^gopher://", "", url)
+
+        # urljoin doesn't like URLs with no scheme in the base URL
+        # such as file paths.  We handle this by prefixing it with
+        # an obscure protocol, gopher, and replacing it with the
+        # original after urljoin
+        if manifest_url.find(":") != manifest_url.find("/") - 1:
+            fetch_url = urllib.parse.urljoin(
+                "gopher://" + manifest_url, fetch_url
+            )
+            fetch_url = re.sub(r"^gopher://", "", fetch_url)
         else:
-            url = urllib.parse.urljoin(manifestUrl, url)
+            fetch_url = urllib.parse.urljoin(manifest_url, fetch_url)
-        return url
+        return fetch_url
 
     def ToRemoteSpec(self, projectName):
         fetchUrl = self.resolvedFetchUrl.rstrip("/")
@@ -275,7 +305,7 @@ class _XmlSubmanifest:
             parent.repodir,
             linkFile,
             parent_groups=",".join(groups) or "",
-            submanifest_path=self.relpath,
+            submanifest_path=os.path.join(parent.path_prefix, self.relpath),
             outer_client=outer_client,
             default_groups=default_groups,
         )
@@ -354,7 +384,7 @@ class SubmanifestSpec:
         self.groups = groups or []
 
 
-class XmlManifest(object):
+class XmlManifest:
     """manages the repo configuration file"""
 
     def __init__(
@@ -727,10 +757,10 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
             self._output_manifest_project_extras(p, e)
 
             if p.subprojects:
-                subprojects = set(subp.name for subp in p.subprojects)
+                subprojects = {subp.name for subp in p.subprojects}
                 output_projects(p, e, list(sorted(subprojects)))
 
-        projects = set(p.name for p in self._paths.values() if not p.parent)
+        projects = {p.name for p in self._paths.values() if not p.parent}
         output_projects(None, root, list(sorted(projects)))
 
         if self._repo_hooks_project:
@@ -800,17 +830,17 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
             for child in node.childNodes:
                 if child.nodeType == xml.dom.Node.ELEMENT_NODE:
                     attrs = child.attributes
-                    element = dict(
+                    element = {
-                        (attrs.item(i).localName, attrs.item(i).value)
+                        attrs.item(i).localName: attrs.item(i).value
                         for i in range(attrs.length)
-                    )
+                    }
                     if child.nodeName in SINGLE_ELEMENTS:
                         ret[child.nodeName] = element
                     elif child.nodeName in MULTI_ELEMENTS:
                         ret.setdefault(child.nodeName, []).append(element)
                     else:
                         raise ManifestParseError(
-                            'Unhandled element "%s"' % (child.nodeName,)
+                            f'Unhandled element "{child.nodeName}"'
                         )
 
                     append_children(element, child)
@@ -857,8 +887,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         self._Load()
         outer = self._outer_client
         yield outer
-        for tree in outer.all_children:
+        yield from outer.all_children
-            yield tree
 
     @property
     def all_children(self):
@@ -867,8 +896,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         for child in self._submanifests.values():
             if child.repo_client:
                 yield child.repo_client
-                for tree in child.repo_client.all_children:
+                yield from child.repo_client.all_children
-                    yield tree
 
     @property
     def path_prefix(self):
@@ -987,7 +1015,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
     @property
     def PartialCloneExclude(self):
         exclude = self.manifest.manifestProject.partial_clone_exclude or ""
-        return set(x.strip() for x in exclude.split(","))
+        return {x.strip() for x in exclude.split(",")}
 
     def SetManifestOverride(self, path):
         """Override manifestFile.  The caller must call Unload()"""
@@ -1260,18 +1288,19 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         try:
             root = xml.dom.minidom.parse(path)
         except (OSError, xml.parsers.expat.ExpatError) as e:
-            raise ManifestParseError(
+            raise ManifestParseError(f"error parsing manifest {path}: {e}")
-                "error parsing manifest %s: %s" % (path, e)
-            )
 
         if not root or not root.childNodes:
-            raise ManifestParseError("no root node in %s" % (path,))
+            raise ManifestParseError(f"no root node in {path}")
 
         for manifest in root.childNodes:
-            if manifest.nodeName == "manifest":
+            if (
+                manifest.nodeType == manifest.ELEMENT_NODE
+                and manifest.nodeName == "manifest"
+            ):
                 break
         else:
-            raise ManifestParseError("no <manifest> in %s" % (path,))
+            raise ManifestParseError(f"no <manifest> in {path}")
 
         nodes = []
         for node in manifest.childNodes:
@@ -1281,7 +1310,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                     msg = self._CheckLocalPath(name)
                     if msg:
                         raise ManifestInvalidPathError(
-                            '<include> invalid "name": %s: %s' % (name, msg)
+                            f'<include> invalid "name": {name}: {msg}'
                         )
                 include_groups = ""
                 if parent_groups:
@@ -1313,7 +1342,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                     raise
                 except Exception as e:
                     raise ManifestParseError(
-                        "failed parsing included manifest %s: %s" % (name, e)
+                        f"failed parsing included manifest {name}: {e}"
                     )
             else:
                 if parent_groups and node.nodeName == "project":
@@ -1764,13 +1793,13 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                 msg = self._CheckLocalPath(name)
                 if msg:
                     raise ManifestInvalidPathError(
-                        '<submanifest> invalid "name": %s: %s' % (name, msg)
+                        f'<submanifest> invalid "name": {name}: {msg}'
                     )
         else:
             msg = self._CheckLocalPath(path)
             if msg:
                 raise ManifestInvalidPathError(
-                    '<submanifest> invalid "path": %s: %s' % (path, msg)
+                    f'<submanifest> invalid "path": {path}: {msg}'
                 )
 
         submanifest = _XmlSubmanifest(
@@ -1805,7 +1834,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         msg = self._CheckLocalPath(name, dir_ok=True)
         if msg:
             raise ManifestInvalidPathError(
-                '<project> invalid "name": %s: %s' % (name, msg)
+                f'<project> invalid "name": {name}: {msg}'
             )
         if parent:
             name = self._JoinName(parent.name, name)
@@ -1815,7 +1844,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
             remote = self._default.remote
         if remote is None:
             raise ManifestParseError(
-                "no remote for project %s within %s" % (name, self.manifestFile)
+                f"no remote for project {name} within {self.manifestFile}"
             )
 
         revisionExpr = node.getAttribute("revision") or remote.revision
@@ -1836,7 +1865,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
             msg = self._CheckLocalPath(path, dir_ok=True, cwd_dot_ok=True)
             if msg:
                 raise ManifestInvalidPathError(
-                    '<project> invalid "path": %s: %s' % (path, msg)
+                    f'<project> invalid "path": {path}: {msg}'
                 )
 
         rebase = XmlBool(node, "rebase", True)
@@ -2093,7 +2122,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         if not cwd_dot_ok or parts != ["."]:
             for part in set(parts):
                 if part in {".", "..", ".git"} or part.startswith(".repo"):
-                    return "bad component: %s" % (part,)
+                    return f"bad component: {part}"
 
         if not dir_ok and resep.match(path[-1]):
             return "dirs not allowed"
@@ -2129,7 +2158,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         msg = cls._CheckLocalPath(dest)
         if msg:
             raise ManifestInvalidPathError(
-                '<%s> invalid "dest": %s: %s' % (element, dest, msg)
+                f'<{element}> invalid "dest": {dest}: {msg}'
             )
 
         # |src| is the file we read from or path we point to for symlinks.
@@ -2140,7 +2169,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         )
         if msg:
             raise ManifestInvalidPathError(
-                '<%s> invalid "src": %s: %s' % (element, src, msg)
+                f'<{element}> invalid "src": {src}: {msg}'
             )
 
     def _ParseCopyFile(self, project, node):
@@ -2184,7 +2213,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         v = self._remotes.get(name)
         if not v:
             raise ManifestParseError(
-                "remote %s not defined in %s" % (name, self.manifestFile)
+                f"remote {name} not defined in {self.manifestFile}"
             )
         return v
 

platform_utils.py

@@ -57,8 +57,8 @@ def _validate_winpath(path):
     if _winpath_is_valid(path):
         return path
     raise ValueError(
-        'Path "{}" must be a relative path or an absolute '
+        f'Path "{path}" must be a relative path or an absolute '
-        "path starting with a drive letter".format(path)
+        "path starting with a drive letter"
     )
 
 
@@ -193,10 +193,9 @@ def _walk_windows_impl(top, topdown, onerror, followlinks):
     for name in dirs:
         new_path = os.path.join(top, name)
         if followlinks or not islink(new_path):
-            for x in _walk_windows_impl(
+            yield from _walk_windows_impl(
                 new_path, topdown, onerror, followlinks
-            ):
+            )
-                yield x
     if not topdown:
         yield top, dirs, nondirs
 

platform_utils_win32.py

@@ -186,9 +186,7 @@ def _create_symlink(source, link_name, dwFlags):
             error_desc = FormatError(code).strip()
             if code == ERROR_PRIVILEGE_NOT_HELD:
                 raise OSError(errno.EPERM, error_desc, link_name)
-            _raise_winerror(
+            _raise_winerror(code, f'Error creating symbolic link "{link_name}"')
-                code, 'Error creating symbolic link "{}"'.format(link_name)
-            )
 
 
 def islink(path):
@@ -210,7 +208,7 @@ def readlink(path):
     )
     if reparse_point_handle == INVALID_HANDLE_VALUE:
         _raise_winerror(
-            get_last_error(), 'Error opening symbolic link "{}"'.format(path)
+            get_last_error(), f'Error opening symbolic link "{path}"'
         )
     target_buffer = c_buffer(MAXIMUM_REPARSE_DATA_BUFFER_SIZE)
     n_bytes_returned = DWORD()
@@ -227,7 +225,7 @@ def readlink(path):
     CloseHandle(reparse_point_handle)
     if not io_result:
         _raise_winerror(
-            get_last_error(), 'Error reading symbolic link "{}"'.format(path)
+            get_last_error(), f'Error reading symbolic link "{path}"'
         )
     rdb = REPARSE_DATA_BUFFER.from_buffer(target_buffer)
     if rdb.ReparseTag == IO_REPARSE_TAG_SYMLINK:
@@ -236,11 +234,11 @@ def readlink(path):
         return rdb.MountPointReparseBuffer.PrintName
     # Unsupported reparse point type.
     _raise_winerror(
-        ERROR_NOT_SUPPORTED, 'Error reading symbolic link "{}"'.format(path)
+        ERROR_NOT_SUPPORTED, f'Error reading symbolic link "{path}"'
     )
 
 
 def _raise_winerror(code, error_desc):
     win_error_desc = FormatError(code).strip()
-    error_desc = "{0}: {1}".format(error_desc, win_error_desc)
+    error_desc = f"{error_desc}: {win_error_desc}"
     raise WinError(code, error_desc)

progress.py

@@ -52,11 +52,11 @@ def duration_str(total):
     uses microsecond resolution.  This makes for noisy output.
     """
     hours, mins, secs = convert_to_hms(total)
-    ret = "%.3fs" % (secs,)
+    ret = f"{secs:.3f}s"
     if mins:
-        ret = "%im%s" % (mins, ret)
+        ret = f"{mins}m{ret}"
     if hours:
-        ret = "%ih%s" % (hours, ret)
+        ret = f"{hours}h{ret}"
     return ret
 
 
@@ -82,7 +82,7 @@ def jobs_str(total):
     return f"{total} job{'s' if total > 1 else ''}"
 
 
-class Progress(object):
+class Progress:
     def __init__(
         self,
         title,

project.py

@@ -21,6 +21,7 @@ import random
 import re
 import shutil
 import stat
+import string
 import subprocess
 import sys
 import tarfile
@@ -152,7 +153,7 @@ def _ProjectHooks():
     return _project_hook_list
 
 
-class DownloadedChange(object):
+class DownloadedChange:
     _commit_cache = None
 
     def __init__(self, project, base, change_id, ps_id, commit):
@@ -178,7 +179,7 @@ class DownloadedChange(object):
         return self._commit_cache
 
 
-class ReviewableBranch(object):
+class ReviewableBranch:
     _commit_cache = None
     _base_exists = None
 
@@ -266,6 +267,7 @@ class ReviewableBranch(object):
         dest_branch=None,
         validate_certs=True,
         push_options=None,
+        patchset_description=None,
     ):
         self.project.UploadForReview(
             branch=self.name,
@@ -281,6 +283,7 @@ class ReviewableBranch(object):
             dest_branch=dest_branch,
             validate_certs=validate_certs,
             push_options=push_options,
+            patchset_description=patchset_description,
         )
 
     def GetPublishedRefs(self):
@@ -319,7 +322,7 @@ class DiffColoring(Coloring):
         self.fail = self.printer("fail", fg="red")
 
 
-class Annotation(object):
+class Annotation:
     def __init__(self, name, value, keep):
         self.name = name
         self.value = value
@@ -365,19 +368,19 @@ def _SafeExpandPath(base, subpath, skipfinal=False):
     for part in components:
         if part in {".", ".."}:
             raise ManifestInvalidPathError(
-                '%s: "%s" not allowed in paths' % (subpath, part)
+                f'{subpath}: "{part}" not allowed in paths'
             )
 
         path = os.path.join(path, part)
         if platform_utils.islink(path):
             raise ManifestInvalidPathError(
-                "%s: traversing symlinks not allow" % (path,)
+                f"{path}: traversing symlinks not allow"
             )
 
         if os.path.exists(path):
             if not os.path.isfile(path) and not platform_utils.isdir(path):
                 raise ManifestInvalidPathError(
-                    "%s: only regular files & directories allowed" % (path,)
+                    f"{path}: only regular files & directories allowed"
                 )
 
     if skipfinal:
@@ -386,7 +389,7 @@ def _SafeExpandPath(base, subpath, skipfinal=False):
     return path
 
 
-class _CopyFile(object):
+class _CopyFile:
     """Container for <copyfile> manifest element."""
 
     def __init__(self, git_worktree, src, topdir, dest):
@@ -409,11 +412,11 @@ class _CopyFile(object):
 
         if platform_utils.isdir(src):
             raise ManifestInvalidPathError(
-                "%s: copying from directory not supported" % (self.src,)
+                f"{self.src}: copying from directory not supported"
             )
         if platform_utils.isdir(dest):
             raise ManifestInvalidPathError(
-                "%s: copying to directory not allowed" % (self.dest,)
+                f"{self.dest}: copying to directory not allowed"
             )
 
         # Copy file if it does not exist or is out of date.
@@ -431,11 +434,11 @@ class _CopyFile(object):
                 mode = os.stat(dest)[stat.ST_MODE]
                 mode = mode & ~(stat.S_IWUSR | stat.S_IWGRP | stat.S_IWOTH)
                 os.chmod(dest, mode)
-            except IOError:
+            except OSError:
                 logger.error("error: Cannot copy file %s to %s", src, dest)
 
 
-class _LinkFile(object):
+class _LinkFile:
     """Container for <linkfile> manifest element."""
 
     def __init__(self, git_worktree, src, topdir, dest):
@@ -466,7 +469,7 @@ class _LinkFile(object):
                     if not platform_utils.isdir(dest_dir):
                         os.makedirs(dest_dir)
                 platform_utils.symlink(relSrc, absDest)
-            except IOError:
+            except OSError:
                 logger.error(
                     "error: Cannot link file %s to %s", relSrc, absDest
                 )
@@ -518,7 +521,7 @@ class _LinkFile(object):
                     self.__linkIt(relSrc, absDest)
 
 
-class RemoteSpec(object):
+class RemoteSpec:
     def __init__(
         self,
         name,
@@ -538,7 +541,7 @@ class RemoteSpec(object):
         self.fetchUrl = fetchUrl
 
 
-class Project(object):
+class Project:
     # These objects can be shared between several working trees.
     @property
     def shareable_dirs(self):
@@ -957,15 +960,11 @@ class Project(object):
                 f_status = "-"
 
             if i and i.src_path:
-                line = " %s%s\t%s => %s (%s%%)" % (
+                line = (
-                    i_status,
+                    f" {i_status}{f_status}\t{i.src_path} => {p} ({i.level}%)"
-                    f_status,
-                    i.src_path,
-                    p,
-                    i.level,
                 )
             else:
-                line = " %s%s\t%s" % (i_status, f_status, p)
+                line = f" {i_status}{f_status}\t{p}"
 
             if i and not f:
                 out.added("%s", line)
@@ -1093,6 +1092,7 @@ class Project(object):
         dest_branch=None,
         validate_certs=True,
         push_options=None,
+        patchset_description=None,
     ):
         """Uploads the named branch for code review."""
         if branch is None:
@@ -1135,7 +1135,7 @@ class Project(object):
         url = branch.remote.ReviewUrl(self.UserEmail, validate_certs)
         if url is None:
             raise UploadError("review not configured", project=self.name)
-        cmd = ["push"]
+        cmd = ["push", "--progress"]
         if dryrun:
             cmd.append("-n")
 
@@ -1157,7 +1157,7 @@ class Project(object):
         if dest_branch.startswith(R_HEADS):
             dest_branch = dest_branch[len(R_HEADS) :]
 
-        ref_spec = "%s:refs/for/%s" % (R_HEADS + branch.name, dest_branch)
+        ref_spec = f"{R_HEADS + branch.name}:refs/for/{dest_branch}"
         opts = []
         if auto_topic:
             opts += ["topic=" + branch.name]
@@ -1175,6 +1175,10 @@ class Project(object):
             opts += ["wip"]
         if ready:
             opts += ["ready"]
+        if patchset_description:
+            opts += [
+                f"m={self._encode_patchset_description(patchset_description)}"
+            ]
         if opts:
             ref_spec = ref_spec + "%" + ",".join(opts)
         cmd.append(ref_spec)
@@ -1182,11 +1186,35 @@ class Project(object):
         GitCommand(self, cmd, bare=True, verify_command=True).Wait()
 
         if not dryrun:
-            msg = "posted to %s for %s" % (branch.remote.review, dest_branch)
+            msg = f"posted to {branch.remote.review} for {dest_branch}"
             self.bare_git.UpdateRef(
                 R_PUB + branch.name, R_HEADS + branch.name, message=msg
             )
 
+    @staticmethod
+    def _encode_patchset_description(original):
+        """Applies percent-encoding for strings sent as patchset description.
+
+        The encoding used is based on but stricter than URL encoding (Section
+        2.1 of RFC 3986). The only non-escaped characters are alphanumerics, and
+        'SPACE' (U+0020) can be represented as 'LOW LINE' (U+005F) or
+        'PLUS SIGN' (U+002B).
+
+        For more information, see the Gerrit docs here:
+        https://gerrit-review.googlesource.com/Documentation/user-upload.html#patch_set_description
+        """
+        SAFE = {ord(x) for x in string.ascii_letters + string.digits}
+
+        def _enc(b):
+            if b in SAFE:
+                return chr(b)
+            elif b == ord(" "):
+                return "_"
+            else:
+                return f"%{b:02x}"
+
+        return "".join(_enc(x) for x in original.encode("utf-8"))
+
     def _ExtractArchive(self, tarpath, path=None):
         """Extract the given tar on its current location
 
@@ -1198,7 +1226,7 @@ class Project(object):
             with tarfile.open(tarpath, "r") as tar:
                 tar.extractall(path=path)
                 return True
-        except (IOError, tarfile.TarError) as e:
+        except (OSError, tarfile.TarError) as e:
             logger.error("error: Cannot extract archive %s: %s", tarpath, e)
         return False
 
@@ -1262,7 +1290,7 @@ class Project(object):
             try:
                 platform_utils.remove(tarpath)
             except OSError as e:
-                logger.warn("warn: Cannot remove archive %s: %s", tarpath, e)
+                logger.warning("warn: Cannot remove archive %s: %s", tarpath, e)
             self._CopyAndLinkFiles()
             return SyncNetworkHalfResult(True)
 
@@ -1281,7 +1309,20 @@ class Project(object):
         if is_new:
             self._InitGitDir(force_sync=force_sync, quiet=quiet)
         else:
-            self._UpdateHooks(quiet=quiet)
+            try:
+                # At this point, it's possible that gitdir points to an old
+                # objdir (e.g. name changed, but objdir exists). Check
+                # references to ensure that's not the case. See
+                # https://issues.gerritcodereview.com/40013418 for more
+                # details.
+                self._CheckDirReference(self.objdir, self.gitdir)
+
+                self._UpdateHooks(quiet=quiet)
+            except GitError as e:
+                if not force_sync:
+                    raise e
+                # Let _InitGitDir fix the issue, force_sync is always True here.
+                self._InitGitDir(force_sync=True, quiet=quiet)
         self._InitRemote()
 
         if self.UseAlternates:
@@ -1309,7 +1350,7 @@ class Project(object):
                     alt_dir = os.path.join(
                         self.objdir, "objects", fd.readline().rstrip()
                     )
-            except IOError:
+            except OSError:
                 alt_dir = None
         else:
             alt_dir = None
@@ -1444,7 +1485,7 @@ class Project(object):
             return self.bare_git.rev_list(self.revisionExpr, "-1")[0]
         except GitError:
             raise ManifestInvalidRevisionError(
-                "revision %s in %s not found" % (self.revisionExpr, self.name)
+                f"revision {self.revisionExpr} in {self.name} not found"
             )
 
     def GetRevisionId(self, all_refs=None):
@@ -1461,7 +1502,7 @@ class Project(object):
             return self.bare_git.rev_parse("--verify", "%s^0" % rev)
         except GitError:
             raise ManifestInvalidRevisionError(
-                "revision %s in %s not found" % (self.revisionExpr, self.name)
+                f"revision {self.revisionExpr} in {self.name} not found"
             )
 
     def SetRevisionId(self, revisionId):
@@ -1471,7 +1512,13 @@ class Project(object):
         self.revisionId = revisionId
 
     def Sync_LocalHalf(
-        self, syncbuf, force_sync=False, submodules=False, errors=None
+        self,
+        syncbuf,
+        force_sync=False,
+        force_checkout=False,
+        submodules=False,
+        errors=None,
+        verbose=False,
     ):
         """Perform only the local IO portion of the sync process.
 
@@ -1552,11 +1599,11 @@ class Project(object):
                     return
             else:
                 lost = self._revlist(not_rev(revid), HEAD)
-                if lost:
+                if lost and verbose:
                     syncbuf.info(self, "discarding %d commits", len(lost))
 
             try:
-                self._Checkout(revid, quiet=True)
+                self._Checkout(revid, force_checkout=force_checkout, quiet=True)
                 if submodules:
                     self._SyncSubmodules(quiet=True)
             except GitError as e:
@@ -1622,9 +1669,9 @@ class Project(object):
             elif pub == head:
                 # All published commits are merged, and thus we are a
                 # strict subset.  We can fast-forward safely.
-                syncbuf.later1(self, _doff)
+                syncbuf.later1(self, _doff, not verbose)
                 if submodules:
-                    syncbuf.later1(self, _dosubmodules)
+                    syncbuf.later1(self, _dosubmodules, not verbose)
                 return
 
         # Examine the local commits not in the remote.  Find the
@@ -1683,10 +1730,10 @@ class Project(object):
             def _dorebase():
                 self._Rebase(upstream="%s^1" % last_mine, onto=revid)
 
-            syncbuf.later2(self, _dorebase)
+            syncbuf.later2(self, _dorebase, not verbose)
             if submodules:
-                syncbuf.later2(self, _dosubmodules)
+                syncbuf.later2(self, _dosubmodules, not verbose)
-            syncbuf.later2(self, _docopyandlink)
+            syncbuf.later2(self, _docopyandlink, not verbose)
         elif local_changes:
             try:
                 self._ResetHard(revid)
@@ -1697,9 +1744,9 @@ class Project(object):
                 fail(e)
                 return
         else:
-            syncbuf.later1(self, _doff)
+            syncbuf.later1(self, _doff, not verbose)
             if submodules:
-                syncbuf.later1(self, _dosubmodules)
+                syncbuf.later1(self, _dosubmodules, not verbose)
 
     def AddCopyFile(self, src, dest, topdir):
         """Mark |src| for copying to |dest| (relative to |topdir|).
@@ -1742,7 +1789,7 @@ class Project(object):
             self.bare_git.rev_parse("FETCH_HEAD"),
         )
 
-    def DeleteWorktree(self, quiet=False, force=False):
+    def DeleteWorktree(self, verbose=False, force=False):
         """Delete the source checkout and any other housekeeping tasks.
 
         This currently leaves behind the internal .repo/ cache state.  This
@@ -1751,7 +1798,7 @@ class Project(object):
         at some point.
 
         Args:
-            quiet: Whether to hide normal messages.
+            verbose: Whether to show verbose messages.
             force: Always delete tree even if dirty.
 
         Returns:
@@ -1759,23 +1806,21 @@ class Project(object):
         """
         if self.IsDirty():
             if force:
-                logger.warn(
+                logger.warning(
                     "warning: %s: Removing dirty project: uncommitted changes "
                     "lost.",
                     self.RelPath(local=False),
                 )
             else:
                 msg = (
-                    "error: %s: Cannot remove project: uncommitted"
+                    "error: %s: Cannot remove project: uncommitted "
                     "changes are present.\n" % self.RelPath(local=False)
                 )
                 logger.error(msg)
-                raise DeleteDirtyWorktreeError(msg, project=self)
+                raise DeleteDirtyWorktreeError(msg, project=self.name)
 
-        if not quiet:
+        if verbose:
-            print(
+            print(f"{self.RelPath(local=False)}: Deleting obsolete checkout.")
-                "%s: Deleting obsolete checkout." % (self.RelPath(local=False),)
-            )
 
         # Unlock and delink from the main worktree.  We don't use git's worktree
         # remove because it will recursively delete projects -- we handle that
@@ -1836,7 +1881,7 @@ class Project(object):
                     platform_utils.remove(path)
                 except OSError as e:
                     if e.errno != errno.ENOENT:
-                        logger.error("error: %s: Failed to remove: %s", path, e)
+                        logger.warning("%s: Failed to remove: %s", path, e)
                         failed = True
                         errors.append(e)
             dirs[:] = [
@@ -1855,7 +1900,7 @@ class Project(object):
                     platform_utils.remove(d)
                 except OSError as e:
                     if e.errno != errno.ENOENT:
-                        logger.error("error: %s: Failed to remove: %s", d, e)
+                        logger.warning("%s: Failed to remove: %s", d, e)
                         failed = True
                         errors.append(e)
             elif not platform_utils.listdir(d):
@@ -1863,18 +1908,30 @@ class Project(object):
                     platform_utils.rmdir(d)
                 except OSError as e:
                     if e.errno != errno.ENOENT:
-                        logger.error("error: %s: Failed to remove: %s", d, e)
+                        logger.warning("%s: Failed to remove: %s", d, e)
                         failed = True
                         errors.append(e)
         if failed:
-            logger.error(
+            rename_path = (
-                "error: %s: Failed to delete obsolete checkout.",
+                f"{self.worktree}_repo_to_be_deleted_{int(time.time())}"
-                self.RelPath(local=False),
             )
-            logger.error(
+            try:
-                "       Remove manually, then run `repo sync -l`.",
+                platform_utils.rename(self.worktree, rename_path)
-            )
+                logger.warning(
-            raise DeleteWorktreeError(aggregate_errors=errors)
+                    "warning: renamed %s to %s. You can delete it, but you "
+                    "might need elevated permissions (e.g. root)",
+                    self.worktree,
+                    rename_path,
+                )
+                # Rename successful! Clear the errors.
+                errors = []
+            except OSError:
+                logger.error(
+                    "%s: Failed to delete obsolete checkout.\n",
+                    "       Remove manually, then run `repo sync -l`.",
+                    self.RelPath(local=False),
+                )
+                raise DeleteWorktreeError(aggregate_errors=errors)
 
         # Try deleting parent dirs if they are empty.
         path = self.worktree
@@ -1968,7 +2025,7 @@ class Project(object):
             # target branch, but otherwise take no other action.
             _lwrite(
                 self.work_git.GetDotgitPath(subpath=HEAD),
-                "ref: %s%s\n" % (R_HEADS, name),
+                f"ref: {R_HEADS}{name}\n",
             )
             return True
 
@@ -2277,7 +2334,7 @@ class Project(object):
             self.config.SetString("core.repositoryFormatVersion", str(version))
 
         # Enable the extension!
-        self.config.SetString("extensions.%s" % (key,), value)
+        self.config.SetString(f"extensions.{key}", value)
 
     def ResolveRemoteHead(self, name=None):
         """Find out what the default branch (HEAD) points to.
@@ -2447,7 +2504,7 @@ class Project(object):
                 old_packed_lines = []
 
                 for r in sorted(all_refs):
-                    line = "%s %s\n" % (all_refs[r], r)
+                    line = f"{all_refs[r]} {r}\n"
                     tmp_packed_lines.append(line)
                     if r not in tmp:
                         old_packed_lines.append(line)
@@ -2617,7 +2674,7 @@ class Project(object):
             # one.
             if not verbose and gitcmd.stdout:
                 print(
-                    "\n%s:\n%s" % (self.name, gitcmd.stdout),
+                    f"\n{self.name}:\n{gitcmd.stdout}",
                     end="",
                     file=output_redir,
                 )
@@ -2752,7 +2809,7 @@ class Project(object):
             proc = None
             with Trace("Fetching bundle: %s", " ".join(cmd)):
                 if verbose:
-                    print("%s: Downloading bundle: %s" % (self.name, srcUrl))
+                    print(f"{self.name}: Downloading bundle: {srcUrl}")
                 stdout = None if verbose else subprocess.PIPE
                 stderr = None if verbose else subprocess.STDOUT
                 try:
@@ -2801,16 +2858,18 @@ class Project(object):
         except OSError:
             return False
 
-    def _Checkout(self, rev, quiet=False):
+    def _Checkout(self, rev, force_checkout=False, quiet=False):
         cmd = ["checkout"]
         if quiet:
             cmd.append("-q")
+        if force_checkout:
+            cmd.append("-f")
         cmd.append(rev)
         cmd.append("--")
         if GitCommand(self, cmd).Wait() != 0:
             if self._allrefs:
                 raise GitError(
-                    "%s checkout %s " % (self.name, rev), project=self.name
+                    f"{self.name} checkout {rev} ", project=self.name
                 )
 
     def _CherryPick(self, rev, ffonly=False, record_origin=False):
@@ -2824,7 +2883,7 @@ class Project(object):
         if GitCommand(self, cmd).Wait() != 0:
             if self._allrefs:
                 raise GitError(
-                    "%s cherry-pick %s " % (self.name, rev), project=self.name
+                    f"{self.name} cherry-pick {rev} ", project=self.name
                 )
 
     def _LsRemote(self, refs):
@@ -2841,9 +2900,7 @@ class Project(object):
         cmd.append("--")
         if GitCommand(self, cmd).Wait() != 0:
             if self._allrefs:
-                raise GitError(
+                raise GitError(f"{self.name} revert {rev} ", project=self.name)
-                    "%s revert %s " % (self.name, rev), project=self.name
-                )
 
     def _ResetHard(self, rev, quiet=True):
         cmd = ["reset", "--hard"]
@@ -2852,7 +2909,7 @@ class Project(object):
         cmd.append(rev)
         if GitCommand(self, cmd).Wait() != 0:
             raise GitError(
-                "%s reset --hard %s " % (self.name, rev), project=self.name
+                f"{self.name} reset --hard {rev} ", project=self.name
             )
 
     def _SyncSubmodules(self, quiet=True):
@@ -2871,18 +2928,16 @@ class Project(object):
             cmd.extend(["--onto", onto])
         cmd.append(upstream)
         if GitCommand(self, cmd).Wait() != 0:
-            raise GitError(
+            raise GitError(f"{self.name} rebase {upstream} ", project=self.name)
-                "%s rebase %s " % (self.name, upstream), project=self.name
-            )
 
-    def _FastForward(self, head, ffonly=False):
+    def _FastForward(self, head, ffonly=False, quiet=True):
         cmd = ["merge", "--no-stat", head]
         if ffonly:
             cmd.append("--ff-only")
+        if quiet:
+            cmd.append("-q")
         if GitCommand(self, cmd).Wait() != 0:
-            raise GitError(
+            raise GitError(f"{self.name} merge {head} ", project=self.name)
-                "%s merge %s " % (self.name, head), project=self.name
-            )
 
     def _InitGitDir(self, mirror_git=None, force_sync=False, quiet=False):
         init_git_dir = not os.path.exists(self.gitdir)
@@ -2990,6 +3045,17 @@ class Project(object):
                 self.config.SetBoolean(
                     "core.bare", True if self.manifest.IsMirror else None
                 )
+
+            if not init_obj_dir:
+                # The project might be shared (obj_dir already initialized), but
+                # such information is not available here. Instead of passing it,
+                # set it as shared, and rely to be unset down the execution
+                # path.
+                if git_require((2, 7, 0)):
+                    self.EnableRepositoryExtension("preciousObjects")
+                else:
+                    self.config.SetString("gc.pruneExpire", "never")
+
         except Exception:
             if init_obj_dir and os.path.exists(self.objdir):
                 platform_utils.rmtree(self.objdir)
@@ -3038,7 +3104,7 @@ class Project(object):
                 # hardlink below.
                 if not filecmp.cmp(stock_hook, dst, shallow=False):
                     if not quiet:
-                        logger.warn(
+                        logger.warning(
                             "warn: %s: Not replacing locally modified %s hook",
                             self.RelPath(local=False),
                             name,
@@ -3160,8 +3226,9 @@ class Project(object):
                         "--force-sync not enabled; cannot overwrite a local "
                         "work tree. If you're comfortable with the "
                         "possibility of losing the work tree's git metadata,"
-                        " use `repo sync --force-sync {0}` to "
+                        " use "
-                        "proceed.".format(self.RelPath(local=False)),
+                        f"`repo sync --force-sync {self.RelPath(local=False)}` "
+                        "to proceed.",
                         project=self.name,
                     )
 
@@ -3227,7 +3294,7 @@ class Project(object):
         # Rewrite the internal state files to use relative paths between the
         # checkouts & worktrees.
         dotgit = os.path.join(self.worktree, ".git")
-        with open(dotgit, "r") as fp:
+        with open(dotgit) as fp:
             # Figure out the checkout->worktree path.
             setting = fp.read()
             assert setting.startswith("gitdir:")
@@ -3274,7 +3341,7 @@ class Project(object):
         if not platform_utils.islink(dotgit) and platform_utils.isdir(dotgit):
             self._MigrateOldWorkTreeGitDir(dotgit, project=self.name)
 
-        init_dotgit = not os.path.exists(dotgit)
+        init_dotgit = not os.path.lexists(dotgit)
         if self.use_git_worktrees:
             if init_dotgit:
                 self._InitGitWorktree()
@@ -3394,7 +3461,8 @@ class Project(object):
         # Now that the dir should be empty, clear it out, and symlink it over.
         platform_utils.rmdir(dotgit)
         platform_utils.symlink(
-            os.path.relpath(gitdir, os.path.dirname(dotgit)), dotgit
+            os.path.relpath(gitdir, os.path.dirname(os.path.realpath(dotgit))),
+            dotgit,
         )
 
     def _get_symlink_error_message(self):
@@ -3474,7 +3542,7 @@ class Project(object):
         )
         return logs
 
-    class _GitGetByExec(object):
+    class _GitGetByExec:
         def __init__(self, project, bare, gitdir):
             self._project = project
             self._bare = bare
@@ -3529,7 +3597,7 @@ class Project(object):
                     except StopIteration:
                         break
 
-                    class _Info(object):
+                    class _Info:
                         def __init__(self, path, omode, nmode, oid, nid, state):
                             self.path = path
                             self.src_path = None
@@ -3583,7 +3651,7 @@ class Project(object):
             try:
                 with open(path) as fd:
                     line = fd.readline()
-            except IOError as e:
+            except OSError as e:
                 raise NoManifestException(path, str(e))
             try:
                 line = line.decode()
@@ -3674,12 +3742,12 @@ class Project(object):
                 config = kwargs.pop("config", None)
                 for k in kwargs:
                     raise TypeError(
-                        "%s() got an unexpected keyword argument %r" % (name, k)
+                        f"{name}() got an unexpected keyword argument {k!r}"
                     )
                 if config is not None:
                     for k, v in config.items():
                         cmdv.append("-c")
-                        cmdv.append("%s=%s" % (k, v))
+                        cmdv.append(f"{k}={v}")
                 cmdv.append(name)
                 cmdv.extend(args)
                 p = GitCommand(
@@ -3715,7 +3783,7 @@ class _DirtyError(LocalSyncFail):
         return "contains uncommitted changes"
 
 
-class _InfoMessage(object):
+class _InfoMessage:
     def __init__(self, project, text):
         self.project = project
         self.text = text
@@ -3727,7 +3795,7 @@ class _InfoMessage(object):
         syncbuf.out.nl()
 
 
-class _Failure(object):
+class _Failure:
     def __init__(self, project, why):
         self.project = project
         self.why = why
@@ -3739,18 +3807,21 @@ class _Failure(object):
         syncbuf.out.nl()
 
 
-class _Later(object):
+class _Later:
-    def __init__(self, project, action):
+    def __init__(self, project, action, quiet):
         self.project = project
         self.action = action
+        self.quiet = quiet
 
     def Run(self, syncbuf):
         out = syncbuf.out
-        out.project("project %s/", self.project.RelPath(local=False))
+        if not self.quiet:
-        out.nl()
+            out.project("project %s/", self.project.RelPath(local=False))
+            out.nl()
         try:
             self.action()
-            out.nl()
+            if not self.quiet:
+                out.nl()
             return True
         except GitError:
             out.nl()
@@ -3765,7 +3836,7 @@ class _SyncColoring(Coloring):
         self.fail = self.printer("fail", fg="red")
 
 
-class SyncBuffer(object):
+class SyncBuffer:
     def __init__(self, config, detach_head=False):
         self._messages = []
         self._failures = []
@@ -3786,11 +3857,11 @@ class SyncBuffer(object):
         self._failures.append(_Failure(project, err))
         self._MarkUnclean()
 
-    def later1(self, project, what):
+    def later1(self, project, what, quiet):
-        self._later_queue1.append(_Later(project, what))
+        self._later_queue1.append(_Later(project, what, quiet))
 
-    def later2(self, project, what):
+    def later2(self, project, what, quiet):
-        self._later_queue2.append(_Later(project, what))
+        self._later_queue2.append(_Later(project, what, quiet))
 
     def Finish(self):
         self._PrintMessages()
@@ -3899,13 +3970,13 @@ class RepoProject(MetaProject):
 class ManifestProject(MetaProject):
     """The MetaProject for manifests."""
 
-    def MetaBranchSwitch(self, submodules=False):
+    def MetaBranchSwitch(self, submodules=False, verbose=False):
         """Prepare for manifest branch switch."""
 
         # detach and delete manifest branch, allowing a new
         # branch to take over
         syncbuf = SyncBuffer(self.config, detach_head=True)
-        self.Sync_LocalHalf(syncbuf, submodules=submodules)
+        self.Sync_LocalHalf(syncbuf, submodules=submodules, verbose=verbose)
         syncbuf.Finish()
 
         return (
@@ -3981,7 +4052,7 @@ class ManifestProject(MetaProject):
     @property
     def depth(self):
         """Partial clone depth."""
-        return self.config.GetString("repo.depth")
+        return self.config.GetInt("repo.depth")
 
     @property
     def clone_filter(self):
@@ -4335,7 +4406,7 @@ class ManifestProject(MetaProject):
             self.config.SetBoolean("repo.worktree", worktree)
             if is_new:
                 self.use_git_worktrees = True
-            logger.warn("warning: --worktree is experimental!")
+            logger.warning("warning: --worktree is experimental!")
 
         if archive:
             if is_new:
@@ -4399,7 +4470,7 @@ class ManifestProject(MetaProject):
 
             self.config.SetBoolean("repo.git-lfs", git_lfs)
             if not is_new:
-                logger.warn(
+                logger.warning(
                     "warning: Changing --git-lfs settings will only affect new "
                     "project checkouts.\n"
                     "         Existing projects will require manual updates.\n"
@@ -4436,10 +4507,10 @@ class ManifestProject(MetaProject):
                 return False
 
             if manifest_branch:
-                self.MetaBranchSwitch(submodules=submodules)
+                self.MetaBranchSwitch(submodules=submodules, verbose=verbose)
 
             syncbuf = SyncBuffer(self.config)
-            self.Sync_LocalHalf(syncbuf, submodules=submodules)
+            self.Sync_LocalHalf(syncbuf, submodules=submodules, verbose=verbose)
             syncbuf.Finish()
 
             if is_new or self.CurrentBranch is None:
@@ -4511,7 +4582,7 @@ class ManifestProject(MetaProject):
                 submanifest = ""
                 if self.manifest.path_prefix:
                     submanifest = f"for {self.manifest.path_prefix} "
-                logger.warn(
+                logger.warning(
                     "warning: git update of superproject %s failed, "
                     "repo sync will not use superproject to fetch source; "
                     "while this error is not fatal, and you can continue to "

pyproject.toml

@@ -15,4 +15,4 @@
 [tool.black]
 line-length = 80
 # NB: Keep in sync with tox.ini.
-target-version = ['py36', 'py37', 'py38', 'py39', 'py310', 'py311']
+target-version = ['py36', 'py37', 'py38', 'py39', 'py310', 'py311'] #, 'py312'

repo

@@ -1,5 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
-# -*- coding:utf-8 -*-
 #
 # Copyright (C) 2008 The Android Open Source Project
 #
@@ -22,8 +21,6 @@ It is used to get an initial repo client checkout, and after that it runs the
 copy of repo in the checkout.
 """
 
-from __future__ import print_function
-
 import datetime
 import os
 import platform
@@ -36,11 +33,11 @@ import sys
 # bit more flexible with older systems.  See that file for more details on the
 # versions we select.
 MIN_PYTHON_VERSION_SOFT = (3, 6)
-MIN_PYTHON_VERSION_HARD = (3, 5)
+MIN_PYTHON_VERSION_HARD = (3, 6)
 
 
 # Keep basic logic in sync with repo_trace.py.
-class Trace(object):
+class Trace:
     """Trace helper logic."""
 
     REPO_TRACE = "REPO_TRACE"
@@ -82,24 +79,13 @@ def check_python_version():
     major = ver.major
     minor = ver.minor
 
-    # Abort on very old Python 2 versions.
+    # Try to re-exec the version specific Python if needed.
-    if (major, minor) < (2, 7):
-        print(
-            "repo: error: Your Python version is too old. "
-            "Please use Python {}.{} or newer instead.".format(
-                *MIN_PYTHON_VERSION_SOFT
-            ),
-            file=sys.stderr,
-        )
-        sys.exit(1)
-
-    # Try to re-exec the version specific Python 3 if needed.
     if (major, minor) < MIN_PYTHON_VERSION_SOFT:
         # Python makes releases ~once a year, so try our min version +10 to help
         # bridge the gap.  This is the fallback anyways so perf isn't critical.
         min_major, min_minor = MIN_PYTHON_VERSION_SOFT
         for inc in range(0, 10):
-            reexec("python{}.{}".format(min_major, min_minor + inc))
+            reexec(f"python{min_major}.{min_minor + inc}")
 
         # Fallback to older versions if possible.
         for inc in range(
@@ -108,47 +94,12 @@ def check_python_version():
             # Don't downgrade, and don't reexec ourselves (which would infinite loop).
             if (min_major, min_minor - inc) <= (major, minor):
                 break
-            reexec("python{}.{}".format(min_major, min_minor - inc))
+            reexec(f"python{min_major}.{min_minor - inc}")
-
-        # Try the generic Python 3 wrapper, but only if it's new enough.  If it
-        # isn't, we want to just give up below and make the user resolve things.
-        try:
-            proc = subprocess.Popen(
-                [
-                    "python3",
-                    "-c",
-                    "import sys; "
-                    "print(sys.version_info.major, sys.version_info.minor)",
-                ],
-                stdout=subprocess.PIPE,
-                stderr=subprocess.PIPE,
-            )
-            (output, _) = proc.communicate()
-            python3_ver = tuple(int(x) for x in output.decode("utf-8").split())
-        except (OSError, subprocess.CalledProcessError):
-            python3_ver = None
-
-        # If the python3 version looks like it's new enough, give it a try.
-        if (
-            python3_ver
-            and python3_ver >= MIN_PYTHON_VERSION_HARD
-            and python3_ver != (major, minor)
-        ):
-            reexec("python3")
 
         # We're still here, so diagnose things for the user.
-        if major < 3:
+        if (major, minor) < MIN_PYTHON_VERSION_HARD:
             print(
-                "repo: error: Python 2 is no longer supported; "
+                "repo: error: Python version is too old; "
-                "Please upgrade to Python {}.{}+.".format(
-                    *MIN_PYTHON_VERSION_HARD
-                ),
-                file=sys.stderr,
-            )
-            sys.exit(1)
-        elif (major, minor) < MIN_PYTHON_VERSION_HARD:
-            print(
-                "repo: error: Python 3 version is too old; "
                 "Please use Python {}.{} or newer.".format(
                     *MIN_PYTHON_VERSION_HARD
                 ),
@@ -173,7 +124,7 @@ if not REPO_REV:
 BUG_URL = "https://issues.gerritcodereview.com/issues/new?component=1370071"
 
 # increment this whenever we make important changes to this script
-VERSION = (2, 37)
+VERSION = (2, 45)
 
 # increment this if the MAINTAINER_KEYS block is modified
 KEYRING_VERSION = (2, 3)
@@ -259,9 +210,8 @@ GIT = "git"  # our git command
 # NB: The version of git that the repo launcher requires may be much older than
 # the version of git that the main repo source tree requires.  Keeping this at
 # an older version also makes it easier for users to upgrade/rollback as needed.
-#
+# See requirements.json for versions.
-# git-1.7 is in (EOL) Ubuntu Precise.
+MIN_GIT_VERSION = (1, 7, 9)  # minimum supported git version
-MIN_GIT_VERSION = (1, 7, 2)  # minimum supported git version
 repodir = ".repo"  # name of repo's private directory
 S_repo = "repo"  # special repo repository
 S_manifests = "manifests"  # special manifest repository
@@ -277,19 +227,8 @@ import optparse
 import re
 import shutil
 import stat
-
+import urllib.error
-
+import urllib.request
-if sys.version_info[0] == 3:
-    import urllib.error
-    import urllib.request
-else:
-    import imp
-
-    import urllib2
-
-    urllib = imp.new_module("urllib")
-    urllib.request = urllib2
-    urllib.error = urllib2
 
 
 repo_config_dir = os.getenv("REPO_CONFIG_DIR", os.path.expanduser("~"))
@@ -569,8 +508,7 @@ def run_command(cmd, **kwargs):
             return output.decode("utf-8")
         except UnicodeError:
             print(
-                "repo: warning: Invalid UTF-8 output:\ncmd: %r\n%r"
+                f"repo: warning: Invalid UTF-8 output:\ncmd: {cmd!r}\n{output}",
-                % (cmd, output),
                 file=sys.stderr,
             )
             return output.decode("utf-8", "backslashreplace")
@@ -593,20 +531,17 @@ def run_command(cmd, **kwargs):
     # If things failed, print useful debugging output.
     if check and ret.returncode:
         print(
-            'repo: error: "%s" failed with exit status %s'
+            f'repo: error: "{cmd[0]}" failed with exit status {ret.returncode}',
-            % (cmd[0], ret.returncode),
-            file=sys.stderr,
-        )
-        print(
-            "  cwd: %s\n  cmd: %r" % (kwargs.get("cwd", os.getcwd()), cmd),
             file=sys.stderr,
         )
+        cwd = kwargs.get("cwd", os.getcwd())
+        print(f"  cwd: {cwd}\n  cmd: {cmd!r}", file=sys.stderr)
 
         def _print_output(name, output):
             if output:
                 print(
-                    "  %s:\n  >> %s"
+                    f"  {name}:"
-                    % (name, "\n  >> ".join(output.splitlines())),
+                    + "".join(f"\n  >> {x}" for x in output.splitlines()),
                     file=sys.stderr,
                 )
 
@@ -625,12 +560,12 @@ def get_gitc_manifest_dir():
     if _gitc_manifest_dir is None:
         _gitc_manifest_dir = ""
         try:
-            with open(GITC_CONFIG_FILE, "r") as gitc_config:
+            with open(GITC_CONFIG_FILE) as gitc_config:
                 for line in gitc_config:
                     match = re.match("gitc_dir=(?P<gitc_manifest_dir>.*)", line)
                     if match:
                         _gitc_manifest_dir = match.group("gitc_manifest_dir")
-        except IOError:
+        except OSError:
             pass
     return _gitc_manifest_dir
 
@@ -722,7 +657,7 @@ def _Init(args, gitc_init=False):
     except OSError as e:
         if e.errno != errno.EEXIST:
             print(
-                "fatal: cannot make %s directory: %s" % (repodir, e.strerror),
+                f"fatal: cannot make {repodir} directory: {e.strerror}",
                 file=sys.stderr,
             )
             # Don't raise CloneFailure; that would delete the
@@ -820,7 +755,7 @@ def _CheckGitVersion():
     if ver_act < MIN_GIT_VERSION:
         need = ".".join(map(str, MIN_GIT_VERSION))
         print(
-            "fatal: git %s or later required; found %s" % (need, ver_act.full),
+            f"fatal: git {need} or later required; found {ver_act.full}",
             file=sys.stderr,
         )
         raise CloneFailure()
@@ -839,7 +774,8 @@ def SetGitTrace2ParentSid(env=None):
     KEY = "GIT_TRACE2_PARENT_SID"
 
     now = datetime.datetime.now(datetime.timezone.utc)
-    value = "repo-%s-P%08x" % (now.strftime("%Y%m%dT%H%M%SZ"), os.getpid())
+    timestamp = now.strftime("%Y%m%dT%H%M%SZ")
+    value = f"repo-{timestamp}-P{os.getpid():08x}"
 
     # If it's already set, then append ourselves.
     if KEY in env:
@@ -883,8 +819,7 @@ def SetupGnuPG(quiet):
     except OSError as e:
         if e.errno != errno.EEXIST:
             print(
-                "fatal: cannot make %s directory: %s"
+                f"fatal: cannot make {home_dot_repo} directory: {e.strerror}",
-                % (home_dot_repo, e.strerror),
                 file=sys.stderr,
             )
             sys.exit(1)
@@ -894,15 +829,15 @@ def SetupGnuPG(quiet):
     except OSError as e:
         if e.errno != errno.EEXIST:
             print(
-                "fatal: cannot make %s directory: %s" % (gpg_dir, e.strerror),
+                f"fatal: cannot make {gpg_dir} directory: {e.strerror}",
                 file=sys.stderr,
             )
             sys.exit(1)
 
     if not quiet:
         print(
-            "repo: Updating release signing keys to keyset ver %s"
+            "repo: Updating release signing keys to keyset ver "
-            % (".".join(str(x) for x in KEYRING_VERSION),)
+            + ".".join(str(x) for x in KEYRING_VERSION),
         )
     # NB: We use --homedir (and cwd below) because some environments (Windows) do
     # not correctly handle full native paths.  We avoid the issue by changing to
@@ -954,7 +889,7 @@ def _GetRepoConfig(name):
         return None
     else:
         print(
-            "repo: error: git %s failed:\n%s" % (" ".join(cmd), ret.stderr),
+            f"repo: error: git {' '.join(cmd)} failed:\n{ret.stderr}",
             file=sys.stderr,
         )
         raise RunError()
@@ -1067,7 +1002,7 @@ def _Clone(url, cwd, clone_bundle, quiet, verbose):
         os.mkdir(cwd)
     except OSError as e:
         print(
-            "fatal: cannot make %s directory: %s" % (cwd, e.strerror),
+            f"fatal: cannot make {cwd} directory: {e.strerror}",
             file=sys.stderr,
         )
         raise CloneFailure()
@@ -1107,7 +1042,7 @@ def resolve_repo_rev(cwd, committish):
         ret = run_git(
             "rev-parse",
             "--verify",
-            "%s^{commit}" % (committish,),
+            f"{committish}^{{commit}}",
             cwd=cwd,
             check=False,
         )
@@ -1120,7 +1055,7 @@ def resolve_repo_rev(cwd, committish):
         rev = resolve("refs/remotes/origin/%s" % committish)
         if rev is None:
             print(
-                'repo: error: unknown branch "%s"' % (committish,),
+                f'repo: error: unknown branch "{committish}"',
                 file=sys.stderr,
             )
             raise CloneFailure()
@@ -1133,7 +1068,8 @@ def resolve_repo_rev(cwd, committish):
         rev = resolve(remote_ref)
         if rev is None:
             print(
-                'repo: error: unknown tag "%s"' % (committish,), file=sys.stderr
+                f'repo: error: unknown tag "{committish}"',
+                file=sys.stderr,
             )
             raise CloneFailure()
         return (remote_ref, rev)
@@ -1141,12 +1077,12 @@ def resolve_repo_rev(cwd, committish):
     # See if it's a short branch name.
     rev = resolve("refs/remotes/origin/%s" % committish)
     if rev:
-        return ("refs/heads/%s" % (committish,), rev)
+        return (f"refs/heads/{committish}", rev)
 
     # See if it's a tag.
-    rev = resolve("refs/tags/%s" % committish)
+    rev = resolve(f"refs/tags/{committish}")
     if rev:
-        return ("refs/tags/%s" % (committish,), rev)
+        return (f"refs/tags/{committish}", rev)
 
     # See if it's a commit.
     rev = resolve(committish)
@@ -1155,7 +1091,8 @@ def resolve_repo_rev(cwd, committish):
 
     # Give up!
     print(
-        'repo: error: unable to resolve "%s"' % (committish,), file=sys.stderr
+        f'repo: error: unable to resolve "{committish}"',
+        file=sys.stderr,
     )
     raise CloneFailure()
 
@@ -1171,8 +1108,8 @@ def verify_rev(cwd, remote_ref, rev, quiet):
         if not quiet:
             print(file=sys.stderr)
             print(
-                "warning: '%s' is not signed; falling back to signed release '%s'"
+                f"warning: '{remote_ref}' is not signed; "
-                % (remote_ref, cur),
+                f"falling back to signed release '{cur}'",
                 file=sys.stderr,
             )
             print(file=sys.stderr)
@@ -1214,7 +1151,7 @@ def _FindRepo():
     return (repo, os.path.join(curdir, repodir))
 
 
-class _Options(object):
+class _Options:
     help = False
     version = False
 
@@ -1225,7 +1162,7 @@ def _ExpandAlias(name):
     if name in {"gitc-init", "help", "init"}:
         return name, []
 
-    alias = _GetRepoConfig("alias.%s" % (name,))
+    alias = _GetRepoConfig(f"alias.{name}")
     if alias is None:
         return name, []
 
@@ -1258,7 +1195,7 @@ def _ParseArguments(args):
     return cmd, opt, arg
 
 
-class Requirements(object):
+class Requirements:
     """Helper for checking repo's system requirements."""
 
     REQUIREMENTS_NAME = "requirements.json"
@@ -1280,8 +1217,7 @@ class Requirements(object):
         try:
             with open(path, "rb") as f:
                 data = f.read()
-        except EnvironmentError:
+        except OSError:
-            # NB: EnvironmentError is used for Python 2 & 3 compatibility.
             # If we couldn't open the file, assume it's an old source tree.
             return None
 
@@ -1321,18 +1257,20 @@ class Requirements(object):
         hard_ver = tuple(self._get_hard_ver(pkg))
         if curr_ver < hard_ver:
             print(
-                'repo: error: Your version of "%s" (%s) is unsupported; '
+                f'repo: error: Your version of "{pkg}" '
-                "Please upgrade to at least version %s to continue."
+                f"({self._format_ver(curr_ver)}) is unsupported; "
-                % (pkg, self._format_ver(curr_ver), self._format_ver(soft_ver)),
+                "Please upgrade to at least version "
+                f"{self._format_ver(soft_ver)} to continue.",
                 file=sys.stderr,
             )
             sys.exit(1)
 
         if curr_ver < soft_ver:
             print(
-                'repo: warning: Your version of "%s" (%s) is no longer supported; '
+                f'repo: error: Your version of "{pkg}" '
-                "Please upgrade to at least version %s to avoid breakage."
+                f"({self._format_ver(curr_ver)}) is no longer supported; "
-                % (pkg, self._format_ver(curr_ver), self._format_ver(soft_ver)),
+                "Please upgrade to at least version "
+                f"{self._format_ver(soft_ver)} to continue.",
                 file=sys.stderr,
             )
 
@@ -1393,20 +1331,14 @@ def _Help(args):
 def _Version():
     """Show version information."""
     print("<repo not installed>")
-    print("repo launcher version %s" % (".".join(str(x) for x in VERSION),))
+    print(f"repo launcher version {'.'.join(str(x) for x in VERSION)}")
-    print("       (from %s)" % (__file__,))
+    print(f"       (from {__file__})")
-    print("git %s" % (ParseGitVersion().full,))
+    print(f"git {ParseGitVersion().full}")
-    print("Python %s" % sys.version)
+    print(f"Python {sys.version}")
     uname = platform.uname()
-    if sys.version_info.major < 3:
+    print(f"OS {uname.system} {uname.release} ({uname.version})")
-        # Python 3 returns a named tuple, but Python 2 is simpler.
+    processor = uname.processor if uname.processor else "unknown"
-        print(uname)
+    print(f"CPU {uname.machine} ({processor})")
-    else:
-        print("OS %s %s (%s)" % (uname.system, uname.release, uname.version))
-        print(
-            "CPU %s (%s)"
-            % (uname.machine, uname.processor if uname.processor else "unknown")
-        )
     print("Bug reports:", BUG_URL)
     sys.exit(0)
 
@@ -1530,6 +1462,14 @@ def main(orig_args):
     if reqs:
         reqs.assert_all()
 
+    # Python 3.11 introduces PYTHONSAFEPATH and the -P flag which, if enabled,
+    # does not prepend the script's directory to sys.path by default.
+    # repo relies on this import path, so add directory of REPO_MAIN to
+    # PYTHONPATH so that this continues to work when PYTHONSAFEPATH is enabled.
+    python_paths = os.environ.get("PYTHONPATH", "").split(os.pathsep)
+    new_python_paths = [os.path.join(rel_repo_dir, S_repo)] + python_paths
+    os.environ["PYTHONPATH"] = os.pathsep.join(new_python_paths)
+
     ver_str = ".".join(map(str, VERSION))
     me = [
         sys.executable,

repo_logging.py

@@ -77,6 +77,7 @@ class RepoLogger(logging.Logger):
 
         if not err.aggregate_errors:
             self.error("Repo command failed: %s", type(err).__name__)
+            self.error("\t%s", str(err))
             return
 
         self.error(

repo_trace.py

@@ -142,7 +142,7 @@ def _GetTraceFile(quiet):
 def _ClearOldTraces():
     """Clear the oldest commands if trace file is too big."""
     try:
-        with open(_TRACE_FILE, "r", errors="ignore") as f:
+        with open(_TRACE_FILE, errors="ignore") as f:
             if os.path.getsize(f.name) / (1024 * 1024) <= _MAX_SIZE:
                 return
             trace_lines = f.readlines()

requirements.json

@@ -46,12 +46,16 @@
 
   # Supported git versions.
   #
-  # git-1.7.2 is in Debian Squeeze.
   # git-1.7.9 is in Ubuntu Precise.
-  # git-1.9.1 is in Ubuntu Trusty.
   # git-1.7.10 is in Debian Wheezy.
+  # git-1.9.1 is in Ubuntu Trusty.
+  # git-2.1.4 is in Debian Jessie.
+  # git-2.7.4 is in Ubuntu Xenial.
+  # git-2.11.0 is in Debian Stretch.
+  # git-2.17.0 is in Ubuntu Bionic.
+  # git-2.20.1 is in Debian Buster.
   "git": {
-    "hard": [1, 7, 2],
+    "hard": [1, 7, 9],
-    "soft": [1, 9, 1]
+    "soft": [2, 7, 4]
   }
 }

ssh.py

@@ -57,8 +57,12 @@ def version():
     except FileNotFoundError:
         print("fatal: ssh not installed", file=sys.stderr)
         sys.exit(1)
-    except subprocess.CalledProcessError:
+    except subprocess.CalledProcessError as e:
-        print("fatal: unable to detect ssh version", file=sys.stderr)
+        print(
+            "fatal: unable to detect ssh version"
+            f" (code={e.returncode}, output={e.stdout})",
+            file=sys.stderr,
+        )
         sys.exit(1)
 
 
@@ -165,7 +169,7 @@ class ProxyManager:
         # Check to see whether we already think that the master is running; if
         # we think it's already running, return right away.
         if port is not None:
-            key = "%s:%s" % (host, port)
+            key = f"{host}:{port}"
         else:
             key = host
 

subcmds/__init__.py

@@ -37,9 +37,7 @@ for py in os.listdir(my_dir):
         try:
             cmd = getattr(mod, clsn)
         except AttributeError:
-            raise SyntaxError(
+            raise SyntaxError(f"{__name__}/{py} does not define class {clsn}")
-                "%s/%s does not define class %s" % (__name__, py, clsn)
-            )
 
         name = name.replace("_", "-")
         cmd.NAME = name

subcmds/abandon.py

@@ -117,7 +117,7 @@ It is equivalent to "git branch -D <branchname>".
             all_projects,
             callback=_ProcessResults,
             output=Progress(
-                "Abandon %s" % (nb,), len(all_projects), quiet=opt.quiet
+                f"Abandon {nb}", len(all_projects), quiet=opt.quiet
             ),
         )
 
@@ -152,4 +152,4 @@ It is equivalent to "git branch -D <branchname>".
                             _RelPath(p) for p in success[br]
                         )
                     )
-                print("%s%s| %s\n" % (br, " " * (width - len(br)), result))
+                print(f"{br}{' ' * (width - len(br))}| {result}\n")

subcmds/branches.py

@@ -28,7 +28,7 @@ class BranchColoring(Coloring):
         self.notinproject = self.printer("notinproject", fg="red")
 
 
-class BranchInfo(object):
+class BranchInfo:
     def __init__(self, name):
         self.name = name
         self.current = 0
@@ -174,7 +174,7 @@ is shown, then the branch appears in all projects.
                         if _RelPath(p) not in have:
                             paths.append(_RelPath(p))
 
-                s = " %s %s" % (in_type, ", ".join(paths))
+                s = f" {in_type} {', '.join(paths)}"
                 if not i.IsSplitCurrent and (width + 7 + len(s) < 80):
                     fmt = out.current if i.IsCurrent else fmt
                     fmt(s)

subcmds/checkout.py

@@ -96,7 +96,7 @@ The command is equivalent to:
             all_projects,
             callback=_ProcessResults,
             output=Progress(
-                "Checkout %s" % (nb,), len(all_projects), quiet=opt.quiet
+                f"Checkout {nb}", len(all_projects), quiet=opt.quiet
             ),
         )
 

subcmds/cherry_pick.py

@@ -86,7 +86,7 @@ change id will be added.
             p.Wait()
         except GitError as e:
             logger.error(e)
-            logger.warn(
+            logger.warning(
                 "NOTE: When committing (please see above) and editing the "
                 "commit message, please remove the old Change-Id-line and "
                 "add:\n%s",

subcmds/diffmanifests.py

@@ -87,25 +87,17 @@ synced and their revisions won't be found.
     def _printRawDiff(self, diff, pretty_format=None, local=False):
         _RelPath = lambda p: p.RelPath(local=local)
         for project in diff["added"]:
-            self.printText(
+            self.printText(f"A {_RelPath(project)} {project.revisionExpr}")
-                "A %s %s" % (_RelPath(project), project.revisionExpr)
-            )
             self.out.nl()
 
         for project in diff["removed"]:
-            self.printText(
+            self.printText(f"R {_RelPath(project)} {project.revisionExpr}")
-                "R %s %s" % (_RelPath(project), project.revisionExpr)
-            )
             self.out.nl()
 
         for project, otherProject in diff["changed"]:
             self.printText(
-                "C %s %s %s"
+                f"C {_RelPath(project)} {project.revisionExpr} "
-                % (
+                f"{otherProject.revisionExpr}"
-                    _RelPath(project),
-                    project.revisionExpr,
-                    otherProject.revisionExpr,
-                )
             )
             self.out.nl()
             self._printLogs(
@@ -118,12 +110,8 @@ synced and their revisions won't be found.
 
         for project, otherProject in diff["unreachable"]:
             self.printText(
-                "U %s %s %s"
+                f"U {_RelPath(project)} {project.revisionExpr} "
-                % (
+                f"{otherProject.revisionExpr}"
-                    _RelPath(project),
-                    project.revisionExpr,
-                    otherProject.revisionExpr,
-                )
             )
             self.out.nl()
 

subcmds/help.py

@@ -150,7 +150,7 @@ Displays detailed usage information about a command.
     def _PrintAllCommandHelp(self):
         for name in sorted(all_commands):
             cmd = all_commands[name](manifest=self.manifest)
-            self._PrintCommandHelp(cmd, header_prefix="[%s] " % (name,))
+            self._PrintCommandHelp(cmd, header_prefix=f"[{name}] ")
 
     def _Options(self, p):
         p.add_option(

subcmds/info.py

@@ -97,7 +97,9 @@ class Info(PagedCommand):
             self.headtext(self.manifest.default.revisionExpr)
         self.out.nl()
         self.heading("Manifest merge branch: ")
-        self.headtext(mergeBranch)
+        # The manifest might not have a merge branch if it isn't in a git repo,
+        # e.g. if `repo init --standalone-manifest` is used.
+        self.headtext(mergeBranch or "")
         self.out.nl()
         self.heading("Manifest groups: ")
         self.headtext(manifestGroups)
@@ -248,7 +250,7 @@ class Info(PagedCommand):
 
             for commit in commits:
                 split = commit.split()
-                self.text("{0:38}{1} ".format("", "-"))
+                self.text(f"{'':38}{'-'} ")
                 self.sha(split[0] + " ")
                 self.text(" ".join(split[1:]))
                 self.out.nl()

subcmds/init.py

@@ -215,7 +215,7 @@ to update the working directory files.
 
             if not opt.quiet:
                 print()
-            print("Your identity is: %s <%s>" % (name, email))
+            print(f"Your identity is: {name} <{email}>")
             print("is this correct [y/N]? ", end="", flush=True)
             a = sys.stdin.readline().strip().lower()
             if a in ("yes", "y", "t", "true"):
@@ -353,7 +353,7 @@ to update the working directory files.
             wrapper = Wrapper()
             try:
                 remote_ref, rev = wrapper.check_repo_rev(
-                    rp.gitdir,
+                    rp.worktree,
                     opt.repo_rev,
                     repo_verify=opt.repo_verify,
                     quiet=opt.quiet,

subcmds/list.py

@@ -131,7 +131,7 @@ This is similar to running: repo forall -c 'echo "$REPO_PATH : $REPO_PROJECT"'.
             elif opt.path_only and not opt.name_only:
                 lines.append("%s" % (_getpath(project)))
             else:
-                lines.append("%s : %s" % (_getpath(project), project.name))
+                lines.append(f"{_getpath(project)} : {project.name}")
 
         if lines:
             lines.sort()

subcmds/manifest.py

@@ -136,7 +136,7 @@ to indicate the remote ref to push changes to via 'repo upload'.
             manifest.SetUseLocalManifests(not opt.ignore_local_manifests)
 
             if opt.json:
-                logger.warn("warning: --json is experimental!")
+                logger.warning("warning: --json is experimental!")
                 doc = manifest.ToDict(
                     peg_rev=opt.peg_rev,
                     peg_rev_upstream=opt.peg_rev_upstream,
@@ -163,13 +163,13 @@ to indicate the remote ref to push changes to via 'repo upload'.
             if output_file != "-":
                 fd.close()
                 if manifest.path_prefix:
-                    logger.warn(
+                    logger.warning(
                         "Saved %s submanifest to %s",
                         manifest.path_prefix,
                         output_file,
                     )
                 else:
-                    logger.warn("Saved manifest to %s", output_file)
+                    logger.warning("Saved manifest to %s", output_file)
 
     def ValidateOptions(self, opt, args):
         if args:

subcmds/prune.py

@@ -83,9 +83,7 @@ class Prune(PagedCommand):
             )
 
             if not branch.base_exists:
-                print(
+                print(f"(ignoring: tracking branch is gone: {branch.base})")
-                    "(ignoring: tracking branch is gone: %s)" % (branch.base,)
-                )
             else:
                 commits = branch.commits
                 date = branch.date

subcmds/rebase.py

@@ -113,7 +113,7 @@ branch but need to incorporate new upstream changes "underneath" them.
             )
 
             if len(args) == 1:
-                logger.warn(
+                logger.warning(
                     "note: project %s is mapped to more than one path", args[0]
                 )
 

subcmds/start.py

@@ -130,7 +130,7 @@ revision specified in the manifest.
             all_projects,
             callback=_ProcessResults,
             output=Progress(
-                "Starting %s" % (nb,), len(all_projects), quiet=opt.quiet
+                f"Starting {nb}", len(all_projects), quiet=opt.quiet
             ),
         )
 

subcmds/sync.py

@@ -21,7 +21,7 @@ import multiprocessing
 import netrc
 import optparse
 import os
-import socket
+from pathlib import Path
 import sys
 import tempfile
 import time
@@ -83,16 +83,54 @@ from wrapper import Wrapper
 
 _ONE_DAY_S = 24 * 60 * 60
 
-# Env var to implicitly turn auto-gc back on.  This was added to allow a user to
-# revert a change in default behavior in v2.29.9.  Remove after 2023-04-01.
-_REPO_AUTO_GC = "REPO_AUTO_GC"
-_AUTO_GC = os.environ.get(_REPO_AUTO_GC) == "1"
-
 _REPO_ALLOW_SHALLOW = os.environ.get("REPO_ALLOW_SHALLOW")
 
 logger = RepoLogger(__file__)
 
 
+def _SafeCheckoutOrder(checkouts: List[Project]) -> List[List[Project]]:
+    """Generate a sequence of checkouts that is safe to perform. The client
+    should checkout everything from n-th index before moving to n+1.
+
+    This is only useful if manifest contains nested projects.
+
+    E.g. if foo, foo/bar and foo/bar/baz are project paths, then foo needs to
+    finish before foo/bar can proceed, and foo/bar needs to finish before
+    foo/bar/baz."""
+    res = [[]]
+    current = res[0]
+
+    # depth_stack contains a current stack of parent paths.
+    depth_stack = []
+    # Checkouts are iterated in the hierarchical order. That way, it can easily
+    # be determined if the previous checkout is parent of the current checkout.
+    # We are splitting by the path separator so the final result is
+    # hierarchical, and not just lexicographical. For example, if the projects
+    # are: foo, foo/bar, foo-bar, lexicographical order produces foo, foo-bar
+    # and foo/bar, which doesn't work.
+    for checkout in sorted(checkouts, key=lambda x: x.relpath.split("/")):
+        checkout_path = Path(checkout.relpath)
+        while depth_stack:
+            try:
+                checkout_path.relative_to(depth_stack[-1])
+            except ValueError:
+                # Path.relative_to returns ValueError if paths are not relative.
+                # TODO(sokcevic): Switch to is_relative_to once min supported
+                # version is py3.9.
+                depth_stack.pop()
+            else:
+                if len(depth_stack) >= len(res):
+                    # Another depth created.
+                    res.append([])
+                break
+
+        current = res[len(depth_stack)]
+        current.append(checkout)
+        depth_stack.append(checkout_path)
+
+    return res
+
+
 class _FetchOneResult(NamedTuple):
     """_FetchOne return value.
 
@@ -186,9 +224,10 @@ class TeeStringIO(io.StringIO):
 
     def write(self, s: str) -> int:
         """Write to additional destination."""
-        super().write(s)
+        ret = super().write(s)
         if self.io is not None:
             self.io.write(s)
+        return ret
 
 
 class Sync(Command, MirrorSafeCommand):
@@ -243,6 +282,11 @@ directories if they have previously been linked to a different
 object directory. WARNING: This may cause data to be lost since
 refs may be removed when overwriting.
 
+The --force-checkout option can be used to force git to switch revs even if the
+index or the working tree differs from HEAD, and if there are untracked files.
+WARNING: This may cause data to be lost since uncommitted changes may be
+removed.
+
 The --force-remove-dirty option can be used to remove previously used
 projects with uncommitted changes. WARNING: This may cause data to be
 lost since uncommitted changes may be removed with projects that no longer
@@ -340,6 +384,14 @@ later is required to fix a server side protocol bug.
             "point to a different object directory. WARNING: this "
             "may cause loss of data",
         )
+        p.add_option(
+            "--force-checkout",
+            dest="force_checkout",
+            action="store_true",
+            help="force checkout even if it results in throwing away "
+            "uncommitted modifications. "
+            "WARNING: this may cause loss of data",
+        )
         p.add_option(
             "--force-remove-dirty",
             dest="force_remove_dirty",
@@ -618,7 +670,7 @@ later is required to fix a server side protocol bug.
 
             if not use_super:
                 continue
-            m.superproject.SetQuiet(opt.quiet)
+            m.superproject.SetQuiet(not opt.verbose)
             print_messages = git_superproject.PrintMessages(
                 opt.use_superproject, m
             )
@@ -943,7 +995,7 @@ later is required to fix a server side protocol bug.
                 break
             # Stop us from non-stopped fetching actually-missing repos: If set
             # of missing repos has not been changed from last fetch, we break.
-            missing_set = set(p.name for p in missing)
+            missing_set = {p.name for p in missing}
             if previously_missing_set == missing_set:
                 break
             previously_missing_set = missing_set
@@ -956,12 +1008,18 @@ later is required to fix a server side protocol bug.
 
         return _FetchMainResult(all_projects)
 
-    def _CheckoutOne(self, detach_head, force_sync, project):
+    def _CheckoutOne(
+        self, detach_head, force_sync, force_checkout, verbose, project
+    ):
         """Checkout work tree for one project
 
         Args:
             detach_head: Whether to leave a detached HEAD.
-            force_sync: Force checking out of the repo.
+            force_sync: Force checking out of .git directory (e.g. overwrite
+            existing git directory that was previously linked to a different
+            object directory).
+            force_checkout: Force checking out of the repo content.
+            verbose: Whether to show verbose messages.
             project: Project object for the project to checkout.
 
         Returns:
@@ -975,7 +1033,11 @@ later is required to fix a server side protocol bug.
         errors = []
         try:
             project.Sync_LocalHalf(
-                syncbuf, force_sync=force_sync, errors=errors
+                syncbuf,
+                force_sync=force_sync,
+                force_checkout=force_checkout,
+                errors=errors,
+                verbose=verbose,
             )
             success = syncbuf.Finish()
         except GitError as e:
@@ -1039,15 +1101,22 @@ later is required to fix a server side protocol bug.
                 pm.update(msg=project.name)
             return ret
 
-        proc_res = self.ExecuteInParallel(
+        for projects in _SafeCheckoutOrder(all_projects):
-            opt.jobs_checkout,
+            proc_res = self.ExecuteInParallel(
-            functools.partial(
+                opt.jobs_checkout,
-                self._CheckoutOne, opt.detach_head, opt.force_sync
+                functools.partial(
-            ),
+                    self._CheckoutOne,
-            all_projects,
+                    opt.detach_head,
-            callback=_ProcessResults,
+                    opt.force_sync,
-            output=Progress("Checking out", len(all_projects), quiet=opt.quiet),
+                    opt.force_checkout,
-        )
+                    opt.verbose,
+                ),
+                projects,
+                callback=_ProcessResults,
+                output=Progress(
+                    "Checking out", len(all_projects), quiet=opt.quiet
+                ),
+            )
 
         self._local_sync_state.Save()
         return proc_res and not err_results
@@ -1130,8 +1199,6 @@ later is required to fix a server side protocol bug.
                     )
                     project.config.SetString("gc.pruneExpire", "never")
             else:
-                if not opt.quiet:
-                    print(f"\r{relpath}: not shared, disabling pruning.")
                 project.config.SetString("extensions.preciousObjects", None)
                 project.config.SetString("gc.pruneExpire", None)
 
@@ -1265,7 +1332,7 @@ later is required to fix a server side protocol bug.
         old_project_paths = []
 
         if os.path.exists(file_path):
-            with open(file_path, "r") as fd:
+            with open(file_path) as fd:
                 old_project_paths = fd.read().split("\n")
             # In reversed order, so subfolders are deleted before parent folder.
             for path in sorted(old_project_paths, reverse=True):
@@ -1290,7 +1357,7 @@ later is required to fix a server side protocol bug.
                             groups=None,
                         )
                         project.DeleteWorktree(
-                            quiet=opt.quiet, force=opt.force_remove_dirty
+                            verbose=opt.verbose, force=opt.force_remove_dirty
                         )
 
         new_project_paths.sort()
@@ -1376,7 +1443,7 @@ later is required to fix a server side protocol bug.
             else:
                 try:
                     info = netrc.netrc()
-                except IOError:
+                except OSError:
                     # .netrc file does not exist or could not be opened.
                     pass
                 else:
@@ -1396,7 +1463,7 @@ later is required to fix a server side protocol bug.
 
             if username and password:
                 manifest_server = manifest_server.replace(
-                    "://", "://%s:%s@" % (username, password), 1
+                    "://", f"://{username}:{password}@", 1
                 )
 
         transport = PersistentTransport(manifest_server)
@@ -1435,7 +1502,7 @@ later is required to fix a server side protocol bug.
                 try:
                     with open(smart_sync_manifest_path, "w") as f:
                         f.write(manifest_str)
-                except IOError as e:
+                except OSError as e:
                     raise SmartSyncError(
                         "error: cannot write manifest to %s:\n%s"
                         % (smart_sync_manifest_path, e),
@@ -1446,7 +1513,7 @@ later is required to fix a server side protocol bug.
                 raise SmartSyncError(
                     "error: manifest server RPC call failed: %s" % manifest_str
                 )
-        except (socket.error, IOError, xmlrpc.client.Fault) as e:
+        except (OSError, xmlrpc.client.Fault) as e:
             raise SmartSyncError(
                 "error: cannot connect to manifest server %s:\n%s"
                 % (manifest.manifest_server, e),
@@ -1502,7 +1569,7 @@ later is required to fix a server side protocol bug.
             buf = TeeStringIO(sys.stdout)
             try:
                 result = mp.Sync_NetworkHalf(
-                    quiet=opt.quiet,
+                    quiet=not opt.verbose,
                     output_redir=buf,
                     verbose=opt.verbose,
                     current_branch_only=self._GetCurrentBranchOnly(
@@ -1535,16 +1602,17 @@ later is required to fix a server side protocol bug.
             syncbuf = SyncBuffer(mp.config)
             start = time.time()
             mp.Sync_LocalHalf(
-                syncbuf, submodules=mp.manifest.HasSubmodules, errors=errors
+                syncbuf,
+                submodules=mp.manifest.HasSubmodules,
+                errors=errors,
+                verbose=opt.verbose,
             )
             clean = syncbuf.Finish()
             self.event_log.AddSync(
                 mp, event_log.TASK_SYNC_LOCAL, start, time.time(), clean
             )
             if not clean:
-                raise UpdateManifestError(
+                raise UpdateManifestError(aggregate_errors=errors)
-                    aggregate_errors=errors, project=mp.name
-                )
             self._ReloadManifest(manifest_name, mp.manifest)
 
     def ValidateOptions(self, opt, args):
@@ -1575,16 +1643,6 @@ later is required to fix a server side protocol bug.
         if opt.prune is None:
             opt.prune = True
 
-        if opt.auto_gc is None and _AUTO_GC:
-            logger.error(
-                "Will run `git gc --auto` because %s is set. %s is deprecated "
-                "and will be removed in a future release.  Use `--auto-gc` "
-                "instead.",
-                _REPO_AUTO_GC,
-                _REPO_AUTO_GC,
-            )
-            opt.auto_gc = True
-
     def _ValidateOptionsWithManifest(self, opt, mp):
         """Like ValidateOptions, but after we've updated the manifest.
 
@@ -1628,7 +1686,7 @@ later is required to fix a server side protocol bug.
         errors = []
         try:
             self._ExecuteHelper(opt, args, errors)
-        except RepoExitError:
+        except (RepoExitError, RepoChangedException):
             raise
         except (KeyboardInterrupt, Exception) as e:
             raise RepoUnhandledExceptionError(e, aggregate_errors=errors)
@@ -1779,7 +1837,6 @@ later is required to fix a server side protocol bug.
                     logger.error("error: Local checkouts *not* updated.")
                     raise SyncFailFastError(aggregate_errors=errors)
 
-            err_update_linkfiles = False
             try:
                 self.UpdateCopyLinkfileList(m)
             except Exception as e:
@@ -1877,7 +1934,7 @@ def _PostRepoUpgrade(manifest, quiet=False):
 
 def _PostRepoFetch(rp, repo_verify=True, verbose=False):
     if rp.HasChanges:
-        logger.warn("info: A new version of repo is available")
+        logger.warning("info: A new version of repo is available")
         wrapper = Wrapper()
         try:
             rev = rp.bare_git.describe(rp.GetRevisionId())
@@ -1905,10 +1962,10 @@ def _PostRepoFetch(rp, repo_verify=True, verbose=False):
             logger.warning("warning: Skipped upgrade to unverified version")
     else:
         if verbose:
-            print("repo version %s is current", rp.work_git.describe(HEAD))
+            print("repo version %s is current" % rp.work_git.describe(HEAD))
 
 
-class _FetchTimes(object):
+class _FetchTimes:
     _ALPHA = 0.5
 
     def __init__(self, manifest):
@@ -1931,7 +1988,7 @@ class _FetchTimes(object):
             try:
                 with open(self._path) as f:
                     self._saved = json.load(f)
-            except (IOError, ValueError):
+            except (OSError, ValueError):
                 platform_utils.remove(self._path, missing_ok=True)
                 self._saved = {}
 
@@ -1947,11 +2004,11 @@ class _FetchTimes(object):
         try:
             with open(self._path, "w") as f:
                 json.dump(self._seen, f, indent=2)
-        except (IOError, TypeError):
+        except (OSError, TypeError):
             platform_utils.remove(self._path, missing_ok=True)
 
 
-class LocalSyncState(object):
+class LocalSyncState:
     _LAST_FETCH = "last_fetch"
     _LAST_CHECKOUT = "last_checkout"
 
@@ -1994,7 +2051,7 @@ class LocalSyncState(object):
             try:
                 with open(self._path) as f:
                     self._state = json.load(f)
-            except (IOError, ValueError):
+            except (OSError, ValueError):
                 platform_utils.remove(self._path, missing_ok=True)
                 self._state = {}
 
@@ -2004,7 +2061,7 @@ class LocalSyncState(object):
         try:
             with open(self._path, "w") as f:
                 json.dump(self._state, f, indent=2)
-        except (IOError, TypeError):
+        except (OSError, TypeError):
             platform_utils.remove(self._path, missing_ok=True)
 
     def PruneRemovedProjects(self):
@@ -2014,7 +2071,7 @@ class LocalSyncState(object):
         delete = set()
         for path in self._state:
             gitdir = os.path.join(self._manifest.topdir, path, ".git")
-            if not os.path.exists(gitdir):
+            if not os.path.exists(gitdir) or os.path.islink(gitdir):
                 delete.add(path)
         if not delete:
             return
@@ -2046,6 +2103,7 @@ class LocalSyncState(object):
 # is passed during initialization.
 class PersistentTransport(xmlrpc.client.Transport):
     def __init__(self, orig_host):
+        super().__init__()
         self.orig_host = orig_host
 
     def request(self, host, handler, request_body, verbose=False):
@@ -2137,7 +2195,7 @@ class PersistentTransport(xmlrpc.client.Transport):
             try:
                 p.feed(data)
             except xml.parsers.expat.ExpatError as e:
-                raise IOError(
+                raise OSError(
                     f"Parsing the manifest failed: {e}\n"
                     f"Please report this to your manifest server admin.\n"
                     f'Here is the full response:\n{data.decode("utf-8")}'

subcmds/upload.py

@@ -72,16 +72,16 @@ def _VerifyPendingCommits(branches: List[ReviewableBranch]) -> bool:
     # If any branch has many commits, prompt the user.
     if many_commits:
         if len(branches) > 1:
-            logger.warn(
+            logger.warning(
                 "ATTENTION: One or more branches has an unusually high number "
                 "of commits."
             )
         else:
-            logger.warn(
+            logger.warning(
                 "ATTENTION: You are uploading an unusually high number of "
                 "commits."
             )
-        logger.warn(
+        logger.warning(
             "YOU PROBABLY DO NOT MEAN TO DO THIS. (Did you rebase across "
             "branches?)"
         )
@@ -244,6 +244,12 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
             default=[],
             help="add a label when uploading",
         )
+        p.add_option(
+            "--pd",
+            "--patchset-description",
+            dest="patchset_description",
+            help="description for patchset",
+        )
         p.add_option(
             "--re",
             "--reviewers",
@@ -655,6 +661,7 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
             dest_branch=destination,
             validate_certs=opt.validate_certs,
             push_options=opt.push_options,
+            patchset_description=opt.patchset_description,
         )
 
         branch.uploaded = True

subcmds/version.py

@@ -42,35 +42,28 @@ class Version(Command, MirrorSafeCommand):
         # These might not be the same.  Report them both.
         src_ver = RepoSourceVersion()
         rp_ver = rp.bare_git.describe(HEAD)
-        print("repo version %s" % rp_ver)
+        print(f"repo version {rp_ver}")
-        print("       (from %s)" % rem.url)
+        print(f"       (from {rem.url})")
-        print("       (tracking %s)" % branch.merge)
+        print(f"       (tracking {branch.merge})")
-        print("       (%s)" % rp.bare_git.log("-1", "--format=%cD", HEAD))
+        print(f"       ({rp.bare_git.log('-1', '--format=%cD', HEAD)})")
 
         if self.wrapper_path is not None:
-            print("repo launcher version %s" % self.wrapper_version)
+            print(f"repo launcher version {self.wrapper_version}")
-            print("       (from %s)" % self.wrapper_path)
+            print(f"       (from {self.wrapper_path})")
 
             if src_ver != rp_ver:
-                print("       (currently at %s)" % src_ver)
+                print(f"       (currently at {src_ver})")
 
-        print("repo User-Agent %s" % user_agent.repo)
+        print(f"repo User-Agent {user_agent.repo}")
-        print("git %s" % git.version_tuple().full)
+        print(f"git {git.version_tuple().full}")
-        print("git User-Agent %s" % user_agent.git)
+        print(f"git User-Agent {user_agent.git}")
-        print("Python %s" % sys.version)
+        print(f"Python {sys.version}")
         uname = platform.uname()
         if sys.version_info.major < 3:
             # Python 3 returns a named tuple, but Python 2 is simpler.
             print(uname)
         else:
-            print(
+            print(f"OS {uname.system} {uname.release} ({uname.version})")
-                "OS %s %s (%s)" % (uname.system, uname.release, uname.version)
+            processor = uname.processor if uname.processor else "unknown"
-            )
+            print(f"CPU {uname.machine} ({processor})")
-            print(
-                "CPU %s (%s)"
-                % (
-                    uname.machine,
-                    uname.processor if uname.processor else "unknown",
-                )
-            )
         print("Bug reports:", Wrapper().BUG_URL)

tests/conftest.py

@@ -14,8 +14,11 @@
 
 """Common fixtures for pytests."""
 
+import pathlib
+
 import pytest
 
+import platform_utils
 import repo_trace
 
 
@@ -23,3 +26,58 @@ import repo_trace
 def disable_repo_trace(tmp_path):
     """Set an environment marker to relax certain strict checks for test code."""  # noqa: E501
     repo_trace._TRACE_FILE = str(tmp_path / "TRACE_FILE_from_test")
+
+
+# adapted from pytest-home 0.5.1
+def _set_home(monkeypatch, path: pathlib.Path):
+    """
+    Set the home dir using a pytest monkeypatch context.
+    """
+    win = platform_utils.isWindows()
+    vars = ["HOME"] + win * ["USERPROFILE"]
+    for var in vars:
+        monkeypatch.setenv(var, str(path))
+    return path
+
+
+# copied from
+# https://github.com/pytest-dev/pytest/issues/363#issuecomment-1335631998
+@pytest.fixture(scope="session")
+def monkeysession():
+    with pytest.MonkeyPatch.context() as mp:
+        yield mp
+
+
+@pytest.fixture(autouse=True, scope="session")
+def session_tmp_home_dir(tmp_path_factory, monkeysession):
+    """Set HOME to a temporary directory, avoiding user's .gitconfig.
+
+    b/302797407
+
+    Set home at session scope to take effect prior to
+    ``test_wrapper.GitCheckoutTestCase.setUpClass``.
+    """
+    return _set_home(monkeysession, tmp_path_factory.mktemp("home"))
+
+
+# adapted from pytest-home 0.5.1
+@pytest.fixture(autouse=True)
+def tmp_home_dir(monkeypatch, tmp_path_factory):
+    """Set HOME to a temporary directory.
+
+    Ensures that state doesn't accumulate in $HOME across tests.
+
+    Note that in conjunction with session_tmp_homedir, the HOME
+    dir is patched twice, once at session scope, and then again at
+    the function scope.
+    """
+    return _set_home(monkeypatch, tmp_path_factory.mktemp("home"))
+
+
+@pytest.fixture(autouse=True)
+def setup_user_identity(monkeysession, scope="session"):
+    """Set env variables for author and committer name and email."""
+    monkeysession.setenv("GIT_AUTHOR_NAME", "Foo Bar")
+    monkeysession.setenv("GIT_COMMITTER_NAME", "Foo Bar")
+    monkeysession.setenv("GIT_AUTHOR_EMAIL", "foo@bar.baz")
+    monkeysession.setenv("GIT_COMMITTER_EMAIL", "foo@bar.baz")

tests/test_git_command.py

@@ -14,16 +14,12 @@
 
 """Unittests for the git_command.py module."""
 
+import io
 import os
 import re
 import subprocess
 import unittest
-
+from unittest import mock
-
-try:
-    from unittest import mock
-except ImportError:
-    import mock
 
 import git_command
 import wrapper
@@ -71,9 +67,13 @@ class GitCommandWaitTest(unittest.TestCase):
     """Tests the GitCommand class .Wait()"""
 
     def setUp(self):
-        class MockPopen(object):
+        class MockPopen:
             rc = 0
 
+            def __init__(self):
+                self.stdout = io.BufferedReader(io.BytesIO())
+                self.stderr = io.BufferedReader(io.BytesIO())
+
             def communicate(
                 self, input: str = None, timeout: float = None
             ) -> [str, str]:
@@ -117,6 +117,115 @@ class GitCommandWaitTest(unittest.TestCase):
         self.assertEqual(1, r.Wait())
 
 
+class GitCommandStreamLogsTest(unittest.TestCase):
+    """Tests the GitCommand class stderr log streaming cases."""
+
+    def setUp(self):
+        self.mock_process = mock.MagicMock()
+        self.mock_process.communicate.return_value = (None, None)
+        self.mock_process.wait.return_value = 0
+
+        self.mock_popen = mock.MagicMock()
+        self.mock_popen.return_value = self.mock_process
+        mock.patch("subprocess.Popen", self.mock_popen).start()
+
+    def tearDown(self):
+        mock.patch.stopall()
+
+    def test_does_not_stream_logs_when_input_is_set(self):
+        git_command.GitCommand(None, ["status"], input="foo")
+
+        self.mock_popen.assert_called_once_with(
+            ["git", "status"],
+            cwd=None,
+            env=mock.ANY,
+            encoding="utf-8",
+            errors="backslashreplace",
+            stdin=subprocess.PIPE,
+            stdout=None,
+            stderr=None,
+        )
+        self.mock_process.communicate.assert_called_once_with(input="foo")
+        self.mock_process.stderr.read1.assert_not_called()
+
+    def test_does_not_stream_logs_when_stdout_is_set(self):
+        git_command.GitCommand(None, ["status"], capture_stdout=True)
+
+        self.mock_popen.assert_called_once_with(
+            ["git", "status"],
+            cwd=None,
+            env=mock.ANY,
+            encoding="utf-8",
+            errors="backslashreplace",
+            stdin=None,
+            stdout=subprocess.PIPE,
+            stderr=None,
+        )
+        self.mock_process.communicate.assert_called_once_with(input=None)
+        self.mock_process.stderr.read1.assert_not_called()
+
+    def test_does_not_stream_logs_when_stderr_is_set(self):
+        git_command.GitCommand(None, ["status"], capture_stderr=True)
+
+        self.mock_popen.assert_called_once_with(
+            ["git", "status"],
+            cwd=None,
+            env=mock.ANY,
+            encoding="utf-8",
+            errors="backslashreplace",
+            stdin=None,
+            stdout=None,
+            stderr=subprocess.PIPE,
+        )
+        self.mock_process.communicate.assert_called_once_with(input=None)
+        self.mock_process.stderr.read1.assert_not_called()
+
+    def test_does_not_stream_logs_when_merge_output_is_set(self):
+        git_command.GitCommand(None, ["status"], merge_output=True)
+
+        self.mock_popen.assert_called_once_with(
+            ["git", "status"],
+            cwd=None,
+            env=mock.ANY,
+            encoding="utf-8",
+            errors="backslashreplace",
+            stdin=None,
+            stdout=None,
+            stderr=subprocess.STDOUT,
+        )
+        self.mock_process.communicate.assert_called_once_with(input=None)
+        self.mock_process.stderr.read1.assert_not_called()
+
+    @mock.patch("sys.stderr")
+    def test_streams_stderr_when_no_stream_is_set(self, mock_stderr):
+        logs = "\n".join(
+            [
+                "Enumerating objects: 5, done.",
+                "Counting objects: 100% (5/5), done.",
+                "Writing objects: 100% (3/3), 330 bytes | 330 KiB/s, done.",
+                "remote: Processing changes: refs: 1, new: 1, done ",
+                "remote: SUCCESS",
+            ]
+        )
+        self.mock_process.stderr = io.BufferedReader(
+            io.BytesIO(bytes(logs, "utf-8"))
+        )
+
+        cmd = git_command.GitCommand(None, ["push"])
+
+        self.mock_popen.assert_called_once_with(
+            ["git", "push"],
+            cwd=None,
+            env=mock.ANY,
+            stdin=None,
+            stdout=None,
+            stderr=subprocess.PIPE,
+        )
+        self.mock_process.communicate.assert_not_called()
+        mock_stderr.write.assert_called_once_with(logs)
+        self.assertEqual(cmd.stderr, logs)
+
+
 class GitCallUnitTest(unittest.TestCase):
     """Tests the _GitCall class (via git_command.git)."""
 
@@ -214,3 +323,22 @@ class GitRequireTests(unittest.TestCase):
         with self.assertRaises(git_command.GitRequireError) as e:
             git_command.git_require((2,), fail=True, msg="so sad")
             self.assertNotEqual(0, e.code)
+
+
+class GitCommandErrorTest(unittest.TestCase):
+    """Test for the GitCommandError class."""
+
+    def test_augument_stderr(self):
+        self.assertEqual(
+            git_command.GitCommandError(
+                git_stderr="couldn't find remote ref refs/heads/foo"
+            ).suggestion,
+            "Check if the provided ref exists in the remote.",
+        )
+
+        self.assertEqual(
+            git_command.GitCommandError(
+                git_stderr="'foobar' does not appear to be a git repository"
+            ).suggestion,
+            "Are you running this repo command outside of a repo workspace?",
+        )

tests/test_git_config.py

@@ -100,7 +100,7 @@ class GitConfigReadOnlyTests(unittest.TestCase):
             ("intg", 10737418240),
         )
         for key, value in TESTS:
-            self.assertEqual(value, self.config.GetInt("section.%s" % (key,)))
+            self.assertEqual(value, self.config.GetInt(f"section.{key}"))
 
 
 class GitConfigReadWriteTests(unittest.TestCase):

tests/test_git_superproject.py

@@ -34,7 +34,7 @@ class SuperprojectTestCase(unittest.TestCase):
     PARENT_SID_KEY = "GIT_TRACE2_PARENT_SID"
     PARENT_SID_VALUE = "parent_sid"
     SELF_SID_REGEX = r"repo-\d+T\d+Z-.*"
-    FULL_SID_REGEX = r"^%s/%s" % (PARENT_SID_VALUE, SELF_SID_REGEX)
+    FULL_SID_REGEX = rf"^{PARENT_SID_VALUE}/{SELF_SID_REGEX}"
 
     def setUp(self):
         """Set up superproject every time."""
@@ -249,7 +249,7 @@ class SuperprojectTestCase(unittest.TestCase):
         os.mkdir(self._superproject._superproject_path)
         manifest_path = self._superproject._WriteManifestFile()
         self.assertIsNotNone(manifest_path)
-        with open(manifest_path, "r") as fp:
+        with open(manifest_path) as fp:
             manifest_xml_data = fp.read()
         self.assertEqual(
             sort_attributes(manifest_xml_data),
@@ -284,7 +284,7 @@ class SuperprojectTestCase(unittest.TestCase):
                     )
                     self.assertIsNotNone(update_result.manifest_path)
                     self.assertFalse(update_result.fatal)
-                    with open(update_result.manifest_path, "r") as fp:
+                    with open(update_result.manifest_path) as fp:
                         manifest_xml_data = fp.read()
                     self.assertEqual(
                         sort_attributes(manifest_xml_data),
@@ -371,7 +371,7 @@ class SuperprojectTestCase(unittest.TestCase):
                     )
                     self.assertIsNotNone(update_result.manifest_path)
                     self.assertFalse(update_result.fatal)
-                    with open(update_result.manifest_path, "r") as fp:
+                    with open(update_result.manifest_path) as fp:
                         manifest_xml_data = fp.read()
                     # Verify platform/vendor/x's project revision hasn't
                     # changed.
@@ -436,7 +436,7 @@ class SuperprojectTestCase(unittest.TestCase):
                     )
                     self.assertIsNotNone(update_result.manifest_path)
                     self.assertFalse(update_result.fatal)
-                    with open(update_result.manifest_path, "r") as fp:
+                    with open(update_result.manifest_path) as fp:
                         manifest_xml_data = fp.read()
                     # Verify platform/vendor/x's project revision hasn't
                     # changed.

tests/test_git_trace2_event_log.py

@@ -61,7 +61,7 @@ class EventLogTestCase(unittest.TestCase):
     PARENT_SID_KEY = "GIT_TRACE2_PARENT_SID"
     PARENT_SID_VALUE = "parent_sid"
     SELF_SID_REGEX = r"repo-\d+T\d+Z-.*"
-    FULL_SID_REGEX = r"^%s/%s" % (PARENT_SID_VALUE, SELF_SID_REGEX)
+    FULL_SID_REGEX = rf"^{PARENT_SID_VALUE}/{SELF_SID_REGEX}"
 
     def setUp(self):
         """Load the event_log module every time."""

tests/test_manifest_xml.py

@@ -198,13 +198,13 @@ class ValueTests(unittest.TestCase):
     def test_bool_true(self):
         """Check XmlBool true values."""
         for value in ("yes", "true", "1"):
-            node = self._get_node('<node a="%s"/>' % (value,))
+            node = self._get_node(f'<node a="{value}"/>')
             self.assertTrue(manifest_xml.XmlBool(node, "a"))
 
     def test_bool_false(self):
         """Check XmlBool false values."""
         for value in ("no", "false", "0"):
-            node = self._get_node('<node a="%s"/>' % (value,))
+            node = self._get_node(f'<node a="{value}"/>')
             self.assertFalse(manifest_xml.XmlBool(node, "a"))
 
     def test_int_default(self):
@@ -220,7 +220,7 @@ class ValueTests(unittest.TestCase):
     def test_int_good(self):
         """Check XmlInt numeric handling."""
         for value in (-1, 0, 1, 50000):
-            node = self._get_node('<node a="%s"/>' % (value,))
+            node = self._get_node(f'<node a="{value}"/>')
             self.assertEqual(value, manifest_xml.XmlInt(node, "a"))
 
     def test_int_invalid(self):
@@ -385,6 +385,21 @@ class XmlManifestTests(ManifestParseTestCase):
             "</manifest>",
         )
 
+    def test_parse_with_xml_doctype(self):
+        """Check correct manifest parse with DOCTYPE node present."""
+        manifest = self.getXmlManifest(
+            """<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE manifest []>
+<manifest>
+  <remote name="test-remote" fetch="http://localhost" />
+  <default remote="test-remote" revision="refs/heads/main" />
+  <project name="test-project" path="src/test-project"/>
+</manifest>
+"""
+        )
+        self.assertEqual(len(manifest.projects), 1)
+        self.assertEqual(manifest.projects[0].name, "test-project")
+
 
 class IncludeElementTests(ManifestParseTestCase):
     """Tests for <include>."""
@@ -1113,3 +1128,79 @@ class ExtendProjectElementTests(ManifestParseTestCase):
         )
         self.assertEqual(len(manifest.projects), 1)
         self.assertEqual(manifest.projects[0].upstream, "bar")
+
+
+class NormalizeUrlTests(ManifestParseTestCase):
+    """Tests for normalize_url() in manifest_xml.py"""
+
+    def test_has_trailing_slash(self):
+        url = "http://foo.com/bar/baz/"
+        self.assertEqual(
+            "http://foo.com/bar/baz", manifest_xml.normalize_url(url)
+        )
+
+        url = "http://foo.com/bar/"
+        self.assertEqual("http://foo.com/bar", manifest_xml.normalize_url(url))
+
+    def test_has_leading_slash(self):
+        """SCP-like syntax except a / comes before the : which git disallows."""
+        url = "/git@foo.com:bar/baf"
+        self.assertEqual(url, manifest_xml.normalize_url(url))
+
+        url = "gi/t@foo.com:bar/baf"
+        self.assertEqual(url, manifest_xml.normalize_url(url))
+
+        url = "git@fo/o.com:bar/baf"
+        self.assertEqual(url, manifest_xml.normalize_url(url))
+
+    def test_has_no_scheme(self):
+        """Deal with cases where we have no scheme, but we also
+        aren't dealing with the git SCP-like syntax
+        """
+        url = "foo.com/baf/bat"
+        self.assertEqual(url, manifest_xml.normalize_url(url))
+
+        url = "foo.com/baf"
+        self.assertEqual(url, manifest_xml.normalize_url(url))
+
+        url = "git@foo.com/baf/bat"
+        self.assertEqual(url, manifest_xml.normalize_url(url))
+
+        url = "git@foo.com/baf"
+        self.assertEqual(url, manifest_xml.normalize_url(url))
+
+        url = "/file/path/here"
+        self.assertEqual(url, manifest_xml.normalize_url(url))
+
+    def test_has_no_scheme_matches_scp_like_syntax(self):
+        url = "git@foo.com:bar/baf"
+        self.assertEqual(
+            "ssh://git@foo.com/bar/baf", manifest_xml.normalize_url(url)
+        )
+
+        url = "git@foo.com:bar/"
+        self.assertEqual(
+            "ssh://git@foo.com/bar", manifest_xml.normalize_url(url)
+        )
+
+    def test_remote_url_resolution(self):
+        remote = manifest_xml._XmlRemote(
+            name="foo",
+            fetch="git@github.com:org2/",
+            manifestUrl="git@github.com:org2/custom_manifest.git",
+        )
+        self.assertEqual("ssh://git@github.com/org2", remote.resolvedFetchUrl)
+
+        remote = manifest_xml._XmlRemote(
+            name="foo",
+            fetch="ssh://git@github.com/org2/",
+            manifestUrl="git@github.com:org2/custom_manifest.git",
+        )
+        self.assertEqual("ssh://git@github.com/org2", remote.resolvedFetchUrl)
+
+        remote = manifest_xml._XmlRemote(
+            name="foo",
+            fetch="git@github.com:org2/",
+            manifestUrl="ssh://git@github.com/org2/custom_manifest.git",
+        )
+        self.assertEqual("ssh://git@github.com/org2", remote.resolvedFetchUrl)

tests/test_project.py

@@ -48,7 +48,7 @@ def TempGitTree():
         yield tempdir
 
 
-class FakeProject(object):
+class FakeProject:
     """A fake for Project for basic functionality."""
 
     def __init__(self, worktree):
@@ -107,6 +107,16 @@ class ReviewableBranchTests(unittest.TestCase):
             self.assertTrue(rb.date)
 
 
+class ProjectTests(unittest.TestCase):
+    """Check Project behavior."""
+
+    def test_encode_patchset_description(self):
+        self.assertEqual(
+            project.Project._encode_patchset_description("abcd00!! +"),
+            "abcd00%21%21_%2b",
+        )
+
+
 class CopyLinkTestCase(unittest.TestCase):
     """TestCase for stub repo client checkouts.
 
@@ -151,7 +161,7 @@ class CopyLinkTestCase(unittest.TestCase):
                     # "".
                     break
                 result = os.path.exists(path)
-                msg.append("\tos.path.exists(%s): %s" % (path, result))
+                msg.append(f"\tos.path.exists({path}): {result}")
                 if result:
                     msg.append("\tcontents: %r" % os.listdir(path))
                     break
@@ -507,7 +517,10 @@ class ManifestPropertiesFetchedCorrectly(unittest.TestCase):
             self.assertFalse(fakeproj.partial_clone)
 
             fakeproj.config.SetString("repo.depth", "48")
-            self.assertEqual(fakeproj.depth, "48")
+            self.assertEqual(fakeproj.depth, 48)
+
+            fakeproj.config.SetString("repo.depth", "invalid_depth")
+            self.assertEqual(fakeproj.depth, None)
 
             fakeproj.config.SetString("repo.clonefilter", "blob:limit=10M")
             self.assertEqual(fakeproj.clone_filter, "blob:limit=10M")

tests/test_subcmds_sync.py

@@ -265,6 +265,95 @@ class LocalSyncState(unittest.TestCase):
         self.assertIsNone(self.state.GetFetchTime(projA))
         self.assertEqual(self.state.GetFetchTime(projB), 7)
 
+    def test_prune_removed_and_symlinked_projects(self):
+        """Removed projects that still exists on disk as symlink are pruned."""
+        with open(self.state._path, "w") as f:
+            f.write(
+                """
+            {
+              "projA": {
+                "last_fetch": 5
+              },
+              "projB": {
+                "last_fetch": 7
+              }
+            }
+            """
+            )
+
+        def mock_exists(path):
+            return True
+
+        def mock_islink(path):
+            if "projB" in path:
+                return True
+            return False
+
+        projA = mock.MagicMock(relpath="projA")
+        projB = mock.MagicMock(relpath="projB")
+        self.state = self._new_state()
+        self.assertEqual(self.state.GetFetchTime(projA), 5)
+        self.assertEqual(self.state.GetFetchTime(projB), 7)
+        with mock.patch("os.path.exists", side_effect=mock_exists):
+            with mock.patch("os.path.islink", side_effect=mock_islink):
+                self.state.PruneRemovedProjects()
+        self.assertIsNone(self.state.GetFetchTime(projB))
+
+        self.state = self._new_state()
+        self.assertIsNone(self.state.GetFetchTime(projB))
+        self.assertEqual(self.state.GetFetchTime(projA), 5)
+
+
+class FakeProject:
+    def __init__(self, relpath):
+        self.relpath = relpath
+
+    def __str__(self):
+        return f"project: {self.relpath}"
+
+    def __repr__(self):
+        return str(self)
+
+
+class SafeCheckoutOrder(unittest.TestCase):
+    def test_no_nested(self):
+        p_f = FakeProject("f")
+        p_foo = FakeProject("foo")
+        out = sync._SafeCheckoutOrder([p_f, p_foo])
+        self.assertEqual(out, [[p_f, p_foo]])
+
+    def test_basic_nested(self):
+        p_foo = p_foo = FakeProject("foo")
+        p_foo_bar = FakeProject("foo/bar")
+        out = sync._SafeCheckoutOrder([p_foo, p_foo_bar])
+        self.assertEqual(out, [[p_foo], [p_foo_bar]])
+
+    def test_complex_nested(self):
+        p_foo = FakeProject("foo")
+        p_foobar = FakeProject("foobar")
+        p_foo_dash_bar = FakeProject("foo-bar")
+        p_foo_bar = FakeProject("foo/bar")
+        p_foo_bar_baz_baq = FakeProject("foo/bar/baz/baq")
+        p_bar = FakeProject("bar")
+        out = sync._SafeCheckoutOrder(
+            [
+                p_foo_bar_baz_baq,
+                p_foo,
+                p_foobar,
+                p_foo_dash_bar,
+                p_foo_bar,
+                p_bar,
+            ]
+        )
+        self.assertEqual(
+            out,
+            [
+                [p_bar, p_foo, p_foo_dash_bar, p_foobar],
+                [p_foo_bar],
+                [p_foo_bar_baz_baq],
+            ],
+        )
+
 
 class GetPreciousObjectsState(unittest.TestCase):
     """Tests for _GetPreciousObjectsState."""

tests/test_wrapper.py

@@ -418,7 +418,7 @@ class SetupGnuPG(RepoWrapperTestCase):
                 self.wrapper.home_dot_repo, "gnupg"
             )
             self.assertTrue(self.wrapper.SetupGnuPG(True))
-            with open(os.path.join(tempdir, "keyring-version"), "r") as fp:
+            with open(os.path.join(tempdir, "keyring-version")) as fp:
                 data = fp.read()
             self.assertEqual(
                 ".".join(str(x) for x in self.wrapper.KEYRING_VERSION),

tox.ini

@@ -15,7 +15,7 @@
 # https://tox.readthedocs.io/
 
 [tox]
-envlist = lint, py36, py37, py38, py39, py310, py311
+envlist = lint, py36, py37, py38, py39, py310, py311, py312
 requires = virtualenv<20.22.0
 
 [gh-actions]
@@ -26,6 +26,7 @@ python =
     3.9: py39
     3.10: py310
     3.11: py311
+    3.12: py312
 
 [testenv]
 deps =
@@ -57,6 +58,3 @@ deps =
 commands =
     black {posargs:.}
     flake8
-
-[pytest]
-timeout = 300