Merge branch 'main' into debt/py36

Change-Id: I439bb1fdab1da524b28b2e27f85035b2fca5aa3b

docs/python-support.md

@@ -1,47 +1,93 @@
 # Supported Python Versions
 
-With Python 2.7 officially going EOL on [01 Jan 2020](https://pythonclock.org/),
+This documents the current supported Python versions, and tries to provide
-we need a support plan for the repo project itself.
+guidance for when we decide to drop support for older versions.
-Inevitably, there will be a long tail of users who still want to use Python 2 on
-their old LTS/corp systems and have little power to change the system.
 
 ## Summary
 
-* Python 3.6 (released Dec 2016) is required by default starting with repo-2.x.
+*   Python 3.6 (released Dec 2016) is required starting with repo-2.0.
-* Older versions of Python (e.g. v2.7) may use the legacy feature-frozen branch
+*   Older versions of Python (e.g. v2.7) may use old releases via the repo-1.x
-  based on repo-1.x.
+    branch, but no support is provided.
 
-## Overview
+## repo hooks
-
-We provide a branch for Python 2 users that is feature-frozen.
-Bugfixes may be added on a best-effort basis or from the community, but largely
-no new features will be added, nor is support guaranteed.
-
-Users can select this during `repo init` time via the [repo launcher].
-Otherwise the default branches (e.g. stable & main) will be used which will
-require Python 3.
-
-This means the [repo launcher] needs to support both Python 2 & Python 3, but
-since it doesn't import any other repo code, this shouldn't be too problematic.
-
-The main branch will require Python 3.6 at a minimum.
-If the system has an older version of Python 3, then users will have to select
-the legacy Python 2 branch instead.
-
-### repo hooks
 
 Projects that use [repo hooks] run on independent schedules.
-They might migrate to Python 3 earlier or later than us.
+Since it's not possible to detect what version of Python the hooks were written
-To support them, we'll probe the shebang of the hook script and if we find an
+or tested against, we always import & exec them with the active Python version.
-interpreter in there that indicates a different version than repo is currently
-running under, we'll attempt to reexec ourselves under that.
 
-For example, a hook with a header like `#!/usr/bin/python2` will have repo
+If the user's Python is too new for the [repo hooks], then it is up to the hooks
-execute `/usr/bin/python2` to execute the hook code specifically if repo is
+maintainer to update.
-currently running Python 3.
 
-For more details, consult the [repo hooks] documentation.
+## Repo launcher
+
+The [repo launcher] is an independent script that can support older versions of
+Python without holding back the rest of the codebase.
+If it detects the current version of Python is too old, it will try to reexec
+via a newer version of Python via standard `pythonX.Y` interpreter names.
+
+However, this is provided as a nicety when it is not onerous, and there is no
+official support for older versions of Python than the rest of the codebase.
+
+If your default python interpreters are too old to run the launcher even though
+you have newer versions installed, your choices are:
+
+*   Modify the [repo launcher]'s shebang to suite your environment.
+*   Download an older version of the [repo launcher] and don't upgrade it.
+    Be aware that there is no guarantee old repo launchers are WILL work with
+    current versions of repo.  Bug reports using old launchers will not be
+    accepted.
+
+## When to drop support
+
+So far, Python 3.6 has provided most of the interesting features that we want
+(e.g. typing & f-strings), and there haven't been features in newer versions
+that are critical to us.
+
+That said, let's assume we need functionality that only exists in Python 3.7.
+How do we decide when it's acceptable to drop Python 3.6?
+
+1.  Review the [Project References](./release-process.md#project-references) to
+    see what major distros are using the previous version of Python, and when
+    they go EOL.  Generally we care about Ubuntu LTS & current/previous Debian
+    stable versions.
+    *   If they're all EOL already, then go for it, drop support.
+    *   If they aren't EOL, start a thread on [repo-discuss] to see how the user
+        base feels about the proposal.
+1.  Update the "soft" versions in the codebase.  This will start warning users
+    that the older version is deprecated.
+    *   Update [repo](/repo) if the launcher needs updating.
+        This only helps with people who download newer launchers.
+    *   Update [main.py](/main.py) for the main codebase.
+        This warns for everyone regardless of [repo launcher] version.
+    *   Update [requirements.json](/requirements.json).
+        This allows [repo launcher] to display warnings/errors without having
+        to execute the new codebase.  This helps in case of syntax or module
+        changes where older versions won't even be able to import the new code.
+1.  After some grace period (ideally at least 2 quarters after the first release
+    with the updated soft requirements), update the "hard" versions, and then
+    start using the new functionality.
+
+## Python 2.7 & 3.0-3.5
+
+> **There is no support for these versions.**
+> **Do not file bugs if you are using old Python versions.**
+> **Any such reports will be marked invalid and ignored.**
+> **Upgrade your distro and/or runtime instead.**
+
+Fetch an old version of the [repo launcher]:
+
+```sh
+$ curl https://storage.googleapis.com/git-repo-downloads/repo-2.32 > ~/.bin/repo-2.32
+$ chmod a+rx ~/.bin/repo-2.32
+```
+
+Then initialize an old version of repo:
+
+```sh
+$ repo-2.32 init --repo-rev=repo-1 ...
+```
 
 
+[repo-discuss]: https://groups.google.com/forum/#!forum/repo-discuss
 [repo hooks]: ./repo-hooks.md
 [repo launcher]: ../repo

hooks.py

@@ -12,11 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import errno
-import json
 import os
 import re
-import subprocess
 import sys
 import traceback
 import urllib.parse
@@ -298,39 +295,6 @@ class RepoHook:
 
         return interp
 
-    def _ExecuteHookViaReexec(self, interp, context, **kwargs):
-        """Execute the hook script through |interp|.
-
-        Note: Support for this feature should be dropped ~Jun 2021.
-
-        Args:
-            interp: The Python program to run.
-            context: Basic Python context to execute the hook inside.
-            kwargs: Arbitrary arguments to pass to the hook script.
-
-        Raises:
-            HookError: When the hooks failed for any reason.
-        """
-        # This logic needs to be kept in sync with _ExecuteHookViaImport below.
-        script = f"""
-import json, os, sys
-path = '''{self._script_fullpath}'''
-kwargs = json.loads('''{json.dumps(kwargs)}''')
-context = json.loads('''{json.dumps(context)}''')
-sys.path.insert(0, os.path.dirname(path))
-data = open(path).read()
-exec(compile(data, path, 'exec'), context)
-context['main'](**kwargs)
-"""
-
-        # We pass the script via stdin to avoid OS argv limits.  It also makes
-        # unhandled exception tracebacks less verbose/confusing for users.
-        cmd = [interp, "-c", "import sys; exec(sys.stdin.read())"]
-        proc = subprocess.Popen(cmd, stdin=subprocess.PIPE)
-        proc.communicate(input=script.encode("utf-8"))
-        if proc.returncode:
-            raise HookError(f"Failed to run {self._hook_type} hook.")
-
     def _ExecuteHookViaImport(self, data, context, **kwargs):
         """Execute the hook code in |data| directly.
 
@@ -408,30 +372,13 @@ context['main'](**kwargs)
             # See what version of python the hook has been written against.
             data = open(self._script_fullpath).read()
             interp = self._ExtractInterpFromShebang(data)
-            reexec = False
             if interp:
                 prog = os.path.basename(interp)
-                if prog.startswith("python2") and sys.version_info.major != 2:
+                if prog.startswith("python2"):
-                    reexec = True
+                    raise HookError("Python 2 is not supported")
-                elif prog.startswith("python3") and sys.version_info.major == 2:
-                    reexec = True
-
-            # Attempt to execute the hooks through the requested version of
-            # Python.
-            if reexec:
-                try:
-                    self._ExecuteHookViaReexec(interp, context, **kwargs)
-                except OSError as e:
-                    if e.errno == errno.ENOENT:
-                        # We couldn't find the interpreter, so fallback to
-                        # importing.
-                        reexec = False
-                    else:
-                        raise
 
             # Run the hook by importing directly.
-            if not reexec:
+            self._ExecuteHookViaImport(data, context, **kwargs)
-                self._ExecuteHookViaImport(data, context, **kwargs)
         finally:
             # Restore sys.path and CWD.
             sys.path = orig_syspath

main.py

@@ -86,27 +86,19 @@ logger = RepoLogger(__file__)
 MIN_PYTHON_VERSION_SOFT = (3, 6)
 MIN_PYTHON_VERSION_HARD = (3, 6)
 
-if sys.version_info.major < 3:
+if sys.version_info < MIN_PYTHON_VERSION_HARD:
     logger.error(
-        "repo: error: Python 2 is no longer supported; "
+        "repo: error: Python version is too old; "
         "Please upgrade to Python %d.%d+.",
         *MIN_PYTHON_VERSION_SOFT,
     )
     sys.exit(1)
-else:
+elif sys.version_info < MIN_PYTHON_VERSION_SOFT:
-    if sys.version_info < MIN_PYTHON_VERSION_HARD:
+    logger.error(
-        logger.error(
+        "repo: warning: your Python version is no longer supported; "
-            "repo: error: Python 3 version is too old; "
+        "Please upgrade to Python %d.%d+.",
-            "Please upgrade to Python %d.%d+.",
+        *MIN_PYTHON_VERSION_SOFT,
-            *MIN_PYTHON_VERSION_SOFT,
+    )
-        )
-        sys.exit(1)
-    elif sys.version_info < MIN_PYTHON_VERSION_SOFT:
-        logger.error(
-            "repo: warning: your Python 3 version is no longer supported; "
-            "Please upgrade to Python %d.%d+.",
-            *MIN_PYTHON_VERSION_SOFT,
-        )
 
 KEYBOARD_INTERRUPT_EXIT = 128 + signal.SIGINT
 MAX_PRINT_ERRORS = 5
@@ -565,9 +557,11 @@ repo: error:
         sys.exit(1)
 
     if exp > ver:
-        logger.warn("\n... A new version of repo (%s) is available.", exp_str)
+        logger.warning(
+            "\n... A new version of repo (%s) is available.", exp_str
+        )
         if os.access(repo_path, os.W_OK):
-            logger.warn(
+            logger.warning(
                 """\
 ... You should upgrade soon:
     cp %s %s
@@ -576,7 +570,7 @@ repo: error:
                 repo_path,
             )
         else:
-            logger.warn(
+            logger.warning(
                 """\
 ... New version is available at: %s
 ... The launcher is run from: %s

project.py

@@ -1258,7 +1258,7 @@ class Project:
             try:
                 platform_utils.remove(tarpath)
             except OSError as e:
-                logger.warn("warn: Cannot remove archive %s: %s", tarpath, e)
+                logger.warning("warn: Cannot remove archive %s: %s", tarpath, e)
             self._CopyAndLinkFiles()
             return SyncNetworkHalfResult(True)
 
@@ -1755,7 +1755,7 @@ class Project:
         """
         if self.IsDirty():
             if force:
-                logger.warn(
+                logger.warning(
                     "warning: %s: Removing dirty project: uncommitted changes "
                     "lost.",
                     self.RelPath(local=False),
@@ -3027,7 +3027,7 @@ class Project:
                 # hardlink below.
                 if not filecmp.cmp(stock_hook, dst, shallow=False):
                     if not quiet:
-                        logger.warn(
+                        logger.warning(
                             "warn: %s: Not replacing locally modified %s hook",
                             self.RelPath(local=False),
                             name,
@@ -4325,7 +4325,7 @@ class ManifestProject(MetaProject):
             self.config.SetBoolean("repo.worktree", worktree)
             if is_new:
                 self.use_git_worktrees = True
-            logger.warn("warning: --worktree is experimental!")
+            logger.warning("warning: --worktree is experimental!")
 
         if archive:
             if is_new:
@@ -4389,7 +4389,7 @@ class ManifestProject(MetaProject):
 
             self.config.SetBoolean("repo.git-lfs", git_lfs)
             if not is_new:
-                logger.warn(
+                logger.warning(
                     "warning: Changing --git-lfs settings will only affect new "
                     "project checkouts.\n"
                     "         Existing projects will require manual updates.\n"
@@ -4501,7 +4501,7 @@ class ManifestProject(MetaProject):
                 submanifest = ""
                 if self.manifest.path_prefix:
                     submanifest = f"for {self.manifest.path_prefix} "
-                logger.warn(
+                logger.warning(
                     "warning: git update of superproject %s failed, "
                     "repo sync will not use superproject to fetch source; "
                     "while this error is not fatal, and you can continue to "

subcmds/cherry_pick.py

@@ -86,7 +86,7 @@ change id will be added.
             p.Wait()
         except GitError as e:
             logger.error(e)
-            logger.warn(
+            logger.warning(
                 "NOTE: When committing (please see above) and editing the "
                 "commit message, please remove the old Change-Id-line and "
                 "add:\n%s",

subcmds/manifest.py

@@ -136,7 +136,7 @@ to indicate the remote ref to push changes to via 'repo upload'.
             manifest.SetUseLocalManifests(not opt.ignore_local_manifests)
 
             if opt.json:
-                logger.warn("warning: --json is experimental!")
+                logger.warning("warning: --json is experimental!")
                 doc = manifest.ToDict(
                     peg_rev=opt.peg_rev,
                     peg_rev_upstream=opt.peg_rev_upstream,
@@ -163,13 +163,13 @@ to indicate the remote ref to push changes to via 'repo upload'.
             if output_file != "-":
                 fd.close()
                 if manifest.path_prefix:
-                    logger.warn(
+                    logger.warning(
                         "Saved %s submanifest to %s",
                         manifest.path_prefix,
                         output_file,
                     )
                 else:
-                    logger.warn("Saved manifest to %s", output_file)
+                    logger.warning("Saved manifest to %s", output_file)
 
     def ValidateOptions(self, opt, args):
         if args:

subcmds/rebase.py

@@ -113,7 +113,7 @@ branch but need to incorporate new upstream changes "underneath" them.
             )
 
             if len(args) == 1:
-                logger.warn(
+                logger.warning(
                     "note: project %s is mapped to more than one path", args[0]
                 )
 

subcmds/sync.py

@@ -1877,7 +1877,7 @@ def _PostRepoUpgrade(manifest, quiet=False):
 
 def _PostRepoFetch(rp, repo_verify=True, verbose=False):
     if rp.HasChanges:
-        logger.warn("info: A new version of repo is available")
+        logger.warning("info: A new version of repo is available")
         wrapper = Wrapper()
         try:
             rev = rp.bare_git.describe(rp.GetRevisionId())

subcmds/upload.py

@@ -72,16 +72,16 @@ def _VerifyPendingCommits(branches: List[ReviewableBranch]) -> bool:
     # If any branch has many commits, prompt the user.
     if many_commits:
         if len(branches) > 1:
-            logger.warn(
+            logger.warning(
                 "ATTENTION: One or more branches has an unusually high number "
                 "of commits."
             )
         else:
-            logger.warn(
+            logger.warning(
                 "ATTENTION: You are uploading an unusually high number of "
                 "commits."
             )
-        logger.warn(
+        logger.warning(
             "YOU PROBABLY DO NOT MEAN TO DO THIS. (Did you rebase across "
             "branches?)"
         )