mirror of
https://gerrit.googlesource.com/git-repo
synced 2024-12-21 07:16:21 +00:00
hooks: support external hooks running different Python version
As we convert repo to support Python 3, the version of Python that we use might not be the version that repo hooks users have written for. Since repo upgrades are not immediate, and not easily under direct control of end users (relative to the projects maintaining the hook code), allow hook authors to declare the version of Python that they want to use. Now repo will read the shebang from the hook script and compare it against the version of Python repo itself is running under. If they differ, we'll try to execute a separate instance of Python and have it load & execute the hook. If things are compatible, then we still use the inprocess execution logic that we have today. This allows repo hook users to upgrade on their own schedule (they could even upgrade to Python 3 ahead of us) without having to worry about their supported version being exactly in sync with repo's. Bug: https://crbug.com/gerrit/10418 Change-Id: I97c7c96b64fb2ee465c39b90e9bdcc76394a146a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228432 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
This commit is contained in:
parent
745be2ede1
commit
f7c51606f0
@ -28,5 +28,20 @@ The master branch will require Python 3.6 at a minimum.
|
|||||||
If the system has an older version of Python 3, then users will have to select
|
If the system has an older version of Python 3, then users will have to select
|
||||||
the legacy Python 2 branch instead.
|
the legacy Python 2 branch instead.
|
||||||
|
|
||||||
|
### repo hooks
|
||||||
|
|
||||||
|
Projects that use [repo hooks] run on independent schedules.
|
||||||
|
They might migrate to Python 3 earlier or later than us.
|
||||||
|
To support them, we'll probe the shebang of the hook script and if we find an
|
||||||
|
interpreter in there that indicates a different version than repo is currently
|
||||||
|
running under, we'll attempt to reexec ourselves under that.
|
||||||
|
|
||||||
|
For example, a hook with a header like `#!/usr/bin/python2` will have repo
|
||||||
|
execute `/usr/bin/python2` to execute the hook code specifically if repo is
|
||||||
|
currently running Python 3.
|
||||||
|
|
||||||
|
For more details, consult the [repo hooks] documentation.
|
||||||
|
|
||||||
|
|
||||||
|
[repo hooks]: ./repo-hooks.md
|
||||||
[repo launcher]: ../repo
|
[repo launcher]: ../repo
|
||||||
|
@ -83,6 +83,31 @@ then check it directly. Hooks should not normally modify the active git repo
|
|||||||
the user. Although user interaction is discouraged in the common case, it can
|
the user. Although user interaction is discouraged in the common case, it can
|
||||||
be useful when deploying automatic fixes.
|
be useful when deploying automatic fixes.
|
||||||
|
|
||||||
|
### Shebang Handling
|
||||||
|
|
||||||
|
*** note
|
||||||
|
This is intended as a transitional feature. Hooks are expected to eventually
|
||||||
|
migrate to Python 3 only as Python 2 is EOL & deprecated.
|
||||||
|
***
|
||||||
|
|
||||||
|
If the hook is written against a specific version of Python (either 2 or 3),
|
||||||
|
the script can declare that explicitly. Repo will then attempt to execute it
|
||||||
|
under the right version of Python regardless of the version repo itself might
|
||||||
|
be executing under.
|
||||||
|
|
||||||
|
Here are the shebangs that are recognized.
|
||||||
|
|
||||||
|
* `#!/usr/bin/env python` & `#!/usr/bin/python`: The hook is compatible with
|
||||||
|
Python 2 & Python 3. For maximum compatibility, these are recommended.
|
||||||
|
* `#!/usr/bin/env python2` & `#!/usr/bin/python2`: The hook requires Python 2.
|
||||||
|
Version specific names like `python2.7` are also recognized.
|
||||||
|
* `#!/usr/bin/env python3` & `#!/usr/bin/python3`: The hook requires Python 3.
|
||||||
|
Version specific names like `python3.6` are also recognized.
|
||||||
|
|
||||||
|
If no shebang is detected, or does not match the forms above, we assume that the
|
||||||
|
hook is compatible with both Python 2 & Python 3 as if `#!/usr/bin/python` was
|
||||||
|
used.
|
||||||
|
|
||||||
## Hooks
|
## Hooks
|
||||||
|
|
||||||
Here are all the points available for hooking.
|
Here are all the points available for hooking.
|
||||||
|
145
project.py
145
project.py
@ -18,6 +18,7 @@ from __future__ import print_function
|
|||||||
import errno
|
import errno
|
||||||
import filecmp
|
import filecmp
|
||||||
import glob
|
import glob
|
||||||
|
import json
|
||||||
import os
|
import os
|
||||||
import random
|
import random
|
||||||
import re
|
import re
|
||||||
@ -544,6 +545,105 @@ class RepoHook(object):
|
|||||||
prompt % (self._GetMustVerb(), self._script_fullpath),
|
prompt % (self._GetMustVerb(), self._script_fullpath),
|
||||||
'Scripts have changed since %s was allowed.' % (self._hook_type,))
|
'Scripts have changed since %s was allowed.' % (self._hook_type,))
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _ExtractInterpFromShebang(data):
|
||||||
|
"""Extract the interpreter used in the shebang.
|
||||||
|
|
||||||
|
Try to locate the interpreter the script is using (ignoring `env`).
|
||||||
|
|
||||||
|
Args:
|
||||||
|
data: The file content of the script.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The basename of the main script interpreter, or None if a shebang is not
|
||||||
|
used or could not be parsed out.
|
||||||
|
"""
|
||||||
|
firstline = data.splitlines()[:1]
|
||||||
|
if not firstline:
|
||||||
|
return None
|
||||||
|
|
||||||
|
# The format here can be tricky.
|
||||||
|
shebang = firstline[0].strip()
|
||||||
|
m = re.match(r'^#!\s*([^\s]+)(?:\s+([^\s]+))?', shebang)
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
|
||||||
|
# If the using `env`, find the target program.
|
||||||
|
interp = m.group(1)
|
||||||
|
if os.path.basename(interp) == 'env':
|
||||||
|
interp = m.group(2)
|
||||||
|
|
||||||
|
return interp
|
||||||
|
|
||||||
|
def _ExecuteHookViaReexec(self, interp, context, **kwargs):
|
||||||
|
"""Execute the hook script through |interp|.
|
||||||
|
|
||||||
|
Note: Support for this feature should be dropped ~Jun 2021.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
interp: The Python program to run.
|
||||||
|
context: Basic Python context to execute the hook inside.
|
||||||
|
kwargs: Arbitrary arguments to pass to the hook script.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
HookError: When the hooks failed for any reason.
|
||||||
|
"""
|
||||||
|
# This logic needs to be kept in sync with _ExecuteHookViaImport below.
|
||||||
|
script = """
|
||||||
|
import json, os, sys
|
||||||
|
path = '''%(path)s'''
|
||||||
|
kwargs = json.loads('''%(kwargs)s''')
|
||||||
|
context = json.loads('''%(context)s''')
|
||||||
|
sys.path.insert(0, os.path.dirname(path))
|
||||||
|
data = open(path).read()
|
||||||
|
exec(compile(data, path, 'exec'), context)
|
||||||
|
context['main'](**kwargs)
|
||||||
|
""" % {
|
||||||
|
'path': self._script_fullpath,
|
||||||
|
'kwargs': json.dumps(kwargs),
|
||||||
|
'context': json.dumps(context),
|
||||||
|
}
|
||||||
|
|
||||||
|
# We pass the script via stdin to avoid OS argv limits. It also makes
|
||||||
|
# unhandled exception tracebacks less verbose/confusing for users.
|
||||||
|
cmd = [interp, '-c', 'import sys; exec(sys.stdin.read())']
|
||||||
|
proc = subprocess.Popen(cmd, stdin=subprocess.PIPE)
|
||||||
|
proc.communicate(input=script.encode('utf-8'))
|
||||||
|
if proc.returncode:
|
||||||
|
raise HookError('Failed to run %s hook.' % (self._hook_type,))
|
||||||
|
|
||||||
|
def _ExecuteHookViaImport(self, data, context, **kwargs):
|
||||||
|
"""Execute the hook code in |data| directly.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
data: The code of the hook to execute.
|
||||||
|
context: Basic Python context to execute the hook inside.
|
||||||
|
kwargs: Arbitrary arguments to pass to the hook script.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
HookError: When the hooks failed for any reason.
|
||||||
|
"""
|
||||||
|
# Exec, storing global context in the context dict. We catch exceptions
|
||||||
|
# and convert to a HookError w/ just the failing traceback.
|
||||||
|
try:
|
||||||
|
exec(compile(data, self._script_fullpath, 'exec'), context)
|
||||||
|
except Exception:
|
||||||
|
raise HookError('%s\nFailed to import %s hook; see traceback above.' %
|
||||||
|
(traceback.format_exc(), self._hook_type))
|
||||||
|
|
||||||
|
# Running the script should have defined a main() function.
|
||||||
|
if 'main' not in context:
|
||||||
|
raise HookError('Missing main() in: "%s"' % self._script_fullpath)
|
||||||
|
|
||||||
|
# Call the main function in the hook. If the hook should cause the
|
||||||
|
# build to fail, it will raise an Exception. We'll catch that convert
|
||||||
|
# to a HookError w/ just the failing traceback.
|
||||||
|
try:
|
||||||
|
context['main'](**kwargs)
|
||||||
|
except Exception:
|
||||||
|
raise HookError('%s\nFailed to run main() for %s hook; see traceback '
|
||||||
|
'above.' % (traceback.format_exc(), self._hook_type))
|
||||||
|
|
||||||
def _ExecuteHook(self, **kwargs):
|
def _ExecuteHook(self, **kwargs):
|
||||||
"""Actually execute the given hook.
|
"""Actually execute the given hook.
|
||||||
|
|
||||||
@ -568,19 +668,8 @@ class RepoHook(object):
|
|||||||
# hooks can't import repo files.
|
# hooks can't import repo files.
|
||||||
sys.path = [os.path.dirname(self._script_fullpath)] + sys.path[1:]
|
sys.path = [os.path.dirname(self._script_fullpath)] + sys.path[1:]
|
||||||
|
|
||||||
# Exec, storing global context in the context dict. We catch exceptions
|
# Initial global context for the hook to run within.
|
||||||
# and convert to a HookError w/ just the failing traceback.
|
|
||||||
context = {'__file__': self._script_fullpath}
|
context = {'__file__': self._script_fullpath}
|
||||||
try:
|
|
||||||
exec(compile(open(self._script_fullpath).read(),
|
|
||||||
self._script_fullpath, 'exec'), context)
|
|
||||||
except Exception:
|
|
||||||
raise HookError('%s\nFailed to import %s hook; see traceback above.' %
|
|
||||||
(traceback.format_exc(), self._hook_type))
|
|
||||||
|
|
||||||
# Running the script should have defined a main() function.
|
|
||||||
if 'main' not in context:
|
|
||||||
raise HookError('Missing main() in: "%s"' % self._script_fullpath)
|
|
||||||
|
|
||||||
# Add 'hook_should_take_kwargs' to the arguments to be passed to main.
|
# Add 'hook_should_take_kwargs' to the arguments to be passed to main.
|
||||||
# We don't actually want hooks to define their main with this argument--
|
# We don't actually want hooks to define their main with this argument--
|
||||||
@ -592,15 +681,31 @@ class RepoHook(object):
|
|||||||
kwargs = kwargs.copy()
|
kwargs = kwargs.copy()
|
||||||
kwargs['hook_should_take_kwargs'] = True
|
kwargs['hook_should_take_kwargs'] = True
|
||||||
|
|
||||||
# Call the main function in the hook. If the hook should cause the
|
# See what version of python the hook has been written against.
|
||||||
# build to fail, it will raise an Exception. We'll catch that convert
|
data = open(self._script_fullpath).read()
|
||||||
# to a HookError w/ just the failing traceback.
|
interp = self._ExtractInterpFromShebang(data)
|
||||||
|
reexec = False
|
||||||
|
if interp:
|
||||||
|
prog = os.path.basename(interp)
|
||||||
|
if prog.startswith('python2') and sys.version_info.major != 2:
|
||||||
|
reexec = True
|
||||||
|
elif prog.startswith('python3') and sys.version_info.major == 2:
|
||||||
|
reexec = True
|
||||||
|
|
||||||
|
# Attempt to execute the hooks through the requested version of Python.
|
||||||
|
if reexec:
|
||||||
try:
|
try:
|
||||||
context['main'](**kwargs)
|
self._ExecuteHookViaReexec(interp, context, **kwargs)
|
||||||
except Exception:
|
except OSError as e:
|
||||||
raise HookError('%s\nFailed to run main() for %s hook; see traceback '
|
if e.errno == errno.ENOENT:
|
||||||
'above.' % (traceback.format_exc(),
|
# We couldn't find the interpreter, so fallback to importing.
|
||||||
self._hook_type))
|
reexec = False
|
||||||
|
else:
|
||||||
|
raise
|
||||||
|
|
||||||
|
# Run the hook by importing directly.
|
||||||
|
if not reexec:
|
||||||
|
self._ExecuteHookViaImport(data, context, **kwargs)
|
||||||
finally:
|
finally:
|
||||||
# Restore sys.path and CWD.
|
# Restore sys.path and CWD.
|
||||||
sys.path = orig_syspath
|
sys.path = orig_syspath
|
||||||
|
58
tests/test_project.py
Normal file
58
tests/test_project.py
Normal file
@ -0,0 +1,58 @@
|
|||||||
|
# -*- coding:utf-8 -*-
|
||||||
|
#
|
||||||
|
# Copyright (C) 2019 The Android Open Source Project
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
import project
|
||||||
|
|
||||||
|
|
||||||
|
class RepoHookShebang(unittest.TestCase):
|
||||||
|
"""Check shebang parsing in RepoHook."""
|
||||||
|
|
||||||
|
def test_no_shebang(self):
|
||||||
|
"""Lines w/out shebangs should be rejected."""
|
||||||
|
DATA = (
|
||||||
|
'',
|
||||||
|
'# -*- coding:utf-8 -*-\n',
|
||||||
|
'#\n# foo\n',
|
||||||
|
'# Bad shebang in script\n#!/foo\n'
|
||||||
|
)
|
||||||
|
for data in DATA:
|
||||||
|
self.assertIsNone(project.RepoHook._ExtractInterpFromShebang(data))
|
||||||
|
|
||||||
|
def test_direct_interp(self):
|
||||||
|
"""Lines whose shebang points directly to the interpreter."""
|
||||||
|
DATA = (
|
||||||
|
('#!/foo', '/foo'),
|
||||||
|
('#! /foo', '/foo'),
|
||||||
|
('#!/bin/foo ', '/bin/foo'),
|
||||||
|
('#! /usr/foo ', '/usr/foo'),
|
||||||
|
('#! /usr/foo -args', '/usr/foo'),
|
||||||
|
)
|
||||||
|
for shebang, interp in DATA:
|
||||||
|
self.assertEqual(project.RepoHook._ExtractInterpFromShebang(shebang),
|
||||||
|
interp)
|
||||||
|
|
||||||
|
def test_env_interp(self):
|
||||||
|
"""Lines whose shebang launches through `env`."""
|
||||||
|
DATA = (
|
||||||
|
('#!/usr/bin/env foo', 'foo'),
|
||||||
|
('#!/bin/env foo', 'foo'),
|
||||||
|
('#! /bin/env /bin/foo ', '/bin/foo'),
|
||||||
|
)
|
||||||
|
for shebang, interp in DATA:
|
||||||
|
self.assertEqual(project.RepoHook._ExtractInterpFromShebang(shebang),
|
||||||
|
interp)
|
Loading…
Reference in New Issue
Block a user