manifest: add support for groups in include

Attrib groups can now be added to manifest include, thus all projects in an included manifest file can easily be tagged with a group without modifying all projects in that manifest file. Include groups will add and recurse, meaning included manifest projects will carry all parent includes. Intentionally, no support added for group remove, to keep complexity down. Group handling for projects is untouched, meaning a group set on a project will still append to whatever was or was not inherited in parent manifest includes, resulting in union of groups inherited and set for the project itself. Test: manual multi-level manifest include structure, in serial and parallel, with different groups set on init Test: added unit tests to cover the inheritance Change-Id: Id2229aa6fd78d355ba598cc15c701b2ee71e5c6f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/283587 Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com> Reviewed-by: Mike Frysinger <vapier@google.com>
Concentrate the RepoHook knowledge in the RepoHook class
2025-06-26 20:17:52 +00:00 · 2020-11-26 09:13:14 +00:00 · 2020-11-23 09:59:16 +00:00 · 2020-11-23 09:17:32 +00:00 · 2020-11-20 20:53:43 +00:00 · 2020-11-20 04:08:19 +00:00
28 changed files with 1148 additions and 668 deletions
--- a/.github/workflows/test-ci.yml
+++ b/.github/workflows/test-ci.yml
@ -5,7 +5,7 @@ name: Test CI

 on:
  push:
-    branches: [master, repo-1, stable, maint]
+    branches: [main, repo-1, stable, maint]
    tags: [v*]

 jobs:
@ -14,10 +14,7 @@ jobs:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: [2.7, 3.6, 3.7, 3.8]
-        exclude:
-          - os: windows-latest
-            python-version: 2.7
+        python-version: [3.6, 3.7, 3.8]
    runs-on: ${{ matrix.os }}

    steps:
--- a/SUBMITTING_PATCHES.md
+++ b/SUBMITTING_PATCHES.md
@ -10,7 +10,7 @@
 - Make corrections if requested.
 - Verify your changes on gerrit so they can be submitted.

-   `git push https://gerrit-review.googlesource.com/git-repo HEAD:refs/for/master`
+   `git push https://gerrit-review.googlesource.com/git-repo HEAD:refs/for/main`


 # Long Version
@ -150,7 +150,7 @@ Push your patches over HTTPS to the review server, possibly through
 a remembered remote to make this easier in the future:

    git config remote.review.url https://gerrit-review.googlesource.com/git-repo
-    git config remote.review.push HEAD:refs/for/master
+    git config remote.review.push HEAD:refs/for/main

    git push review

--- a/docs/internal-fs-layout.md
+++ b/docs/internal-fs-layout.md
@ -106,7 +106,7 @@ support, see the [manifest-format.md] file.
    setting in the manifest (i.e. the path on the remote server) with a `.git`
    suffix.  This allows for multiple checkouts of the same remote git repo to
    share their objects.  For example, you could have different branches of
-    `foo/bar.git` checked out to `foo/bar-master`, `foo/bar-release`, etc...
+    `foo/bar.git` checked out to `foo/bar-main`, `foo/bar-release`, etc...
    There will be multiple trees under `projects/` for each one, but only one
    under `project-objects/`.

--- a/docs/manifest-format.md
+++ b/docs/manifest-format.md
@ -99,7 +99,8 @@ following DTD:
  <!ATTLIST repo-hooks enabled-list CDATA #REQUIRED>

  <!ELEMENT include EMPTY>
-  <!ATTLIST include name CDATA #REQUIRED>
+  <!ATTLIST include name   CDATA #REQUIRED>
+  <!ATTLIST include groups CDATA #IMPLIED>
 ]>
 ```

@ -142,8 +143,8 @@ Attribute `review`: Hostname of the Gerrit server where reviews
 are uploaded to by `repo upload`.  This attribute is optional;
 if not specified then `repo upload` will not function.

-Attribute `revision`: Name of a Git branch (e.g. `master` or
-`refs/heads/master`). Remotes with their own revision will override
+Attribute `revision`: Name of a Git branch (e.g. `main` or
+`refs/heads/main`). Remotes with their own revision will override
 the default revision.

 ### Element default
@ -156,11 +157,11 @@ Attribute `remote`: Name of a previously defined remote element.
 Project elements lacking a remote attribute of their own will use
 this remote.

-Attribute `revision`: Name of a Git branch (e.g. `master` or
-`refs/heads/master`).  Project elements lacking their own
+Attribute `revision`: Name of a Git branch (e.g. `main` or
+`refs/heads/main`).  Project elements lacking their own
 revision attribute will use this revision.

-Attribute `dest-branch`: Name of a Git branch (e.g. `master`).
+Attribute `dest-branch`: Name of a Git branch (e.g. `main`).
 Project elements not setting their own `dest-branch` will inherit
 this value. If this value is not set, projects will use `revision`
 by default instead.
@ -247,13 +248,13 @@ If not supplied the remote given by the default element is used.

 Attribute `revision`: Name of the Git branch the manifest wants
 to track for this project.  Names can be relative to refs/heads
-(e.g. just "master") or absolute (e.g. "refs/heads/master").
+(e.g. just "main") or absolute (e.g. "refs/heads/main").
 Tags and/or explicit SHA-1s should work in theory, but have not
 been extensively tested.  If not supplied the revision given by
 the remote element is used if applicable, else the default
 element is used.

-Attribute `dest-branch`: Name of a Git branch (e.g. `master`).
+Attribute `dest-branch`: Name of a Git branch (e.g. `main`).
 When using `repo upload`, changes will be submitted for code
 review on this branch. If unspecified both here and in the
 default element, `revision` is used instead.
@ -368,6 +369,10 @@ target manifest to include - it must be a usable manifest on its own.
 Attribute `name`: the manifest to include, specified relative to
 the manifest repository's root.

+Attribute `groups`: List of additional groups to which all projects
+in the included manifest belong. This appends and recurses, meaning
+all projects in sub-manifests carry all parent include groups.
+Same syntax as the corresponding element of `project`.

 ## Local Manifests

@ -396,10 +401,4 @@ these extra projects.
 Manifest files stored in `$TOP_DIR/.repo/local_manifests/*.xml` will
 be loaded in alphabetical order.

-Additional remotes and projects may also be added through a local
-manifest, stored in `$TOP_DIR/.repo/local_manifest.xml`. This method
-is deprecated in favor of using multiple manifest files as mentioned
-above.
-
-If `$TOP_DIR/.repo/local_manifest.xml` exists, it will be loaded before
-any manifest files stored in `$TOP_DIR/.repo/local_manifests/*.xml`.
+The legacy `$TOP_DIR/.repo/local_manifest.xml` path is no longer supported.
--- a/docs/python-support.md
+++ b/docs/python-support.md
@ -18,13 +18,13 @@ Bugfixes may be added on a best-effort basis or from the community, but largely
 no new features will be added, nor is support guaranteed.

 Users can select this during `repo init` time via the [repo launcher].
-Otherwise the default branches (e.g. stable & master) will be used which will
+Otherwise the default branches (e.g. stable & main) will be used which will
 require Python 3.

 This means the [repo launcher] needs to support both Python 2 & Python 3, but
 since it doesn't import any other repo code, this shouldn't be too problematic.

-The master branch will require Python 3.6 at a minimum.
+The main branch will require Python 3.6 at a minimum.
 If the system has an older version of Python 3, then users will have to select
 the legacy Python 2 branch instead.

--- a/docs/release-process.md
+++ b/docs/release-process.md
@ -5,6 +5,37 @@ related topics and flows.

 [TOC]

+## Schedule
+
+There is no specific schedule for when releases are made.
+Usually it's more along the lines of "enough minor changes have been merged",
+or "there's a known issue the maintainers know should get fixed".
+If you find a fix has been merged for an issue important to you, but hasn't been
+released after a week or so, feel free to [contact] us to request a new release.
+
+### Release Freezes {#freeze}
+
+We try to observe a regular schedule for when **not** to release.
+If something goes wrong, staff need to be active in order to respond quickly &
+effectively.
+We also don't want to disrupt non-Google organizations if possible.
+
+We generally follow the rules:
+
+* Release during Mon - Thu, 9:00 - 14:00 [US PT]
+* Avoid holidays
+  * All regular [US holidays]
+  * Large international ones if possible
+  * All the various [New Years]
+    * Jan 1 in Gregorian calendar is the most obvious
+    * Check for large Lunar New Years too
+* Follow the normal [Google production freeze schedule]
+
+[US holidays]: https://en.wikipedia.org/wiki/Federal_holidays_in_the_United_States
+[US PT]: https://en.wikipedia.org/wiki/Pacific_Time_Zone
+[New Years]: https://en.wikipedia.org/wiki/New_Year
+[Google production freeze schedule]: http://goto.google.com/prod-freeze
+
 ## Launcher script

 The main repo script serves as a standalone program and is often referred to as
@ -66,7 +97,7 @@ If that tag cannot be verified, it gives up and forces the user to resolve.

 ## Branch management

-All development happens on the `master` branch and should generally be stable.
+All development happens on the `main` branch and should generally be stable.

 Since the repo launcher defaults to tracking the `stable` branch, it is not
 normally updated until a new release is available.
@ -81,7 +112,7 @@ For example, when `stable` moves from `v1.10.x` to `v1.11.x`, then the `maint`
 branch will be updated from `v1.9.x` to `v1.10.x`.

 We don't have parallel release branches/series.
-Typically all tags are made against the `master` branch and then pushed to the
+Typically all tags are made against the `main` branch and then pushed to the
 `stable` branch to make it available to the rest of the world.
 Since repo doesn't typically see a lot of changes, this tends to be OK.

@ -89,10 +120,10 @@ Since repo doesn't typically see a lot of changes, this tends to be OK.

 When you want to create a new release, you'll need to select a good version and
 create a signed tag using a key registered in repo itself.
-Typically we just tag the latest version of the `master` branch.
+Typically we just tag the latest version of the `main` branch.
 The tag could be pushed now, but it won't be used by clients normally (since the
 default `repo-rev` setting is `stable`).
-This would allow some early testing on systems who explicitly select `master`.
+This would allow some early testing on systems who explicitly select `main`.

 ### Creating a signed tag

@ -113,7 +144,7 @@ $ export GNUPGHOME=~/.gnupg/repo/
 $ gpg -K

 # Pick whatever branch or commit you want to tag.
-$ r=master
+$ r=main

 # Pick the new version.
 $ t=1.12.10
@ -242,6 +273,7 @@ Things in italics are things we used to care about but probably don't anymore.
 | Apr 2020 | **Apr 2030** |              |                 | **20.04 Focal**      | 2.25.0   | 2.7.17 3.7.5 |


+[contact]: ../README.md#contact
 [rel-d]: https://en.wikipedia.org/wiki/Debian_version_history
 [rel-g]: https://en.wikipedia.org/wiki/Git#Releases
 [rel-p]: https://en.wikipedia.org/wiki/History_of_Python#Table_of_versions
--- a/docs/repo-hooks.md
+++ b/docs/repo-hooks.md
@ -27,7 +27,7 @@ repohooks project is updated and a hook is triggered.
 For the full syntax, see the [repo manifest format](./manifest-format.md).

 Here's a short example from
-[Android](https://android.googlesource.com/platform/manifest/+/master/default.xml).
+[Android](https://android.googlesource.com/platform/manifest/+/HEAD/default.xml).
 The `<project>` line checks out the repohooks git repo to the local
 `tools/repohooks/` path.  The `<repo-hooks>` line says to look in the project
 with the name `platform/tools/repohooks` for hooks to run during the
--- a/hooks.py
+++ b/hooks.py
@ -0,0 +1,520 @@
+# -*- coding:utf-8 -*-
+#
+# Copyright (C) 2008 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import errno
+import json
+import os
+import re
+import subprocess
+import sys
+import traceback
+
+from error import HookError
+from git_refs import HEAD
+
+from pyversion import is_python3
+if is_python3():
+  import urllib.parse
+else:
+  import imp
+  import urlparse
+  urllib = imp.new_module('urllib')
+  urllib.parse = urlparse
+  input = raw_input  # noqa: F821
+
+
+class RepoHook(object):
+  """A RepoHook contains information about a script to run as a hook.
+
+  Hooks are used to run a python script before running an upload (for instance,
+  to run presubmit checks).  Eventually, we may have hooks for other actions.
+
+  This shouldn't be confused with files in the 'repo/hooks' directory.  Those
+  files are copied into each '.git/hooks' folder for each project.  Repo-level
+  hooks are associated instead with repo actions.
+
+  Hooks are always python.  When a hook is run, we will load the hook into the
+  interpreter and execute its main() function.
+
+  Combinations of hook option flags:
+  - no-verify=False, verify=False (DEFAULT):
+    If stdout is a tty, can prompt about running hooks if needed.
+    If user denies running hooks, the action is cancelled. If stdout is
+    not a tty and we would need to prompt about hooks, action is
+    cancelled.
+  - no-verify=False, verify=True:
+    Always run hooks with no prompt.
+  - no-verify=True, verify=False:
+    Never run hooks, but run action anyway (AKA bypass hooks).
+  - no-verify=True, verify=True:
+    Invalid
+  """
+
+  def __init__(self,
+               hook_type,
+               hooks_project,
+               repo_topdir,
+               manifest_url,
+               bypass_hooks=False,
+               allow_all_hooks=False,
+               ignore_hooks=False,
+               abort_if_user_denies=False):
+    """RepoHook constructor.
+
+    Params:
+      hook_type: A string representing the type of hook.  This is also used
+          to figure out the name of the file containing the hook.  For
+          example: 'pre-upload'.
+      hooks_project: The project containing the repo hooks.
+          If you have a manifest, this is manifest.repo_hooks_project.
+          OK if this is None, which will make the hook a no-op.
+      repo_topdir: The top directory of the repo client checkout.
+          This is the one containing the .repo directory. Scripts will
+          run with CWD as this directory.
+          If you have a manifest, this is manifest.topdir.
+      manifest_url: The URL to the manifest git repo.
+      bypass_hooks: If True, then 'Do not run the hook'.
+      allow_all_hooks: If True, then 'Run the hook without prompting'.
+      ignore_hooks: If True, then 'Do not abort action if hooks fail'.
+      abort_if_user_denies: If True, we'll abort running the hook if the user
+          doesn't allow us to run the hook.
+    """
+    self._hook_type = hook_type
+    self._hooks_project = hooks_project
+    self._repo_topdir = repo_topdir
+    self._manifest_url = manifest_url
+    self._bypass_hooks = bypass_hooks
+    self._allow_all_hooks = allow_all_hooks
+    self._ignore_hooks = ignore_hooks
+    self._abort_if_user_denies = abort_if_user_denies
+
+    # Store the full path to the script for convenience.
+    if self._hooks_project:
+      self._script_fullpath = os.path.join(self._hooks_project.worktree,
+                                           self._hook_type + '.py')
+    else:
+      self._script_fullpath = None
+
+  def _GetHash(self):
+    """Return a hash of the contents of the hooks directory.
+
+    We'll just use git to do this.  This hash has the property that if anything
+    changes in the directory we will return a different has.
+
+    SECURITY CONSIDERATION:
+      This hash only represents the contents of files in the hook directory, not
+      any other files imported or called by hooks.  Changes to imported files
+      can change the script behavior without affecting the hash.
+
+    Returns:
+      A string representing the hash.  This will always be ASCII so that it can
+      be printed to the user easily.
+    """
+    assert self._hooks_project, "Must have hooks to calculate their hash."
+
+    # We will use the work_git object rather than just calling GetRevisionId().
+    # That gives us a hash of the latest checked in version of the files that
+    # the user will actually be executing.  Specifically, GetRevisionId()
+    # doesn't appear to change even if a user checks out a different version
+    # of the hooks repo (via git checkout) nor if a user commits their own revs.
+    #
+    # NOTE: Local (non-committed) changes will not be factored into this hash.
+    # I think this is OK, since we're really only worried about warning the user
+    # about upstream changes.
+    return self._hooks_project.work_git.rev_parse(HEAD)
+
+  def _GetMustVerb(self):
+    """Return 'must' if the hook is required; 'should' if not."""
+    if self._abort_if_user_denies:
+      return 'must'
+    else:
+      return 'should'
+
+  def _CheckForHookApproval(self):
+    """Check to see whether this hook has been approved.
+
+    We'll accept approval of manifest URLs if they're using secure transports.
+    This way the user can say they trust the manifest hoster.  For insecure
+    hosts, we fall back to checking the hash of the hooks repo.
+
+    Note that we ask permission for each individual hook even though we use
+    the hash of all hooks when detecting changes.  We'd like the user to be
+    able to approve / deny each hook individually.  We only use the hash of all
+    hooks because there is no other easy way to detect changes to local imports.
+
+    Returns:
+      True if this hook is approved to run; False otherwise.
+
+    Raises:
+      HookError: Raised if the user doesn't approve and abort_if_user_denies
+          was passed to the consturctor.
+    """
+    if self._ManifestUrlHasSecureScheme():
+      return self._CheckForHookApprovalManifest()
+    else:
+      return self._CheckForHookApprovalHash()
+
+  def _CheckForHookApprovalHelper(self, subkey, new_val, main_prompt,
+                                  changed_prompt):
+    """Check for approval for a particular attribute and hook.
+
+    Args:
+      subkey: The git config key under [repo.hooks.<hook_type>] to store the
+          last approved string.
+      new_val: The new value to compare against the last approved one.
+      main_prompt: Message to display to the user to ask for approval.
+      changed_prompt: Message explaining why we're re-asking for approval.
+
+    Returns:
+      True if this hook is approved to run; False otherwise.
+
+    Raises:
+      HookError: Raised if the user doesn't approve and abort_if_user_denies
+          was passed to the consturctor.
+    """
+    hooks_config = self._hooks_project.config
+    git_approval_key = 'repo.hooks.%s.%s' % (self._hook_type, subkey)
+
+    # Get the last value that the user approved for this hook; may be None.
+    old_val = hooks_config.GetString(git_approval_key)
+
+    if old_val is not None:
+      # User previously approved hook and asked not to be prompted again.
+      if new_val == old_val:
+        # Approval matched.  We're done.
+        return True
+      else:
+        # Give the user a reason why we're prompting, since they last told
+        # us to "never ask again".
+        prompt = 'WARNING: %s\n\n' % (changed_prompt,)
+    else:
+      prompt = ''
+
+    # Prompt the user if we're not on a tty; on a tty we'll assume "no".
+    if sys.stdout.isatty():
+      prompt += main_prompt + ' (yes/always/NO)? '
+      response = input(prompt).lower()
+      print()
+
+      # User is doing a one-time approval.
+      if response in ('y', 'yes'):
+        return True
+      elif response == 'always':
+        hooks_config.SetString(git_approval_key, new_val)
+        return True
+
+    # For anything else, we'll assume no approval.
+    if self._abort_if_user_denies:
+      raise HookError('You must allow the %s hook or use --no-verify.' %
+                      self._hook_type)
+
+    return False
+
+  def _ManifestUrlHasSecureScheme(self):
+    """Check if the URI for the manifest is a secure transport."""
+    secure_schemes = ('file', 'https', 'ssh', 'persistent-https', 'sso', 'rpc')
+    parse_results = urllib.parse.urlparse(self._manifest_url)
+    return parse_results.scheme in secure_schemes
+
+  def _CheckForHookApprovalManifest(self):
+    """Check whether the user has approved this manifest host.
+
+    Returns:
+      True if this hook is approved to run; False otherwise.
+    """
+    return self._CheckForHookApprovalHelper(
+        'approvedmanifest',
+        self._manifest_url,
+        'Run hook scripts from %s' % (self._manifest_url,),
+        'Manifest URL has changed since %s was allowed.' % (self._hook_type,))
+
+  def _CheckForHookApprovalHash(self):
+    """Check whether the user has approved the hooks repo.
+
+    Returns:
+      True if this hook is approved to run; False otherwise.
+    """
+    prompt = ('Repo %s run the script:\n'
+              '  %s\n'
+              '\n'
+              'Do you want to allow this script to run')
+    return self._CheckForHookApprovalHelper(
+        'approvedhash',
+        self._GetHash(),
+        prompt % (self._GetMustVerb(), self._script_fullpath),
+        'Scripts have changed since %s was allowed.' % (self._hook_type,))
+
+  @staticmethod
+  def _ExtractInterpFromShebang(data):
+    """Extract the interpreter used in the shebang.
+
+    Try to locate the interpreter the script is using (ignoring `env`).
+
+    Args:
+      data: The file content of the script.
+
+    Returns:
+      The basename of the main script interpreter, or None if a shebang is not
+      used or could not be parsed out.
+    """
+    firstline = data.splitlines()[:1]
+    if not firstline:
+      return None
+
+    # The format here can be tricky.
+    shebang = firstline[0].strip()
+    m = re.match(r'^#!\s*([^\s]+)(?:\s+([^\s]+))?', shebang)
+    if not m:
+      return None
+
+    # If the using `env`, find the target program.
+    interp = m.group(1)
+    if os.path.basename(interp) == 'env':
+      interp = m.group(2)
+
+    return interp
+
+  def _ExecuteHookViaReexec(self, interp, context, **kwargs):
+    """Execute the hook script through |interp|.
+
+    Note: Support for this feature should be dropped ~Jun 2021.
+
+    Args:
+      interp: The Python program to run.
+      context: Basic Python context to execute the hook inside.
+      kwargs: Arbitrary arguments to pass to the hook script.
+
+    Raises:
+      HookError: When the hooks failed for any reason.
+    """
+    # This logic needs to be kept in sync with _ExecuteHookViaImport below.
+    script = """
+import json, os, sys
+path = '''%(path)s'''
+kwargs = json.loads('''%(kwargs)s''')
+context = json.loads('''%(context)s''')
+sys.path.insert(0, os.path.dirname(path))
+data = open(path).read()
+exec(compile(data, path, 'exec'), context)
+context['main'](**kwargs)
+""" % {
+        'path': self._script_fullpath,
+        'kwargs': json.dumps(kwargs),
+        'context': json.dumps(context),
+    }
+
+    # We pass the script via stdin to avoid OS argv limits.  It also makes
+    # unhandled exception tracebacks less verbose/confusing for users.
+    cmd = [interp, '-c', 'import sys; exec(sys.stdin.read())']
+    proc = subprocess.Popen(cmd, stdin=subprocess.PIPE)
+    proc.communicate(input=script.encode('utf-8'))
+    if proc.returncode:
+      raise HookError('Failed to run %s hook.' % (self._hook_type,))
+
+  def _ExecuteHookViaImport(self, data, context, **kwargs):
+    """Execute the hook code in |data| directly.
+
+    Args:
+      data: The code of the hook to execute.
+      context: Basic Python context to execute the hook inside.
+      kwargs: Arbitrary arguments to pass to the hook script.
+
+    Raises:
+      HookError: When the hooks failed for any reason.
+    """
+    # Exec, storing global context in the context dict.  We catch exceptions
+    # and convert to a HookError w/ just the failing traceback.
+    try:
+      exec(compile(data, self._script_fullpath, 'exec'), context)
+    except Exception:
+      raise HookError('%s\nFailed to import %s hook; see traceback above.' %
+                      (traceback.format_exc(), self._hook_type))
+
+    # Running the script should have defined a main() function.
+    if 'main' not in context:
+      raise HookError('Missing main() in: "%s"' % self._script_fullpath)
+
+    # Call the main function in the hook.  If the hook should cause the
+    # build to fail, it will raise an Exception.  We'll catch that convert
+    # to a HookError w/ just the failing traceback.
+    try:
+      context['main'](**kwargs)
+    except Exception:
+      raise HookError('%s\nFailed to run main() for %s hook; see traceback '
+                      'above.' % (traceback.format_exc(), self._hook_type))
+
+  def _ExecuteHook(self, **kwargs):
+    """Actually execute the given hook.
+
+    This will run the hook's 'main' function in our python interpreter.
+
+    Args:
+      kwargs: Keyword arguments to pass to the hook.  These are often specific
+          to the hook type.  For instance, pre-upload hooks will contain
+          a project_list.
+    """
+    # Keep sys.path and CWD stashed away so that we can always restore them
+    # upon function exit.
+    orig_path = os.getcwd()
+    orig_syspath = sys.path
+
+    try:
+      # Always run hooks with CWD as topdir.
+      os.chdir(self._repo_topdir)
+
+      # Put the hook dir as the first item of sys.path so hooks can do
+      # relative imports.  We want to replace the repo dir as [0] so
+      # hooks can't import repo files.
+      sys.path = [os.path.dirname(self._script_fullpath)] + sys.path[1:]
+
+      # Initial global context for the hook to run within.
+      context = {'__file__': self._script_fullpath}
+
+      # Add 'hook_should_take_kwargs' to the arguments to be passed to main.
+      # We don't actually want hooks to define their main with this argument--
+      # it's there to remind them that their hook should always take **kwargs.
+      # For instance, a pre-upload hook should be defined like:
+      #   def main(project_list, **kwargs):
+      #
+      # This allows us to later expand the API without breaking old hooks.
+      kwargs = kwargs.copy()
+      kwargs['hook_should_take_kwargs'] = True
+
+      # See what version of python the hook has been written against.
+      data = open(self._script_fullpath).read()
+      interp = self._ExtractInterpFromShebang(data)
+      reexec = False
+      if interp:
+        prog = os.path.basename(interp)
+        if prog.startswith('python2') and sys.version_info.major != 2:
+          reexec = True
+        elif prog.startswith('python3') and sys.version_info.major == 2:
+          reexec = True
+
+      # Attempt to execute the hooks through the requested version of Python.
+      if reexec:
+        try:
+          self._ExecuteHookViaReexec(interp, context, **kwargs)
+        except OSError as e:
+          if e.errno == errno.ENOENT:
+            # We couldn't find the interpreter, so fallback to importing.
+            reexec = False
+          else:
+            raise
+
+      # Run the hook by importing directly.
+      if not reexec:
+        self._ExecuteHookViaImport(data, context, **kwargs)
+    finally:
+      # Restore sys.path and CWD.
+      sys.path = orig_syspath
+      os.chdir(orig_path)
+
+  def _CheckHook(self):
+    # Bail with a nice error if we can't find the hook.
+    if not os.path.isfile(self._script_fullpath):
+      raise HookError('Couldn\'t find repo hook: %s' % self._script_fullpath)
+
+  def Run(self, **kwargs):
+    """Run the hook.
+
+    If the hook doesn't exist (because there is no hooks project or because
+    this particular hook is not enabled), this is a no-op.
+
+    Args:
+      user_allows_all_hooks: If True, we will never prompt about running the
+          hook--we'll just assume it's OK to run it.
+      kwargs: Keyword arguments to pass to the hook.  These are often specific
+          to the hook type.  For instance, pre-upload hooks will contain
+          a project_list.
+
+    Returns:
+      True: On success or ignore hooks by user-request
+      False: The hook failed. The caller should respond with aborting the action.
+        Some examples in which False is returned:
+        * Finding the hook failed while it was enabled, or
+        * the user declined to run a required hook (from _CheckForHookApproval)
+        In all these cases the user did not pass the proper arguments to
+        ignore the result through the option combinations as listed in
+        AddHookOptionGroup().
+    """
+    # Do not do anything in case bypass_hooks is set, or
+    # no-op if there is no hooks project or if hook is disabled.
+    if (self._bypass_hooks or
+        not self._hooks_project or
+        self._hook_type not in self._hooks_project.enabled_repo_hooks):
+      return True
+
+    passed = True
+    try:
+      self._CheckHook()
+
+      # Make sure the user is OK with running the hook.
+      if self._allow_all_hooks or self._CheckForHookApproval():
+        # Run the hook with the same version of python we're using.
+        self._ExecuteHook(**kwargs)
+    except SystemExit as e:
+      passed = False
+      print('ERROR: %s hooks exited with exit code: %s' % (self._hook_type, str(e)),
+            file=sys.stderr)
+    except HookError as e:
+      passed = False
+      print('ERROR: %s' % str(e), file=sys.stderr)
+
+    if not passed and self._ignore_hooks:
+      print('\nWARNING: %s hooks failed, but continuing anyways.' % self._hook_type,
+            file=sys.stderr)
+      passed = True
+
+    return passed
+
+  @classmethod
+  def FromSubcmd(cls, manifest, opt, *args, **kwargs):
+    """Method to construct the repo hook class
+
+    Args:
+      manifest: The current active manifest for this command from which we
+          extract a couple of fields.
+      opt: Contains the commandline options for the action of this hook.
+          It should contain the options added by AddHookOptionGroup() in which
+          we are interested in RepoHook execution.
+    """
+    for key in ('bypass_hooks', 'allow_all_hooks', 'ignore_hooks'):
+      kwargs.setdefault(key, getattr(opt, key))
+    kwargs.update({
+        'hooks_project': manifest.repo_hooks_project,
+        'repo_topdir': manifest.topdir,
+        'manifest_url': manifest.manifestProject.GetRemote('origin').url,
+    })
+    return cls(*args, **kwargs)
+
+  @staticmethod
+  def AddOptionGroup(parser, name):
+    """Help options relating to the various hooks."""
+
+    # Note that verify and no-verify are NOT opposites of each other, which
+    # is why they store to different locations. We are using them to match
+    # 'git commit' syntax.
+    group = parser.add_option_group(name + ' hooks')
+    group.add_option('--no-verify',
+                     dest='bypass_hooks', action='store_true',
+                     help='Do not run the %s hook.' % name)
+    group.add_option('--verify',
+                     dest='allow_all_hooks', action='store_true',
+                     help='Run the %s hook without prompting.' % name)
+    group.add_option('--ignore-hooks',
+                     action='store_true',
+                     help='Do not abort if %s hooks fail.' % name)
--- a/main.py
+++ b/main.py
@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # -*- coding:utf-8 -*-
 #
 # Copyright (C) 2008 The Android Open Source Project
@ -63,7 +63,7 @@ from error import NoManifestException
 from error import NoSuchProjectError
 from error import RepoChangedException
 import gitc_utils
-from manifest_xml import GitcManifest, XmlManifest
+from manifest_xml import GitcClient, RepoClient
 from pager import RunPager, TerminatePager
 from wrapper import WrapperPath, Wrapper

@ -85,9 +85,10 @@ MIN_PYTHON_VERSION_SOFT = (3, 6)
 MIN_PYTHON_VERSION_HARD = (3, 4)

 if sys.version_info.major < 3:
-  print('repo: warning: Python 2 is no longer supported; '
+  print('repo: error: Python 2 is no longer supported; '
        'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION_SOFT),
        file=sys.stderr)
+  sys.exit(1)
 else:
  if sys.version_info < MIN_PYTHON_VERSION_HARD:
    print('repo: error: Python 3 version is too old; '
@ -211,14 +212,15 @@ class _Repo(object):
      return 1

    cmd.repodir = self.repodir
-    cmd.manifest = XmlManifest(cmd.repodir)
+    cmd.client = RepoClient(cmd.repodir)
+    cmd.manifest = cmd.client.manifest
    cmd.gitc_manifest = None
    gitc_client_name = gitc_utils.parse_clientdir(os.getcwd())
    if gitc_client_name:
-      cmd.gitc_manifest = GitcManifest(cmd.repodir, gitc_client_name)
-      cmd.manifest.isGitcClient = True
+      cmd.gitc_manifest = GitcClient(cmd.repodir, gitc_client_name)
+      cmd.client.isGitcClient = True

-    Editor.globalConfig = cmd.manifest.globalConfig
+    Editor.globalConfig = cmd.client.globalConfig

    if not isinstance(cmd, MirrorSafeCommand) and cmd.manifest.IsMirror:
      print("fatal: '%s' requires a working directory" % name,
@ -246,7 +248,7 @@ class _Repo(object):
      return 1

    if gopts.pager is not False and not isinstance(cmd, InteractiveCommand):
-      config = cmd.manifest.globalConfig
+      config = cmd.client.globalConfig
      if gopts.pager:
        use_pager = True
      else:
--- a/manifest_xml.py
+++ b/manifest_xml.py
@ -31,7 +31,7 @@ else:
  urllib.parse = urlparse

 import gitc_utils
-from git_config import GitConfig
+from git_config import GitConfig, IsId
 from git_refs import R_HEADS, HEAD
 import platform_utils
 from project import RemoteSpec, Project, MetaProject
@ -187,13 +187,24 @@ class _XmlRemote(object):
 class XmlManifest(object):
  """manages the repo configuration file"""

-  def __init__(self, repodir):
+  def __init__(self, repodir, manifest_file, local_manifests=None):
+    """Initialize.
+
+    Args:
+      repodir: Path to the .repo/ dir for holding all internal checkout state.
+          It must be in the top directory of the repo client checkout.
+      manifest_file: Full path to the manifest file to parse.  This will usually
+          be |repodir|/|MANIFEST_FILE_NAME|.
+      local_manifests: Full path to the directory of local override manifests.
+          This will usually be |repodir|/|LOCAL_MANIFESTS_DIR_NAME|.
+    """
+    # TODO(vapier): Move this out of this class.
+    self.globalConfig = GitConfig.ForUser()
+
    self.repodir = os.path.abspath(repodir)
    self.topdir = os.path.dirname(self.repodir)
-    self.manifestFile = os.path.join(self.repodir, MANIFEST_FILE_NAME)
-    self.globalConfig = GitConfig.ForUser()
-    self.localManifestWarning = False
-    self.isGitcClient = False
+    self.manifestFile = manifest_file
+    self.local_manifests = local_manifests
    self._load_local_manifests = True

    self.repoProject = MetaProject(self, 'repo',
@ -284,9 +295,8 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
  def _ParseGroups(self, groups):
    return [x for x in re.split(r'[,\s]+', groups) if x]

-  def Save(self, fd, peg_rev=False, peg_rev_upstream=True, peg_rev_dest_branch=True, groups=None):
-    """Write the current manifest out to the given file descriptor.
-    """
+  def ToXml(self, peg_rev=False, peg_rev_upstream=True, peg_rev_dest_branch=True, groups=None):
+    """Return the current manifest XML."""
    mp = self.manifestProject

    if groups is None:
@ -460,6 +470,56 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                     ' '.join(self._repo_hooks_project.enabled_repo_hooks))
      root.appendChild(e)

+    return doc
+
+  def ToDict(self, **kwargs):
+    """Return the current manifest as a dictionary."""
+    # Elements that may only appear once.
+    SINGLE_ELEMENTS = {
+        'notice',
+        'default',
+        'manifest-server',
+        'repo-hooks',
+    }
+    # Elements that may be repeated.
+    MULTI_ELEMENTS = {
+        'remote',
+        'remove-project',
+        'project',
+        'extend-project',
+        'include',
+        # These are children of 'project' nodes.
+        'annotation',
+        'project',
+        'copyfile',
+        'linkfile',
+    }
+
+    doc = self.ToXml(**kwargs)
+    ret = {}
+
+    def append_children(ret, node):
+      for child in node.childNodes:
+        if child.nodeType == xml.dom.Node.ELEMENT_NODE:
+          attrs = child.attributes
+          element = dict((attrs.item(i).localName, attrs.item(i).value)
+                         for i in range(attrs.length))
+          if child.nodeName in SINGLE_ELEMENTS:
+            ret[child.nodeName] = element
+          elif child.nodeName in MULTI_ELEMENTS:
+            ret.setdefault(child.nodeName, []).append(element)
+          else:
+            raise ManifestParseError('Unhandled element "%s"' % (child.nodeName,))
+
+          append_children(element, child)
+
+    append_children(ret, doc.firstChild)
+
+    return ret
+
+  def Save(self, fd, **kwargs):
+    """Write the current manifest out to the given file descriptor."""
+    doc = self.ToXml(**kwargs)
    doc.writexml(fd, '', '  ', '\n', 'UTF-8')

  def _output_manifest_project_extras(self, p, e):
@ -554,23 +614,11 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
      nodes.append(self._ParseManifestXml(self.manifestFile,
                                          self.manifestProject.worktree))

-      if self._load_local_manifests:
-        local = os.path.join(self.repodir, LOCAL_MANIFEST_NAME)
-        if os.path.exists(local):
-          if not self.localManifestWarning:
-            self.localManifestWarning = True
-            print('warning: %s is deprecated; put local manifests '
-                  'in `%s` instead' % (LOCAL_MANIFEST_NAME,
-                                       os.path.join(self.repodir, LOCAL_MANIFESTS_DIR_NAME)),
-                  file=sys.stderr)
-          nodes.append(self._ParseManifestXml(local, self.repodir))
-
-        local_dir = os.path.abspath(os.path.join(self.repodir,
-                                                 LOCAL_MANIFESTS_DIR_NAME))
+      if self._load_local_manifests and self.local_manifests:
        try:
-          for local_file in sorted(platform_utils.listdir(local_dir)):
+          for local_file in sorted(platform_utils.listdir(self.local_manifests)):
            if local_file.endswith('.xml'):
-              local = os.path.join(local_dir, local_file)
+              local = os.path.join(self.local_manifests, local_file)
              nodes.append(self._ParseManifestXml(local, self.repodir))
        except OSError:
          pass
@ -589,7 +637,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md

      self._loaded = True

-  def _ParseManifestXml(self, path, include_root):
+  def _ParseManifestXml(self, path, include_root, parent_groups=''):
    try:
      root = xml.dom.minidom.parse(path)
    except (OSError, xml.parsers.expat.ExpatError) as e:
@ -608,12 +656,17 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
    for node in manifest.childNodes:
      if node.nodeName == 'include':
        name = self._reqatt(node, 'name')
+        include_groups = ''
+        if parent_groups:
+          include_groups = parent_groups
+        if node.hasAttribute('groups'):
+          include_groups = node.getAttribute('groups') + ',' + include_groups
        fp = os.path.join(include_root, name)
        if not os.path.isfile(fp):
          raise ManifestParseError("include %s doesn't exist or isn't a file"
                                   % (name,))
        try:
-          nodes.extend(self._ParseManifestXml(fp, include_root))
+          nodes.extend(self._ParseManifestXml(fp, include_root, include_groups))
        # should isolate this to the exact exception, but that's
        # tricky.  actual parsing implementation may vary.
        except (KeyboardInterrupt, RuntimeError, SystemExit):
@ -622,6 +675,11 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
          raise ManifestParseError(
              "failed parsing included manifest %s: %s" % (name, e))
      else:
+        if parent_groups and node.nodeName == 'project':
+          nodeGroups = parent_groups
+          if node.hasAttribute('groups'):
+            nodeGroups = node.getAttribute('groups') + ',' + nodeGroups
+          node.setAttribute('groups', nodeGroups)
        nodes.append(node)
    return nodes

@ -709,6 +767,10 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
            p.groups.extend(groups)
          if revision:
            p.revisionExpr = revision
+            if IsId(revision):
+              p.revisionId = revision
+            else:
+              p.revisionId = None
          if remote:
            p.remote = remote.ToRemoteSpec(name)
      if node.nodeName == 'repo-hooks':
@ -978,6 +1040,10 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
    return project

  def GetProjectPaths(self, name, path):
+    # The manifest entries might have trailing slashes.  Normalize them to avoid
+    # unexpected filesystem behavior since we do string concatenation below.
+    path = path.rstrip('/')
+    name = name.rstrip('/')
    use_git_worktrees = False
    relpath = path
    if self.IsMirror:
@ -1010,6 +1076,10 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
    return os.path.relpath(relpath, parent_relpath)

  def GetSubprojectPaths(self, parent, name, path):
+    # The manifest entries might have trailing slashes.  Normalize them to avoid
+    # unexpected filesystem behavior since we do string concatenation below.
+    path = path.rstrip('/')
+    name = name.rstrip('/')
    relpath = self._JoinRelpath(parent.relpath, path)
    gitdir = os.path.join(parent.gitdir, 'subprojects', '%s.git' % path)
    objdir = os.path.join(parent.gitdir, 'subproject-objects', '%s.git' % name)
@ -1196,15 +1266,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md


 class GitcManifest(XmlManifest):
-
-  def __init__(self, repodir, gitc_client_name):
-    """Initialize the GitcManifest object."""
-    super(GitcManifest, self).__init__(repodir)
-    self.isGitcClient = True
-    self.gitc_client_name = gitc_client_name
-    self.gitc_client_dir = os.path.join(gitc_utils.get_gitc_manifest_dir(),
-                                        gitc_client_name)
-    self.manifestFile = os.path.join(self.gitc_client_dir, '.manifest')
+  """Parser for GitC (git-in-the-cloud) manifests."""

  def _ParseProject(self, node, parent=None):
    """Override _ParseProject and add support for GITC specific attributes."""
@ -1215,3 +1277,38 @@ class GitcManifest(XmlManifest):
    """Output GITC Specific Project attributes"""
    if p.old_revision:
      e.setAttribute('old-revision', str(p.old_revision))
+
+
+class RepoClient(XmlManifest):
+  """Manages a repo client checkout."""
+
+  def __init__(self, repodir, manifest_file=None):
+    self.isGitcClient = False
+
+    if os.path.exists(os.path.join(repodir, LOCAL_MANIFEST_NAME)):
+      print('error: %s is not supported; put local manifests in `%s` instead' %
+            (LOCAL_MANIFEST_NAME, os.path.join(repodir, LOCAL_MANIFESTS_DIR_NAME)),
+            file=sys.stderr)
+      sys.exit(1)
+
+    if manifest_file is None:
+      manifest_file = os.path.join(repodir, MANIFEST_FILE_NAME)
+    local_manifests = os.path.abspath(os.path.join(repodir, LOCAL_MANIFESTS_DIR_NAME))
+    super(RepoClient, self).__init__(repodir, manifest_file, local_manifests)
+
+    # TODO: Completely separate manifest logic out of the client.
+    self.manifest = self
+
+
+class GitcClient(RepoClient, GitcManifest):
+  """Manages a GitC client checkout."""
+
+  def __init__(self, repodir, gitc_client_name):
+    """Initialize the GitcManifest object."""
+    self.gitc_client_name = gitc_client_name
+    self.gitc_client_dir = os.path.join(gitc_utils.get_gitc_manifest_dir(),
+                                        gitc_client_name)
+
+    super(GitcManifest, self).__init__(
+        repodir, os.path.join(self.gitc_client_dir, '.manifest'))
+    self.isGitcClient = True
--- a/project.py
+++ b/project.py
@ -18,7 +18,6 @@ from __future__ import print_function
 import errno
 import filecmp
 import glob
-import json
 import os
 import random
 import re
@ -29,13 +28,12 @@ import sys
 import tarfile
 import tempfile
 import time
-import traceback

 from color import Coloring
 from git_command import GitCommand, git_require
 from git_config import GitConfig, IsId, GetSchemeFromUrl, GetUrlCookieFile, \
    ID_RE
-from error import GitError, HookError, UploadError, DownloadError
+from error import GitError, UploadError, DownloadError
 from error import ManifestInvalidRevisionError, ManifestInvalidPathError
 from error import NoManifestException
 import platform_utils
@ -64,7 +62,8 @@ RETRY_JITTER_PERCENT = 0.1
 def _lwrite(path, content):
  lock = '%s.lock' % path

-  with open(lock, 'w') as fd:
+  # Maintain Unix line endings on all OS's to match git behavior.
+  with open(lock, 'w', newline='\n') as fd:
    fd.write(content)

  try:
@ -451,406 +450,6 @@ class RemoteSpec(object):
    self.orig_name = orig_name
    self.fetchUrl = fetchUrl

-
-class RepoHook(object):
-
-  """A RepoHook contains information about a script to run as a hook.
-
-  Hooks are used to run a python script before running an upload (for instance,
-  to run presubmit checks).  Eventually, we may have hooks for other actions.
-
-  This shouldn't be confused with files in the 'repo/hooks' directory.  Those
-  files are copied into each '.git/hooks' folder for each project.  Repo-level
-  hooks are associated instead with repo actions.
-
-  Hooks are always python.  When a hook is run, we will load the hook into the
-  interpreter and execute its main() function.
-  """
-
-  def __init__(self,
-               hook_type,
-               hooks_project,
-               topdir,
-               manifest_url,
-               abort_if_user_denies=False):
-    """RepoHook constructor.
-
-    Params:
-      hook_type: A string representing the type of hook.  This is also used
-          to figure out the name of the file containing the hook.  For
-          example: 'pre-upload'.
-      hooks_project: The project containing the repo hooks.  If you have a
-          manifest, this is manifest.repo_hooks_project.  OK if this is None,
-          which will make the hook a no-op.
-      topdir: Repo's top directory (the one containing the .repo directory).
-          Scripts will run with CWD as this directory.  If you have a manifest,
-          this is manifest.topdir
-      manifest_url: The URL to the manifest git repo.
-      abort_if_user_denies: If True, we'll throw a HookError() if the user
-          doesn't allow us to run the hook.
-    """
-    self._hook_type = hook_type
-    self._hooks_project = hooks_project
-    self._manifest_url = manifest_url
-    self._topdir = topdir
-    self._abort_if_user_denies = abort_if_user_denies
-
-    # Store the full path to the script for convenience.
-    if self._hooks_project:
-      self._script_fullpath = os.path.join(self._hooks_project.worktree,
-                                           self._hook_type + '.py')
-    else:
-      self._script_fullpath = None
-
-  def _GetHash(self):
-    """Return a hash of the contents of the hooks directory.
-
-    We'll just use git to do this.  This hash has the property that if anything
-    changes in the directory we will return a different has.
-
-    SECURITY CONSIDERATION:
-      This hash only represents the contents of files in the hook directory, not
-      any other files imported or called by hooks.  Changes to imported files
-      can change the script behavior without affecting the hash.
-
-    Returns:
-      A string representing the hash.  This will always be ASCII so that it can
-      be printed to the user easily.
-    """
-    assert self._hooks_project, "Must have hooks to calculate their hash."
-
-    # We will use the work_git object rather than just calling GetRevisionId().
-    # That gives us a hash of the latest checked in version of the files that
-    # the user will actually be executing.  Specifically, GetRevisionId()
-    # doesn't appear to change even if a user checks out a different version
-    # of the hooks repo (via git checkout) nor if a user commits their own revs.
-    #
-    # NOTE: Local (non-committed) changes will not be factored into this hash.
-    # I think this is OK, since we're really only worried about warning the user
-    # about upstream changes.
-    return self._hooks_project.work_git.rev_parse('HEAD')
-
-  def _GetMustVerb(self):
-    """Return 'must' if the hook is required; 'should' if not."""
-    if self._abort_if_user_denies:
-      return 'must'
-    else:
-      return 'should'
-
-  def _CheckForHookApproval(self):
-    """Check to see whether this hook has been approved.
-
-    We'll accept approval of manifest URLs if they're using secure transports.
-    This way the user can say they trust the manifest hoster.  For insecure
-    hosts, we fall back to checking the hash of the hooks repo.
-
-    Note that we ask permission for each individual hook even though we use
-    the hash of all hooks when detecting changes.  We'd like the user to be
-    able to approve / deny each hook individually.  We only use the hash of all
-    hooks because there is no other easy way to detect changes to local imports.
-
-    Returns:
-      True if this hook is approved to run; False otherwise.
-
-    Raises:
-      HookError: Raised if the user doesn't approve and abort_if_user_denies
-          was passed to the consturctor.
-    """
-    if self._ManifestUrlHasSecureScheme():
-      return self._CheckForHookApprovalManifest()
-    else:
-      return self._CheckForHookApprovalHash()
-
-  def _CheckForHookApprovalHelper(self, subkey, new_val, main_prompt,
-                                  changed_prompt):
-    """Check for approval for a particular attribute and hook.
-
-    Args:
-      subkey: The git config key under [repo.hooks.<hook_type>] to store the
-          last approved string.
-      new_val: The new value to compare against the last approved one.
-      main_prompt: Message to display to the user to ask for approval.
-      changed_prompt: Message explaining why we're re-asking for approval.
-
-    Returns:
-      True if this hook is approved to run; False otherwise.
-
-    Raises:
-      HookError: Raised if the user doesn't approve and abort_if_user_denies
-          was passed to the consturctor.
-    """
-    hooks_config = self._hooks_project.config
-    git_approval_key = 'repo.hooks.%s.%s' % (self._hook_type, subkey)
-
-    # Get the last value that the user approved for this hook; may be None.
-    old_val = hooks_config.GetString(git_approval_key)
-
-    if old_val is not None:
-      # User previously approved hook and asked not to be prompted again.
-      if new_val == old_val:
-        # Approval matched.  We're done.
-        return True
-      else:
-        # Give the user a reason why we're prompting, since they last told
-        # us to "never ask again".
-        prompt = 'WARNING: %s\n\n' % (changed_prompt,)
-    else:
-      prompt = ''
-
-    # Prompt the user if we're not on a tty; on a tty we'll assume "no".
-    if sys.stdout.isatty():
-      prompt += main_prompt + ' (yes/always/NO)? '
-      response = input(prompt).lower()
-      print()
-
-      # User is doing a one-time approval.
-      if response in ('y', 'yes'):
-        return True
-      elif response == 'always':
-        hooks_config.SetString(git_approval_key, new_val)
-        return True
-
-    # For anything else, we'll assume no approval.
-    if self._abort_if_user_denies:
-      raise HookError('You must allow the %s hook or use --no-verify.' %
-                      self._hook_type)
-
-    return False
-
-  def _ManifestUrlHasSecureScheme(self):
-    """Check if the URI for the manifest is a secure transport."""
-    secure_schemes = ('file', 'https', 'ssh', 'persistent-https', 'sso', 'rpc')
-    parse_results = urllib.parse.urlparse(self._manifest_url)
-    return parse_results.scheme in secure_schemes
-
-  def _CheckForHookApprovalManifest(self):
-    """Check whether the user has approved this manifest host.
-
-    Returns:
-      True if this hook is approved to run; False otherwise.
-    """
-    return self._CheckForHookApprovalHelper(
-        'approvedmanifest',
-        self._manifest_url,
-        'Run hook scripts from %s' % (self._manifest_url,),
-        'Manifest URL has changed since %s was allowed.' % (self._hook_type,))
-
-  def _CheckForHookApprovalHash(self):
-    """Check whether the user has approved the hooks repo.
-
-    Returns:
-      True if this hook is approved to run; False otherwise.
-    """
-    prompt = ('Repo %s run the script:\n'
-              '  %s\n'
-              '\n'
-              'Do you want to allow this script to run')
-    return self._CheckForHookApprovalHelper(
-        'approvedhash',
-        self._GetHash(),
-        prompt % (self._GetMustVerb(), self._script_fullpath),
-        'Scripts have changed since %s was allowed.' % (self._hook_type,))
-
-  @staticmethod
-  def _ExtractInterpFromShebang(data):
-    """Extract the interpreter used in the shebang.
-
-    Try to locate the interpreter the script is using (ignoring `env`).
-
-    Args:
-      data: The file content of the script.
-
-    Returns:
-      The basename of the main script interpreter, or None if a shebang is not
-      used or could not be parsed out.
-    """
-    firstline = data.splitlines()[:1]
-    if not firstline:
-      return None
-
-    # The format here can be tricky.
-    shebang = firstline[0].strip()
-    m = re.match(r'^#!\s*([^\s]+)(?:\s+([^\s]+))?', shebang)
-    if not m:
-      return None
-
-    # If the using `env`, find the target program.
-    interp = m.group(1)
-    if os.path.basename(interp) == 'env':
-      interp = m.group(2)
-
-    return interp
-
-  def _ExecuteHookViaReexec(self, interp, context, **kwargs):
-    """Execute the hook script through |interp|.
-
-    Note: Support for this feature should be dropped ~Jun 2021.
-
-    Args:
-      interp: The Python program to run.
-      context: Basic Python context to execute the hook inside.
-      kwargs: Arbitrary arguments to pass to the hook script.
-
-    Raises:
-      HookError: When the hooks failed for any reason.
-    """
-    # This logic needs to be kept in sync with _ExecuteHookViaImport below.
-    script = """
-import json, os, sys
-path = '''%(path)s'''
-kwargs = json.loads('''%(kwargs)s''')
-context = json.loads('''%(context)s''')
-sys.path.insert(0, os.path.dirname(path))
-data = open(path).read()
-exec(compile(data, path, 'exec'), context)
-context['main'](**kwargs)
-""" % {
-        'path': self._script_fullpath,
-        'kwargs': json.dumps(kwargs),
-        'context': json.dumps(context),
-    }
-
-    # We pass the script via stdin to avoid OS argv limits.  It also makes
-    # unhandled exception tracebacks less verbose/confusing for users.
-    cmd = [interp, '-c', 'import sys; exec(sys.stdin.read())']
-    proc = subprocess.Popen(cmd, stdin=subprocess.PIPE)
-    proc.communicate(input=script.encode('utf-8'))
-    if proc.returncode:
-      raise HookError('Failed to run %s hook.' % (self._hook_type,))
-
-  def _ExecuteHookViaImport(self, data, context, **kwargs):
-    """Execute the hook code in |data| directly.
-
-    Args:
-      data: The code of the hook to execute.
-      context: Basic Python context to execute the hook inside.
-      kwargs: Arbitrary arguments to pass to the hook script.
-
-    Raises:
-      HookError: When the hooks failed for any reason.
-    """
-    # Exec, storing global context in the context dict.  We catch exceptions
-    # and convert to a HookError w/ just the failing traceback.
-    try:
-      exec(compile(data, self._script_fullpath, 'exec'), context)
-    except Exception:
-      raise HookError('%s\nFailed to import %s hook; see traceback above.' %
-                      (traceback.format_exc(), self._hook_type))
-
-    # Running the script should have defined a main() function.
-    if 'main' not in context:
-      raise HookError('Missing main() in: "%s"' % self._script_fullpath)
-
-    # Call the main function in the hook.  If the hook should cause the
-    # build to fail, it will raise an Exception.  We'll catch that convert
-    # to a HookError w/ just the failing traceback.
-    try:
-      context['main'](**kwargs)
-    except Exception:
-      raise HookError('%s\nFailed to run main() for %s hook; see traceback '
-                      'above.' % (traceback.format_exc(), self._hook_type))
-
-  def _ExecuteHook(self, **kwargs):
-    """Actually execute the given hook.
-
-    This will run the hook's 'main' function in our python interpreter.
-
-    Args:
-      kwargs: Keyword arguments to pass to the hook.  These are often specific
-          to the hook type.  For instance, pre-upload hooks will contain
-          a project_list.
-    """
-    # Keep sys.path and CWD stashed away so that we can always restore them
-    # upon function exit.
-    orig_path = os.getcwd()
-    orig_syspath = sys.path
-
-    try:
-      # Always run hooks with CWD as topdir.
-      os.chdir(self._topdir)
-
-      # Put the hook dir as the first item of sys.path so hooks can do
-      # relative imports.  We want to replace the repo dir as [0] so
-      # hooks can't import repo files.
-      sys.path = [os.path.dirname(self._script_fullpath)] + sys.path[1:]
-
-      # Initial global context for the hook to run within.
-      context = {'__file__': self._script_fullpath}
-
-      # Add 'hook_should_take_kwargs' to the arguments to be passed to main.
-      # We don't actually want hooks to define their main with this argument--
-      # it's there to remind them that their hook should always take **kwargs.
-      # For instance, a pre-upload hook should be defined like:
-      #   def main(project_list, **kwargs):
-      #
-      # This allows us to later expand the API without breaking old hooks.
-      kwargs = kwargs.copy()
-      kwargs['hook_should_take_kwargs'] = True
-
-      # See what version of python the hook has been written against.
-      data = open(self._script_fullpath).read()
-      interp = self._ExtractInterpFromShebang(data)
-      reexec = False
-      if interp:
-        prog = os.path.basename(interp)
-        if prog.startswith('python2') and sys.version_info.major != 2:
-          reexec = True
-        elif prog.startswith('python3') and sys.version_info.major == 2:
-          reexec = True
-
-      # Attempt to execute the hooks through the requested version of Python.
-      if reexec:
-        try:
-          self._ExecuteHookViaReexec(interp, context, **kwargs)
-        except OSError as e:
-          if e.errno == errno.ENOENT:
-            # We couldn't find the interpreter, so fallback to importing.
-            reexec = False
-          else:
-            raise
-
-      # Run the hook by importing directly.
-      if not reexec:
-        self._ExecuteHookViaImport(data, context, **kwargs)
-    finally:
-      # Restore sys.path and CWD.
-      sys.path = orig_syspath
-      os.chdir(orig_path)
-
-  def Run(self, user_allows_all_hooks, **kwargs):
-    """Run the hook.
-
-    If the hook doesn't exist (because there is no hooks project or because
-    this particular hook is not enabled), this is a no-op.
-
-    Args:
-      user_allows_all_hooks: If True, we will never prompt about running the
-          hook--we'll just assume it's OK to run it.
-      kwargs: Keyword arguments to pass to the hook.  These are often specific
-          to the hook type.  For instance, pre-upload hooks will contain
-          a project_list.
-
-    Raises:
-      HookError: If there was a problem finding the hook or the user declined
-          to run a required hook (from _CheckForHookApproval).
-    """
-    # No-op if there is no hooks project or if hook is disabled.
-    if ((not self._hooks_project) or (self._hook_type not in
-                                      self._hooks_project.enabled_repo_hooks)):
-      return
-
-    # Bail with a nice error if we can't find the hook.
-    if not os.path.isfile(self._script_fullpath):
-      raise HookError('Couldn\'t find repo hook: "%s"' % self._script_fullpath)
-
-    # Make sure the user is OK with running the hook.
-    if (not user_allows_all_hooks) and (not self._CheckForHookApproval()):
-      return
-
-    # Run the hook with the same version of python we're using.
-    self._ExecuteHook(**kwargs)
-
-
 class Project(object):
  # These objects can be shared between several working trees.
  shareable_files = ['description', 'info']
@ -912,7 +511,7 @@ class Project(object):
                     with exponential backoff and jitter.
      old_revision: saved git commit id for open GITC projects.
    """
-    self.manifest = manifest
+    self.client = self.manifest = manifest
    self.name = name
    self.remote = remote
    self.gitdir = gitdir.replace('\\', '/')
@ -953,7 +552,7 @@ class Project(object):
    self.linkfiles = []
    self.annotations = []
    self.config = GitConfig.ForRepository(gitdir=self.gitdir,
-                                          defaults=self.manifest.globalConfig)
+                                          defaults=self.client.globalConfig)

    if self.worktree:
      self.work_git = self._GitGetByExec(self, bare=False, gitdir=gitdir)
@ -1428,10 +1027,11 @@ class Project(object):
    if GitCommand(self, cmd, bare=True).Wait() != 0:
      raise UploadError('Upload failed')

-    msg = "posted to %s for %s" % (branch.remote.review, dest_branch)
-    self.bare_git.UpdateRef(R_PUB + branch.name,
-                            R_HEADS + branch.name,
-                            message=msg)
+    if not dryrun:
+      msg = "posted to %s for %s" % (branch.remote.review, dest_branch)
+      self.bare_git.UpdateRef(R_PUB + branch.name,
+                              R_HEADS + branch.name,
+                              message=msg)

 # Sync ##
  def _ExtractArchive(self, tarpath, path=None):
@ -1570,7 +1170,7 @@ class Project(object):
    self._InitHooks()

  def _CopyAndLinkFiles(self):
-    if self.manifest.isGitcClient:
+    if self.client.isGitcClient:
      return
    for copyfile in self.copyfiles:
      copyfile._Copy()
@ -2311,6 +1911,27 @@ class Project(object):
    # Enable the extension!
    self.config.SetString('extensions.%s' % (key,), value)

+  def ResolveRemoteHead(self, name=None):
+    """Find out what the default branch (HEAD) points to.
+
+    Normally this points to refs/heads/master, but projects are moving to main.
+    Support whatever the server uses rather than hardcoding "master" ourselves.
+    """
+    if name is None:
+      name = self.remote.name
+
+    # The output will look like (NB: tabs are separators):
+    # ref: refs/heads/master	HEAD
+    # 5f6803b100bb3cd0f534e96e88c91373e8ed1c44	HEAD
+    output = self.bare_git.ls_remote('-q', '--symref', '--exit-code', name, 'HEAD')
+
+    for line in output.splitlines():
+      lhs, rhs = line.split('\t', 1)
+      if rhs == 'HEAD' and lhs.startswith('ref:'):
+        return lhs[4:].strip()
+
+    return None
+
  def _CheckForImmutableRevision(self):
    try:
      # if revision (sha or tag) is not present then following function
@ -2793,7 +2414,7 @@ class Project(object):

          # Enable per-worktree config file support if possible.  This is more a
          # nice-to-have feature for users rather than a hard requirement.
-          if git_require((2, 19, 0)):
+          if git_require((2, 20, 0)):
            self.EnableRepositoryExtension('worktreeConfig')

      # If we have a separate directory to hold refs, initialize it as well.
@ -3082,12 +2703,14 @@ class Project(object):
    # Some platforms (e.g. Windows) won't let us update dotgit in situ because
    # of file permissions.  Delete it and recreate it from scratch to avoid.
    platform_utils.remove(dotgit)
-    # Use relative path from checkout->worktree.
-    with open(dotgit, 'w') as fp:
+    # Use relative path from checkout->worktree & maintain Unix line endings
+    # on all OS's to match git behavior.
+    with open(dotgit, 'w', newline='\n') as fp:
      print('gitdir:', os.path.relpath(git_worktree_path, self.worktree),
            file=fp)
-    # Use relative path from worktree->checkout.
-    with open(os.path.join(git_worktree_path, 'gitdir'), 'w') as fp:
+    # Use relative path from worktree->checkout & maintain Unix line endings
+    # on all OS's to match git behavior.
+    with open(os.path.join(git_worktree_path, 'gitdir'), 'w', newline='\n') as fp:
      print(os.path.relpath(dotgit, git_worktree_path), file=fp)

    self._InitMRef()
@ -3208,6 +2831,13 @@ class Project(object):
      self._bare = bare
      self._gitdir = gitdir

+    # __getstate__ and __setstate__ are required for pickling because __getattr__ exists.
+    def __getstate__(self):
+      return (self._project, self._bare, self._gitdir)
+
+    def __setstate__(self, state):
+      self._project, self._bare, self._gitdir = state
+
    def LsOthers(self):
      p = GitCommand(self._project,
                     ['ls-files',
--- a/release/sign-tag.py
+++ b/release/sign-tag.py
@ -18,11 +18,7 @@
 This is intended to be run only by the official Repo release managers, but it
 could be run by people maintaining their own fork of the project.

-NB: Avoid new releases on off-hours.  If something goes wrong, staff/oncall need
-to be active in order to respond quickly & effectively.  Recommend sticking to:
-* Mon - Thu, 9:00 - 14:00 PT (i.e. MTV time)
-* Avoid US holidays (and large international ones if possible)
-* Follow the normal Google production freeze schedule
+NB: Check docs/release-process.md for production freeze information.
 """

 import argparse
--- a/4
+++ b/4
@ -966,9 +966,7 @@ def _FindRepo():
  repo = None

  olddir = None
-  while curdir != '/' \
-          and curdir != olddir \
-          and not repo:
+  while curdir != olddir and not repo:
    repo = os.path.join(curdir, repodir, REPO_MAIN)
    if not os.path.isfile(repo):
      repo = None
--- a/subcmds/diffmanifests.py
+++ b/subcmds/diffmanifests.py
@ -16,7 +16,7 @@

 from color import Coloring
 from command import PagedCommand
-from manifest_xml import XmlManifest
+from manifest_xml import RepoClient


 class _Coloring(Coloring):
@ -183,7 +183,7 @@ synced and their revisions won't be found.
      self.OptionParser.error('missing manifests to diff')

  def Execute(self, opt, args):
-    self.out = _Coloring(self.manifest.globalConfig)
+    self.out = _Coloring(self.client.globalConfig)
    self.printText = self.out.nofmt_printer('text')
    if opt.color:
      self.printProject = self.out.nofmt_printer('project', attr='bold')
@ -193,12 +193,12 @@ synced and their revisions won't be found.
    else:
      self.printProject = self.printAdded = self.printRemoved = self.printRevision = self.printText

-    manifest1 = XmlManifest(self.manifest.repodir)
+    manifest1 = RepoClient(self.manifest.repodir)
    manifest1.Override(args[0], load_local_manifests=False)
    if len(args) == 1:
      manifest2 = self.manifest
    else:
-      manifest2 = XmlManifest(self.manifest.repodir)
+      manifest2 = RepoClient(self.manifest.repodir)
      manifest2.Override(args[1], load_local_manifests=False)

    diff = manifest1.projectsDiff(manifest2)
--- a/subcmds/help.py
+++ b/subcmds/help.py
@ -65,7 +65,7 @@ Displays detailed usage information about a command.
    def gitc_supported(cmd):
      if not isinstance(cmd, GitcAvailableCommand) and not isinstance(cmd, GitcClientCommand):
        return True
-      if self.manifest.isGitcClient:
+      if self.client.isGitcClient:
        return True
      if isinstance(cmd, GitcClientCommand):
        return False
@ -127,7 +127,7 @@ Displays detailed usage information about a command.
          self.wrap.end_paragraph(1)
        self.wrap.end_paragraph(0)

-    out = _Out(self.manifest.globalConfig)
+    out = _Out(self.client.globalConfig)
    out._PrintSection('Summary', 'helpSummary')
    cmd.OptionParser.print_help()
    out._PrintSection('Description', 'helpDescription')
--- a/subcmds/info.py
+++ b/subcmds/info.py
@ -44,7 +44,7 @@ class Info(PagedCommand):
                 help="Disable all remote operations")

  def Execute(self, opt, args):
-    self.out = _Coloring(self.manifest.globalConfig)
+    self.out = _Coloring(self.client.globalConfig)
    self.heading = self.out.printer('heading', attr='bold')
    self.headtext = self.out.nofmt_printer('headtext', fg='yellow')
    self.redtext = self.out.printer('redtext', fg='red')
--- a/subcmds/init.py
+++ b/subcmds/init.py
@ -54,7 +54,8 @@ from the server and is installed in the .repo/ directory in the
 current working directory.

 The optional -b argument can be used to select the manifest branch
-to checkout and use.  If no branch is specified, master is assumed.
+to checkout and use.  If no branch is specified, the remote's default
+branch is used.

 The optional -m argument can be used to specify an alternate manifest
 to be used. If no manifest is specified, the manifest default.xml
@ -215,24 +216,27 @@ to update the working directory files.

      m._InitGitDir(mirror_git=mirrored_manifest_git)

-      if opt.manifest_branch:
-        m.revisionExpr = opt.manifest_branch
-      else:
-        m.revisionExpr = 'refs/heads/master'
-    else:
-      if opt.manifest_branch:
-        m.revisionExpr = opt.manifest_branch
-      else:
-        m.PreSync()
-
    self._ConfigureDepth(opt)

+    # Set the remote URL before the remote branch as we might need it below.
    if opt.manifest_url:
      r = m.GetRemote(m.remote.name)
      r.url = opt.manifest_url
      r.ResetFetch()
      r.Save()

+    if opt.manifest_branch:
+      m.revisionExpr = opt.manifest_branch
+    else:
+      if is_new:
+        default_branch = m.ResolveRemoteHead()
+        if default_branch is None:
+          # If the remote doesn't have HEAD configured, default to master.
+          default_branch = 'refs/heads/master'
+        m.revisionExpr = default_branch
+      else:
+        m.PreSync()
+
    groups = re.split(r'[,\s]+', opt.groups)
    all_platforms = ['linux', 'darwin', 'windows']
    platformize = lambda x: 'platform-' + x
@ -361,7 +365,7 @@ to update the working directory files.
    return a

  def _ShouldConfigureUser(self, opt):
-    gc = self.manifest.globalConfig
+    gc = self.client.globalConfig
    mp = self.manifest.manifestProject

    # If we don't have local settings, get from global.
@ -410,7 +414,7 @@ to update the working directory files.
    return False

  def _ConfigureColor(self):
-    gc = self.manifest.globalConfig
+    gc = self.client.globalConfig
    if self._HasColorSet(gc):
      return

@ -488,6 +492,9 @@ to update the working directory files.
    if opt.archive and opt.mirror:
      self.OptionParser.error('--mirror and --archive cannot be used together.')

+    if args:
+      self.OptionParser.error('init takes no arguments')
+
  def Execute(self, opt, args):
    git_require(MIN_GIT_VERSION_HARD, fail=True)
    if not git_require(MIN_GIT_VERSION_SOFT):
--- a/subcmds/manifest.py
+++ b/subcmds/manifest.py
@ -15,6 +15,8 @@
 # limitations under the License.

 from __future__ import print_function
+
+import json
 import os
 import sys

@ -30,7 +32,7 @@ class Manifest(PagedCommand):
  _helpDescription = """

 With the -o option, exports the current manifest for inspection.
-The manifest and (if present) local_manifest.xml are combined
+The manifest and (if present) local_manifests/ are combined
 together to produce a single manifest file.  This file can be stored
 in a Git repository for use during future 'repo init' invocations.

@ -68,6 +70,10 @@ to indicate the remote ref to push changes to via 'repo upload'.
                 help='If in -r mode, do not write the dest-branch field.  '
                 'Only of use if the branch names for a sha1 manifest are '
                 'sensitive.')
+    p.add_option('--json', default=False, action='store_true',
+                 help='Output manifest in JSON format (experimental).')
+    p.add_option('--pretty', default=False, action='store_true',
+                 help='Format output for humans to read.')
    p.add_option('-o', '--output-file',
                 dest='output_file',
                 default='-',
@ -83,10 +89,26 @@ to indicate the remote ref to push changes to via 'repo upload'.
      fd = sys.stdout
    else:
      fd = open(opt.output_file, 'w')
-    self.manifest.Save(fd,
-                       peg_rev=opt.peg_rev,
-                       peg_rev_upstream=opt.peg_rev_upstream,
-                       peg_rev_dest_branch=opt.peg_rev_dest_branch)
+    if opt.json:
+      print('warning: --json is experimental!', file=sys.stderr)
+      doc = self.manifest.ToDict(peg_rev=opt.peg_rev,
+                                 peg_rev_upstream=opt.peg_rev_upstream,
+                                 peg_rev_dest_branch=opt.peg_rev_dest_branch)
+
+      json_settings = {
+          # JSON style guide says Uunicode characters are fully allowed.
+          'ensure_ascii': False,
+          # We use 2 space indent to match JSON style guide.
+          'indent': 2 if opt.pretty else None,
+          'separators': (',', ': ') if opt.pretty else (',', ':'),
+          'sort_keys': True,
+      }
+      fd.write(json.dumps(doc, **json_settings))
+    else:
+      self.manifest.Save(fd,
+                         peg_rev=opt.peg_rev,
+                         peg_rev_upstream=opt.peg_rev_upstream,
+                         peg_rev_dest_branch=opt.peg_rev_dest_branch)
    fd.close()
    if opt.output_file != '-':
      print('Saved manifest to %s' % opt.output_file, file=sys.stderr)
--- a/subcmds/status.py
+++ b/subcmds/status.py
@ -16,17 +16,13 @@

 from __future__ import print_function

+import functools
 import glob
-import itertools
+import multiprocessing
 import os

 from command import PagedCommand

-try:
-  import threading as _threading
-except ImportError:
-  import dummy_threading as _threading
-
 from color import Coloring
 import platform_utils

@ -95,25 +91,20 @@ the following meanings:
    p.add_option('-q', '--quiet', action='store_true',
                 help="only print the name of modified projects")

-  def _StatusHelper(self, project, clean_counter, sem, quiet):
+  def _StatusHelper(self, quiet, project):
    """Obtains the status for a specific project.

    Obtains the status for a project, redirecting the output to
-    the specified object. It will release the semaphore
-    when done.
+    the specified object.

    Args:
+      quiet: Where to output the status.
      project: Project to get status of.
-      clean_counter: Counter for clean projects.
-      sem: Semaphore, will call release() when complete.
-      output: Where to output the status.
+
+    Returns:
+      The status of the project.
    """
-    try:
-      state = project.PrintWorkTreeStatus(quiet=quiet)
-      if state == 'CLEAN':
-        next(clean_counter)
-    finally:
-      sem.release()
+    return project.PrintWorkTreeStatus(quiet=quiet)

  def _FindOrphans(self, dirs, proj_dirs, proj_dirs_parents, outstring):
    """find 'dirs' that are present in 'proj_dirs_parents' but not in 'proj_dirs'"""
@ -133,27 +124,18 @@ the following meanings:

  def Execute(self, opt, args):
    all_projects = self.GetProjects(args)
-    counter = itertools.count()
+    counter = 0

    if opt.jobs == 1:
      for project in all_projects:
        state = project.PrintWorkTreeStatus(quiet=opt.quiet)
        if state == 'CLEAN':
-          next(counter)
+          counter += 1
    else:
-      sem = _threading.Semaphore(opt.jobs)
-      threads = []
-      for project in all_projects:
-        sem.acquire()
-
-        t = _threading.Thread(target=self._StatusHelper,
-                              args=(project, counter, sem, opt.quiet))
-        threads.append(t)
-        t.daemon = True
-        t.start()
-      for t in threads:
-        t.join()
-    if not opt.quiet and len(all_projects) == next(counter):
+      with multiprocessing.Pool(opt.jobs) as pool:
+        states = pool.map(functools.partial(self._StatusHelper, opt.quiet), all_projects)
+        counter += states.count('CLEAN')
+    if not opt.quiet and len(all_projects) == counter:
      print('nothing to commit (working directory clean)')

    if opt.orphans:
@ -183,7 +165,7 @@ the following meanings:
                          proj_dirs, proj_dirs_parents, outstring)

        if outstring:
-          output = StatusColoring(self.manifest.globalConfig)
+          output = StatusColoring(self.client.globalConfig)
          output.project('Objects not within a project (orphans)')
          output.nl()
          for entry in outstring:
--- a/subcmds/sync.py
+++ b/subcmds/sync.py
@ -138,11 +138,11 @@ if the manifest server specified in the manifest file already includes
 credentials.

 By default, all projects will be synced. The --fail-fast option can be used
-to halt syncing as soon as possible when the the first project fails to sync.
+to halt syncing as soon as possible when the first project fails to sync.

 The --force-sync option can be used to overwrite existing git
 directories if they have previously been linked to a different
-object direcotry. WARNING: This may cause data to be lost since
+object directory. WARNING: This may cause data to be lost since
 refs may be removed when overwriting.

 The --force-remove-dirty option can be used to remove previously used
@ -780,6 +780,7 @@ later is required to fix a server side protocol bug.
      start = time.time()
      success = mp.Sync_NetworkHalf(quiet=opt.quiet, verbose=opt.verbose,
                                    current_branch_only=opt.current_branch_only,
+                                    force_sync=opt.force_sync,
                                    tags=opt.tags,
                                    optimized_fetch=opt.optimized_fetch,
                                    retry_fetches=opt.retry_fetches,
--- a/subcmds/upload.py
+++ b/subcmds/upload.py
@ -21,10 +21,10 @@ import sys

 from command import InteractiveCommand
 from editor import Editor
-from error import HookError, UploadError
+from error import UploadError
 from git_command import GitCommand
 from git_refs import R_HEADS
-from project import RepoHook
+from hooks import RepoHook

 from pyversion import is_python3
 if not is_python3():
@ -205,33 +205,7 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
    p.add_option('--no-cert-checks',
                 dest='validate_certs', action='store_false', default=True,
                 help='Disable verifying ssl certs (unsafe).')
-
-    # Options relating to upload hook.  Note that verify and no-verify are NOT
-    # opposites of each other, which is why they store to different locations.
-    # We are using them to match 'git commit' syntax.
-    #
-    # Combinations:
-    # - no-verify=False, verify=False (DEFAULT):
-    #   If stdout is a tty, can prompt about running upload hooks if needed.
-    #   If user denies running hooks, the upload is cancelled.  If stdout is
-    #   not a tty and we would need to prompt about upload hooks, upload is
-    #   cancelled.
-    # - no-verify=False, verify=True:
-    #   Always run upload hooks with no prompt.
-    # - no-verify=True, verify=False:
-    #   Never run upload hooks, but upload anyway (AKA bypass hooks).
-    # - no-verify=True, verify=True:
-    #   Invalid
-    g = p.add_option_group('Upload hooks')
-    g.add_option('--no-verify',
-                 dest='bypass_hooks', action='store_true',
-                 help='Do not run the upload hook.')
-    g.add_option('--verify',
-                 dest='allow_all_hooks', action='store_true',
-                 help='Run the upload hook without prompting.')
-    g.add_option('--ignore-hooks',
-                 dest='ignore_hooks', action='store_true',
-                 help='Do not abort uploading if upload hooks fail.')
+    RepoHook.AddOptionGroup(p, 'pre-upload')

  def _SingleBranch(self, opt, branch, people):
    project = branch.project
@ -554,10 +528,10 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
          avail = [up_branch]
        else:
          avail = None
-          print('ERROR: Current branch (%s) not uploadable. '
-                'You may be able to type '
-                '"git branch --set-upstream-to m/master" to fix '
-                'your branch.' % str(cbr),
+          print('repo: error: Unable to upload branch "%s". '
+                'You might be able to fix the branch by running:\n'
+                '  git branch --set-upstream-to m/%s' %
+                (str(cbr), self.manifest.branch),
                file=sys.stderr)
      else:
        avail = project.GetUploadableBranches(branch)
@ -572,31 +546,15 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
              (branch,), file=sys.stderr)
      return 1

-    if not opt.bypass_hooks:
-      hook = RepoHook('pre-upload', self.manifest.repo_hooks_project,
-                      self.manifest.topdir,
-                      self.manifest.manifestProject.GetRemote('origin').url,
-                      abort_if_user_denies=True)
-      pending_proj_names = [project.name for (project, available) in pending]
-      pending_worktrees = [project.worktree for (project, available) in pending]
-      passed = True
-      try:
-        hook.Run(opt.allow_all_hooks, project_list=pending_proj_names,
-                 worktree_list=pending_worktrees)
-      except SystemExit:
-        passed = False
-        if not opt.ignore_hooks:
-          raise
-      except HookError as e:
-        passed = False
-        print("ERROR: %s" % str(e), file=sys.stderr)
-
-      if not passed:
-        if opt.ignore_hooks:
-          print('\nWARNING: pre-upload hooks failed, but uploading anyways.',
-                file=sys.stderr)
-        else:
-          return
+    pending_proj_names = [project.name for (project, available) in pending]
+    pending_worktrees = [project.worktree for (project, available) in pending]
+    hook = RepoHook.FromSubcmd(
+        hook_type='pre-upload', manifest=self.manifest,
+        opt=opt, abort_if_user_denies=True)
+    if not hook.Run(
+        project_list=pending_proj_names,
+        worktree_list=pending_worktrees):
+      return 1

    if opt.reviewers:
      reviewers = _SplitEmails(opt.reviewers)
--- a/tests/test_hooks.py
+++ b/tests/test_hooks.py
@ -0,0 +1,60 @@
+# -*- coding:utf-8 -*-
+#
+# Copyright (C) 2019 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unittests for the hooks.py module."""
+
+from __future__ import print_function
+
+import hooks
+import unittest
+
+class RepoHookShebang(unittest.TestCase):
+  """Check shebang parsing in RepoHook."""
+
+  def test_no_shebang(self):
+    """Lines w/out shebangs should be rejected."""
+    DATA = (
+        '',
+        '# -*- coding:utf-8 -*-\n',
+        '#\n# foo\n',
+        '# Bad shebang in script\n#!/foo\n'
+    )
+    for data in DATA:
+      self.assertIsNone(hooks.RepoHook._ExtractInterpFromShebang(data))
+
+  def test_direct_interp(self):
+    """Lines whose shebang points directly to the interpreter."""
+    DATA = (
+        ('#!/foo', '/foo'),
+        ('#! /foo', '/foo'),
+        ('#!/bin/foo ', '/bin/foo'),
+        ('#! /usr/foo ', '/usr/foo'),
+        ('#! /usr/foo -args', '/usr/foo'),
+    )
+    for shebang, interp in DATA:
+      self.assertEqual(hooks.RepoHook._ExtractInterpFromShebang(shebang),
+                       interp)
+
+  def test_env_interp(self):
+    """Lines whose shebang launches through `env`."""
+    DATA = (
+        ('#!/usr/bin/env foo', 'foo'),
+        ('#!/bin/env foo', 'foo'),
+        ('#! /bin/env /bin/foo ', '/bin/foo'),
+    )
+    for shebang, interp in DATA:
+      self.assertEqual(hooks.RepoHook._ExtractInterpFromShebang(shebang),
+                       interp)
--- a/tests/test_manifest_xml.py
+++ b/tests/test_manifest_xml.py
@ -19,6 +19,8 @@
 from __future__ import print_function

 import os
+import shutil
+import tempfile
 import unittest
 import xml.dom.minidom

@ -146,3 +148,133 @@ class ValueTests(unittest.TestCase):
    with self.assertRaises(error.ManifestParseError):
      node = self._get_node('<node a="xx"/>')
      manifest_xml.XmlInt(node, 'a')
+
+
+class XmlManifestTests(unittest.TestCase):
+  """Check manifest processing."""
+
+  def setUp(self):
+    self.tempdir = tempfile.mkdtemp(prefix='repo_tests')
+    self.repodir = os.path.join(self.tempdir, '.repo')
+    self.manifest_dir = os.path.join(self.repodir, 'manifests')
+    self.manifest_file = os.path.join(
+        self.repodir, manifest_xml.MANIFEST_FILE_NAME)
+    self.local_manifest_dir = os.path.join(
+        self.repodir, manifest_xml.LOCAL_MANIFESTS_DIR_NAME)
+    os.mkdir(self.repodir)
+    os.mkdir(self.manifest_dir)
+
+    # The manifest parsing really wants a git repo currently.
+    gitdir = os.path.join(self.repodir, 'manifests.git')
+    os.mkdir(gitdir)
+    with open(os.path.join(gitdir, 'config'), 'w') as fp:
+      fp.write("""[remote "origin"]
+        url = https://localhost:0/manifest
+""")
+
+  def tearDown(self):
+    shutil.rmtree(self.tempdir, ignore_errors=True)
+
+  def getXmlManifest(self, data):
+    """Helper to initialize a manifest for testing."""
+    with open(self.manifest_file, 'w') as fp:
+      fp.write(data)
+    return manifest_xml.XmlManifest(self.repodir, self.manifest_file)
+
+  def test_empty(self):
+    """Parse an 'empty' manifest file."""
+    manifest = self.getXmlManifest(
+        '<?xml version="1.0" encoding="UTF-8"?>'
+        '<manifest></manifest>')
+    self.assertEqual(manifest.remotes, {})
+    self.assertEqual(manifest.projects, [])
+
+  def test_link(self):
+    """Verify Link handling with new names."""
+    manifest = manifest_xml.XmlManifest(self.repodir, self.manifest_file)
+    with open(os.path.join(self.manifest_dir, 'foo.xml'), 'w') as fp:
+      fp.write('<manifest></manifest>')
+    manifest.Link('foo.xml')
+    with open(self.manifest_file) as fp:
+      self.assertIn('<include name="foo.xml" />', fp.read())
+
+  def test_toxml_empty(self):
+    """Verify the ToXml() helper."""
+    manifest = self.getXmlManifest(
+        '<?xml version="1.0" encoding="UTF-8"?>'
+        '<manifest></manifest>')
+    self.assertEqual(manifest.ToXml().toxml(), '<?xml version="1.0" ?><manifest/>')
+
+  def test_todict_empty(self):
+    """Verify the ToDict() helper."""
+    manifest = self.getXmlManifest(
+        '<?xml version="1.0" encoding="UTF-8"?>'
+        '<manifest></manifest>')
+    self.assertEqual(manifest.ToDict(), {})
+
+  def test_project_group(self):
+    """Check project group settings."""
+    manifest = self.getXmlManifest("""
+<manifest>
+  <remote name="test-remote" fetch="http://localhost" />
+  <default remote="test-remote" revision="refs/heads/main" />
+  <project name="test-name" path="test-path"/>
+  <project name="extras" path="path" groups="g1,g2,g1"/>
+</manifest>
+""")
+    self.assertEqual(len(manifest.projects), 2)
+    # Ordering isn't guaranteed.
+    result = {
+        manifest.projects[0].name: manifest.projects[0].groups,
+        manifest.projects[1].name: manifest.projects[1].groups,
+    }
+    project = manifest.projects[0]
+    self.assertCountEqual(
+        result['test-name'],
+        ['name:test-name', 'all', 'path:test-path'])
+    self.assertCountEqual(
+        result['extras'],
+        ['g1', 'g2', 'g1', 'name:extras', 'all', 'path:path'])
+
+  def test_include_levels(self):
+    root_m = os.path.join(self.manifest_dir, 'root.xml')
+    with open(root_m, 'w') as fp:
+      fp.write("""
+<manifest>
+  <remote name="test-remote" fetch="http://localhost" />
+  <default remote="test-remote" revision="refs/heads/main" />
+  <include name="level1.xml" groups="level1-group" />
+  <project name="root-name1" path="root-path1" />
+  <project name="root-name2" path="root-path2" groups="r2g1,r2g2" />
+</manifest>
+""")
+    with open(os.path.join(self.manifest_dir, 'level1.xml'), 'w') as fp:
+      fp.write("""
+<manifest>
+  <include name="level2.xml" groups="level2-group" />
+  <project name="level1-name1" path="level1-path1" />
+</manifest>
+""")
+    with open(os.path.join(self.manifest_dir, 'level2.xml'), 'w') as fp:
+      fp.write("""
+<manifest>
+  <project name="level2-name1" path="level2-path1" groups="l2g1,l2g2" />
+</manifest>
+""")
+    include_m = manifest_xml.XmlManifest(self.repodir, root_m)
+    for proj in include_m.projects:
+      if proj.name == 'root-name1':
+        # Check include group not set on root level proj.
+        self.assertNotIn('level1-group', proj.groups)
+      if proj.name == 'root-name2':
+        # Check root proj group not removed.
+        self.assertIn('r2g1', proj.groups)
+      if proj.name == 'level1-name1':
+        # Check level1 proj has inherited group level 1.
+        self.assertIn('level1-group', proj.groups)
+      if proj.name == 'level2-name1':
+        # Check level2 proj has inherited group levels 1 and 2.
+        self.assertIn('level1-group', proj.groups)
+        self.assertIn('level2-group', proj.groups)
+        # Check level2 proj group not removed.
+        self.assertIn('l2g1', proj.groups)
--- a/tests/test_project.py
+++ b/tests/test_project.py
@ -44,45 +44,6 @@ def TempGitTree():
    platform_utils.rmtree(tempdir)


-class RepoHookShebang(unittest.TestCase):
-  """Check shebang parsing in RepoHook."""
-
-  def test_no_shebang(self):
-    """Lines w/out shebangs should be rejected."""
-    DATA = (
-        '',
-        '# -*- coding:utf-8 -*-\n',
-        '#\n# foo\n',
-        '# Bad shebang in script\n#!/foo\n'
-    )
-    for data in DATA:
-      self.assertIsNone(project.RepoHook._ExtractInterpFromShebang(data))
-
-  def test_direct_interp(self):
-    """Lines whose shebang points directly to the interpreter."""
-    DATA = (
-        ('#!/foo', '/foo'),
-        ('#! /foo', '/foo'),
-        ('#!/bin/foo ', '/bin/foo'),
-        ('#! /usr/foo ', '/usr/foo'),
-        ('#! /usr/foo -args', '/usr/foo'),
-    )
-    for shebang, interp in DATA:
-      self.assertEqual(project.RepoHook._ExtractInterpFromShebang(shebang),
-                       interp)
-
-  def test_env_interp(self):
-    """Lines whose shebang launches through `env`."""
-    DATA = (
-        ('#!/usr/bin/env foo', 'foo'),
-        ('#!/bin/env foo', 'foo'),
-        ('#! /bin/env /bin/foo ', '/bin/foo'),
-    )
-    for shebang, interp in DATA:
-      self.assertEqual(project.RepoHook._ExtractInterpFromShebang(shebang),
-                       interp)
-
-
 class FakeProject(object):
  """A fake for Project for basic functionality."""

@ -116,7 +77,7 @@ class ReviewableBranchTests(unittest.TestCase):

      # Start off with the normal details.
      rb = project.ReviewableBranch(
-          fakeproj, fakeproj.config.GetBranch('work'), 'master')
+          fakeproj, fakeproj.config.GetBranch('work'), 'main')
      self.assertEqual('work', rb.name)
      self.assertEqual(1, len(rb.commits))
      self.assertIn('Del file', rb.commits[0])
@ -129,9 +90,9 @@ class ReviewableBranchTests(unittest.TestCase):
      self.assertTrue(rb.date)

      # Now delete the tracking branch!
-      fakeproj.work_git.branch('-D', 'master')
+      fakeproj.work_git.branch('-D', 'main')
      rb = project.ReviewableBranch(
-          fakeproj, fakeproj.config.GetBranch('work'), 'master')
+          fakeproj, fakeproj.config.GetBranch('work'), 'main')
      self.assertEqual(0, len(rb.commits))
      self.assertFalse(rb.base_exists)
      # Hard to assert anything useful about this.
--- a/tests/test_subcmds.py
+++ b/tests/test_subcmds.py
@ -0,0 +1,43 @@
+# Copyright (C) 2020 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unittests for the subcmds module (mostly __init__.py than subcommands)."""
+
+import unittest
+
+import subcmds
+
+
+class AllCommands(unittest.TestCase):
+  """Check registered all_commands."""
+
+  def test_required_basic(self):
+    """Basic checking of registered commands."""
+    # NB: We don't test all subcommands as we want to avoid "change detection"
+    # tests, so we just look for the most common/important ones here that are
+    # unlikely to ever change.
+    for cmd in {'cherry-pick', 'help', 'init', 'start', 'sync', 'upload'}:
+      self.assertIn(cmd, subcmds.all_commands)
+
+  def test_naming(self):
+    """Verify we don't add things that we shouldn't."""
+    for cmd in subcmds.all_commands:
+      # Reject filename suffixes like "help.py".
+      self.assertNotIn('.', cmd)
+
+      # Make sure all '_' were converted to '-'.
+      self.assertNotIn('_', cmd)
+
+      # Reject internal python paths like "__init__".
+      self.assertFalse(cmd.startswith('__'))
--- a/tests/test_subcmds_init.py
+++ b/tests/test_subcmds_init.py
@ -0,0 +1,49 @@
+# Copyright (C) 2020 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unittests for the subcmds/init.py module."""
+
+import unittest
+
+from subcmds import init
+
+
+class InitCommand(unittest.TestCase):
+  """Check registered all_commands."""
+
+  def setUp(self):
+    self.cmd = init.Init()
+
+  def test_cli_parser_good(self):
+    """Check valid command line options."""
+    ARGV = (
+        [],
+    )
+    for argv in ARGV:
+      opts, args = self.cmd.OptionParser.parse_args(argv)
+      self.cmd.ValidateOptions(opts, args)
+
+  def test_cli_parser_bad(self):
+    """Check invalid command line options."""
+    ARGV = (
+        # Too many arguments.
+        ['asdf'],
+
+        # Conflicting options.
+        ['--mirror', '--archive'],
+    )
+    for argv in ARGV:
+      opts, args = self.cmd.OptionParser.parse_args(argv)
+      with self.assertRaises(SystemExit):
+        self.cmd.ValidateOptions(opts, args)
--- a/tests/test_wrapper.py
+++ b/tests/test_wrapper.py
@ -402,8 +402,8 @@ class ResolveRepoRev(GitCheckoutTestCase):
    self.assertEqual('refs/heads/stable', rrev)
    self.assertEqual(self.REV_LIST[1], lrev)

-    rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, 'master')
-    self.assertEqual('refs/heads/master', rrev)
+    rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, 'main')
+    self.assertEqual('refs/heads/main', rrev)
    self.assertEqual(self.REV_LIST[0], lrev)

  def test_tag_name(self):
--- a/tox.ini
+++ b/tox.ini
@ -15,11 +15,10 @@
 # https://tox.readthedocs.io/

 [tox]
-envlist = py27, py36, py37, py38
+envlist = py36, py37, py38

 [gh-actions]
 python =
-    2.7: py27
    3.6: py36
    3.7: py37
    3.8: py38
@ -31,8 +30,3 @@ setenv =
    GIT_AUTHOR_NAME = Repo test author
    GIT_COMMITTER_NAME = Repo test committer
    EMAIL = repo@gerrit.nodomain
-
-[testenv:py27]
-deps =
-    mock
-    pytest
Author	SHA1	Message	Date
Fredrik de Groot	352c93b680	manifest: add support for groups in include Attrib groups can now be added to manifest include, thus all projects in an included manifest file can easily be tagged with a group without modifying all projects in that manifest file. Include groups will add and recurse, meaning included manifest projects will carry all parent includes. Intentionally, no support added for group remove, to keep complexity down. Group handling for projects is untouched, meaning a group set on a project will still append to whatever was or was not inherited in parent manifest includes, resulting in union of groups inherited and set for the project itself. Test: manual multi-level manifest include structure, in serial and parallel, with different groups set on init Test: added unit tests to cover the inheritance Change-Id: Id2229aa6fd78d355ba598cc15c701b2ee71e5c6f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/283587 Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com> Reviewed-by: Mike Frysinger <vapier@google.com>	2020-11-26 09:13:14 +00:00
Remy Bohmer	7f7acfe9fd	Concentrate the RepoHook knowledge in the RepoHook class The knowledge about running hooks and all its exception handling is scattered over multiple files. This makes the code harder to read, but also it requires duplication of logic in case other RepoHooks are added to different commands. This refactoring also creates uniform behavior of the hooks across multiple commands and it guarantees the re-use of the same arguments on all of them. Signed-off-by: Remy Bohmer <github@bohmer.net> Change-Id: Ia4d90eab429e4af00943306e89faec8db35ba29d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/277562 Tested-by: Remy Bohmer <oss@bohmer.net> Reviewed-by: Mike Frysinger <vapier@google.com>	2020-11-23 09:59:16 +00:00
Remy Bohmer	169b0218b3	Fix --reference option under Windows When intializing a new repo with the --reference option on Windows 10 the objects/info/alternates in each git repository is created with Windows line endings (\r\n), leading to the following error: error: object directory C:/<PATH_TO_MIRROR>/<REPO_NAME>.git/objects? does not exist; check .git/objects/info/alternates This can be fixed by simply using unix line endings on both Windows and unix platforms. Reported-by: Francisco Javier Alvarez Garcia <javier.alvarez.garcia.17@gmail.com> Follow-up-from: I268fe029ede68802c21037b0f2ae8a95afb85e48 Bug: https://crbug.com/gerrit/13208 Change-Id: I6da60c4ca957778b3c42ab6b9ad85c40483f0042 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289431 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Remy Bohmer <oss@bohmer.net>	2020-11-23 09:17:32 +00:00
Remy Bohmer	44bc9643ed	Always use Unix EOL for worktree .git and gitdir files Worktree .git and gitdir reference files are written by Git with Unix line ending, even on Windows & macOS. The conversion to relative paths makes these files end with DOS line endings in Windows. The Git integration in Visual Studio 2019 cannot deal with these DOS line endings and considers these worktrees invalid. Signed-off-by: Remy Bohmer <github@bohmer.net> Change-Id: I088cfd994f3cc31db4e0ca7791fa0a4ee3ac222f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289310 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Remy Bohmer <linux@bohmer.net>	2020-11-20 20:53:43 +00:00
Mike Frysinger	d7f8683daf	project: do not update local published/ refs in dryrun mode Bug: https://crbug.com/gerrit/13087 Change-Id: I197e6d6d07c7d325ac294b597d42e895f77c737f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289182 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-11-20 04:08:19 +00:00
Mike Frysinger	8c1e9cbef1	manifest_xml: refactor manifest parsing from client management We conflate the manifest & parsing logic with the management of the repo client checkout in a single class. This makes testing just one part (the manifest parsing) hard as it requires a full checkout too. Start splitting the two apart into separate classes to make it easy to reason about & test. Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-11-18 19:10:57 +00:00
Mike Frysinger	a488af5ea5	main: require Python 3 now We've been warning about this for more than 6 months (with public announcements even older). Lets make it a failure now to see who hasn't upgraded yet. Bug: https://crbug.com/gerrit/10418 Change-Id: Iec3e2cbf87de434021921616683d360bc4fef77a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280796 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-11-17 15:04:20 +00:00
Mike Frysinger	e283b95cf2	tests: use new main branch Now that we clone "main" by default, use that for our local test. Change-Id: Ib8420074bdfabfcb9d5252a3a0ecd3d852ca36e8 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288422 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-11-17 04:29:09 +00:00
Erwan Yvin	dc5c4d1d11	sync: respect --force-sync when fetching manifest project updates The --force-sync option was being passed down for all updates except for the manifest project, so add that there too. Bug: https://crbug.com/gerrit/11034 Change-Id: I33818b652f828c6b847dbc70f1fedfac5ac17bbe Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228146 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>	2020-11-17 03:06:06 +00:00
Mike Frysinger	23411d3f9c	manifest: add a --json output option Sometimes parsing JSON is easier than parsing XML, especially when the XML format is limited (which ours is). Add a --json option to the manifest command to quickly emit that form. Bug: https://crbug.com/gerrit/11743 Change-Id: Ia2bb254a78ae2b70a851638b4545fcafe8c1a76b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280436 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-11-17 01:38:00 +00:00
Mike Frysinger	160748f828	upload: improve tip for fixing upload remote Instead of assuming the repo client is tracking the "master" branch of the manifest repo, use the existing info we have to display the right info to the user. Bug: https://crbug.com/gerrit/13339 Change-Id: I8b265f4b2e075fdc41909b1f3dff9aee87384353 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/287279 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-11-16 23:13:02 +00:00
Mike Frysinger	6e89c965f4	switch to "main" branch for development We're migrating from "master" to "main" as the default development branch. This only affects repo itself, not manifests. Change-Id: I27489dd721c9a467a1c43736808cb3b3c1365433 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288082 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-11-16 05:07:33 +00:00
Miguel Gaio	1f20776dbb	manifest_xml: correct project revisionId for extend-project Using sha1 manifest, project's revisionId is initialized first by the manifest. An update of a projet revision by extend-project node does not apply to the revisionId which is therefore kept to the initial value. Resets revisionId value when revision is updated by an extend-project node. Change-Id: I873af283890cebaeaabde966f04b125642af929f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/275715 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Miguel Gaio <miguel.gaio@renault.com>	2020-11-12 09:00:08 +00:00
Remy Bohmer	16c1328fec	Move RepoHook class from project.py file to dedicated file The project.py file is huge and contains multiple classes. By moving it to seperate class files the code becomes more readable and maintainable. Signed-off-by: Remy Bohmer <github@bohmer.net> Change-Id: Ida9d99d31751d627ae1ea0373418080696d2e14b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/281293 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Remy Bohmer <linux@bohmer.net>	2020-11-03 22:08:08 +00:00
Jonathan Nieder	6248e0fd1d	launcher: simplify .repo search ceiling check In the .repo discovery loop while curdir != '/' and curdir != olddir: ... break if we found .repo ... olddir = curdir curdir = os.path.dirname(curdir) the "while" condition is meant to avoid searching forever if we do not find .repo before reaching the top-level directory of the filesystem. For that purpose, the first half of the condition is redundant; once we reach "/", the parent directory will be "/" again and the curdir != olddir check would suffice to terminate the search. Simplify by removing the redundant first half of the check. Noticed by code inspection. The first half of the check was retained when introducing the second half in `df14a70c` ("Make path references OS independent", 2011-01-09), in an excess of caution. This also improves consistency a little: if I start with curdir = '/home/me', then with the redundant check in place we search /home/me /home before hitting / and giving up. On Windows, if I start with 'c:/users/me', then we search c:/users/me c:/users c:/ before hitting a repetition and giving up. Fortunately it is not common for people to set up repo clients at the top level of filesystems, but consistently following the latter behavior should make debugging a little easier in case it comes up. Link: https://gerrit-review.googlesource.com/id/Ib9e830e3b9adfb1c4e56f3bcfba4746c401fb84f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/286002 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Jonathan Nieder <jrn@google.com>	2020-11-03 20:27:19 +00:00
Mike Frysinger	50a81de2bc	init: use the remote default manifest branch Instead of hardcoding "master" as our default, use the remote server's default branch instead. For most people, this should be the same as "master" already. For projects moving to "main", it means we'll use the new name automatically rather than forcing people to use -b main. For repositories that never set up a default HEAD, we should still use the historical "master" default. Bug: https://crbug.com/gerrit/13339 Change-Id: I4117c81a760c9495f98dbb1111a3e6c127f45eba Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280799 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-09-09 05:46:07 +00:00
Kimiyuki Onaka	0501b29e7a	status: Use multiprocessing for `repo status -j<num>` instead of threading This change increases the speed of the command with parallelization with processes. The parallelization with threads doesn't work well, and increasing the number of jobs to many (8 threads ~) didn't increase the speed. Possibly, the global interpreter lock of Python affects. Bug: https://crbug.com/gerrit/12389 Change-Id: Icbe5df8ba037dd91422b96f4e43708068d7be924 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279936 Tested-by: Kimiyuki Onaka <kimiyuki@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>	2020-09-09 03:52:24 +00:00
Mike Frysinger	4e1fc1013c	manifest: drop support for local_manifest.xml We deprecated this 8 years ago. Time to drop it to simplify the code as it'll help with refactoring in this module to not migrate it. Change-Id: I2deae5496d1f66a4491408fcdc95cd527062f8b6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280798 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Michael Mortensen <mmortensen@google.com>	2020-09-08 17:00:06 +00:00
Mike Frysinger	4b325813fc	stop testing Python 2.7 A recent change broke `repo version` on Python 2.7. Rather than fix it, lets drop Python 2.7 support so it can slowly rot. Bug: https://crbug.com/gerrit/10418 Change-Id: I5c6e3d18e4a193b0a978062c23f7cea392e95d0f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259155 Reviewed-by: David Pursehouse <david.pursehouse@gmail.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-09-06 17:53:47 +00:00
Mike Frysinger	0578ebf61a	init: reject unknown args If you pass args to `repo init` when first creating a checkout, the repo launcher throws an error. But the init subcommand that runs in an existing checkout silently ignores them. Throw a proper error. Change-Id: I433bfcc73902d25f6b6a2974e77f6a977a75ed16 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279696 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-09-02 07:53:16 +00:00
Adrien Bioteau	65f51ad29b	Fix Git base version for worktreeconfig extension worktreeconfig extension only appears with version Git 2.20.0 Change-Id: I3ea8b7d9f8a1f7953e536edd77b09cbc4f8f3158 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/276700 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Adrien Bioteau <adrien.bioteau@gmail.com>	2020-07-30 20:46:11 +00:00
Mike Frysinger	80944b538d	upload: exit non-zero when preupload hooks fail Bug: https://crbug.com/gerrit/13159 Change-Id: Id140b619242c841223c6bc5d4aa0c37a7ce0219d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/276294 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>	2020-07-25 08:31:52 +00:00
Mike Frysinger	89f3ae5ae6	release-process: document schedule (including freezes) publicly Change-Id: Ic037b54630017740d7859292b32b8c57f4af7854 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/274772 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>	2020-07-23 08:07:38 +00:00
Roger Shimizu	ac29ac397f	subcmds/sync.py: Fix typo in help Change-Id: I70b63477241284249e395b8b0a220cb6f44f836b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/270183 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@digital.ai> Tested-by: David Pursehouse <dpursehouse@digital.ai>	2020-06-06 23:46:00 +00:00
Mike Frysinger	cebf227026	manifest: normalize name & path when constructing fs paths If the manifest uses a trailing slash on the name attribute, repo will construct bad internal filesystem paths which confuses tools later on. For example, this manifest entry: <project name="aosp/platform/system/libhidl/" ... will cause repo to use paths like: .repo/project-objects/aosp/platform/system/libhidl/.git/ when it really should be using: .repo/project-objects/aosp/platform/system/libhidl.git Apply the normalization when we construct the local filesystem paths as we cannot guarantee that the remote URL constructed from these will behave the same. A server might really want: https://example.com/aosp/platform/system/libhidl/ and would throw an error if we instead tried to fetch: https://example.com/aosp/platform/system/libhidl Unfortunately, any existing repo client checkouts that use such a manifest will hit a one-time sync error as the internal git location has changed. I'm not sure there's a way to cleanly migrate that. Bug: https://crbug.com/1086043 Change-Id: I30bea0ffd23e478de89a035f408055e48a102658 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268742 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@digital.ai>	2020-05-26 05:15:58 +00:00
Mike Frysinger	7ae210a15b	sync: fix duplicate word in description Bug: https://crbug.com/gerrit/12814 Change-Id: Id722eec9a59dded588f13bc605ce2c94b4047265 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268739 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@digital.ai>	2020-05-24 23:51:28 +00:00