event_log: turn id generation from a generator to a func call

Running lots of sync processes in parallel can hit the failure:
Fetching projects:  23% (124/523)Exception in thread Thread-201:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/src/repo/subcmds/sync.py", line 278, in _FetchProjectList
    success = self._FetchHelper(opt, project, *args, **kwargs)
  File "/usr/local/src/repo/subcmds/sync.py", line 357, in _FetchHelper
    start, finish, success)
  File "/usr/local/src/repo/event_log.py", line 104, in AddSync
    event = self.Add(project.relpath, task_name, start, finish, success)
  File "/usr/local/src/repo/event_log.py", line 74, in Add
    'id': (kind, next(self._next_id)),
ValueError: generator already executing

It looks like, while we lock the multiprocessing value correctly, the
generator that wraps the value isn't parallel safe.  Since we don't
have a way of doing that (as it's part of the language), turn it into
a plain function call instead.

Bug: https://crbug.com/gerrit/10293
Change-Id: I0db03601986ca0370a1699bab32adb03e7b2910a

.flake8

@@ -0,0 +1,3 @@
+[flake8]
+max-line-length=80
+ignore=E111,E114,E402

.gitattributes

@@ -0,0 +1,4 @@
+# Prevent /bin/sh scripts from being clobbered by autocrlf=true
+git_ssh text eol=lf
+repo text eol=lf
+hooks/* text eol=lf

.gitignore

@@ -1 +1,3 @@
 *.pyc
+.repopickle_*
+/repoc

.mailmap

@@ -0,0 +1,12 @@
+Anthony Newnam <anthony.newnam@garmin.com>    Anthony <anthony@bnovc.com>
+He Ping <tdihp@hotmail.com>                   heping <tdihp@hotmail.com>
+Hu Xiuyun <xiuyun.hu@hisilicon.com>           Hu xiuyun <xiuyun.hu@hisilicon.com>
+Hu Xiuyun <xiuyun.hu@hisilicon.com>           Hu Xiuyun <clouds08@qq.com>
+Jelly Chen <chenguodong@huawei.com>           chenguodong <chenguodong@huawei.com>
+Jia Bi <bijia@xiaomi.com>                     bijia <bijia@xiaomi.com>
+JoonCheol Park <jooncheol@gmail.com>          Jooncheol Park <jooncheol@gmail.com>
+Sergii Pylypenko <x.pelya.x@gmail.com>        pelya <x.pelya.x@gmail.com>
+Shawn Pearce <sop@google.com>                 Shawn O. Pearce <sop@google.com>
+Ulrik Sjölin <ulrik.sjolin@sonyericsson.com>  Ulrik Sjolin <ulrik.sjolin@gmail.com>
+Ulrik Sjölin <ulrik.sjolin@sonyericsson.com>  Ulrik Sjolin <ulrik.sjolin@sonyericsson.com>
+Ulrik Sjölin <ulrik.sjolin@sonyericsson.com>  Ulrik Sjölin <ulrik.sjolin@sonyericsson.com>

.project

@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<projectDescription>
+	<name>git-repo</name>
+	<comment></comment>
+	<projects>
+	</projects>
+	<buildSpec>
+		<buildCommand>
+			<name>org.python.pydev.PyDevBuilder</name>
+			<arguments>
+			</arguments>
+		</buildCommand>
+	</buildSpec>
+	<natures>
+		<nature>org.python.pydev.pythonNature</nature>
+	</natures>
+</projectDescription>

.pydevproject

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<?eclipse-pydev version="1.0"?>
+
+<pydev_project>
+<pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
+<path>/git-repo</path>
+</pydev_pathproperty>
+<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python 2.7</pydev_property>
+<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>
+</pydev_project>

README.md

@@ -0,0 +1,16 @@
+# repo
+
+Repo is a tool built on top of Git.  Repo helps manage many Git repositories,
+does the uploads to revision control systems, and automates parts of the
+development workflow.  Repo is not meant to replace Git, only to make it
+easier to work with Git.  The repo command is an executable Python script
+that you can put anywhere in your path.
+
+* Homepage: https://gerrit.googlesource.com/git-repo/
+* Bug reports: https://bugs.chromium.org/p/gerrit/issues/list?q=component:repo
+* Source: https://gerrit.googlesource.com/git-repo/
+* Overview: https://source.android.com/source/developing.html
+* Docs: https://source.android.com/source/using-repo.html
+* [repo Manifest Format](./docs/manifest-format.md)
+* [repo Hooks](./docs/repo-hooks.md)
+* [Submitting patches](./SUBMITTING_PATCHES.md)

SUBMITTING_PATCHES.md

@@ -0,0 +1,135 @@
+# Short Version
+
+ - Make small logical changes.
+ - Provide a meaningful commit message.
+ - Check for coding errors and style nits with pyflakes and flake8
+ - Make sure all code is under the Apache License, 2.0.
+ - Publish your changes for review.
+ - Make corrections if requested.
+ - Verify your changes on gerrit so they can be submitted.
+
+   `git push https://gerrit-review.googlesource.com/git-repo HEAD:refs/for/master`
+
+
+# Long Version
+
+I wanted a file describing how to submit patches for repo,
+so I started with the one found in the core Git distribution
+(Documentation/SubmittingPatches), which itself was based on the
+patch submission guidelines for the Linux kernel.
+
+However there are some differences, so please review and familiarize
+yourself with the following relevant bits.
+
+
+## Make separate commits for logically separate changes.
+
+Unless your patch is really trivial, you should not be sending
+out a patch that was generated between your working tree and your
+commit head.  Instead, always make a commit with complete commit
+message and generate a series of patches from your repository.
+It is a good discipline.
+
+Describe the technical detail of the change(s).
+
+If your description starts to get too long, that's a sign that you
+probably need to split up your commit to finer grained pieces.
+
+
+## Check for coding errors and style nits with pyflakes and flake8
+
+### Coding errors
+
+Run `pyflakes` on changed modules:
+
+    pyflakes file.py
+
+Ideally there should be no new errors or warnings introduced.
+
+### Style violations
+
+Run `flake8` on changes modules:
+
+    flake8 file.py
+
+Note that repo generally follows [Google's python style guide]
+(https://google.github.io/styleguide/pyguide.html) rather than [PEP 8]
+(https://www.python.org/dev/peps/pep-0008/), so it's possible that
+the output of `flake8` will be quite noisy. It's not mandatory to
+avoid all warnings, but at least the maximum line length should be
+followed.
+
+If there are many occurrences of the same warning that cannot be
+avoided without going against the Google style guide, these may be
+suppressed in the included `.flake8` file.
+
+## Check the license
+
+repo is licensed under the Apache License, 2.0.
+
+Because of this licensing model *every* file within the project
+*must* list the license that covers it in the header of the file.
+Any new contributions to an existing file *must* be submitted under
+the current license of that file.  Any new files *must* clearly
+indicate which license they are provided under in the file header.
+
+Please verify that you are legally allowed and willing to submit your
+changes under the license covering each file *prior* to submitting
+your patch.  It is virtually impossible to remove a patch once it
+has been applied and pushed out.
+
+
+## Sending your patches.
+
+Do not email your patches to anyone.
+
+Instead, login to the Gerrit Code Review tool at:
+
+  https://gerrit-review.googlesource.com/
+
+Ensure you have completed one of the necessary contributor
+agreements, providing documentation to the project maintainers that
+they have right to redistribute your work under the Apache License:
+
+  https://gerrit-review.googlesource.com/#/settings/agreements
+
+Ensure you have obtained an HTTP password to authenticate:
+
+  https://gerrit-review.googlesource.com/new-password
+
+Ensure that you have the local commit hook installed to automatically
+add a ChangeId to your commits:
+
+    curl -Lo `git rev-parse --git-dir`/hooks/commit-msg https://gerrit-review.googlesource.com/tools/hooks/commit-msg
+    chmod +x `git rev-parse --git-dir`/hooks/commit-msg
+
+If you have already committed your changes you will need to amend the commit
+to get the ChangeId added.
+
+    git commit --amend
+
+Push your patches over HTTPS to the review server, possibly through
+a remembered remote to make this easier in the future:
+
+    git config remote.review.url https://gerrit-review.googlesource.com/git-repo
+    git config remote.review.push HEAD:refs/for/master
+
+    git push review
+
+You will be automatically emailed a copy of your commits, and any
+comments made by the project maintainers.
+
+
+## Make changes if requested
+
+The project maintainer who reviews your changes might request changes to your
+commit. If you make the requested changes you will need to amend your commit
+and push it to the review server again.
+
+
+## Verify your changes on gerrit
+
+After you receive a Code-Review+2 from the maintainer, select the Verified
+button on the gerrit page for the change. This verifies that you have tested
+your changes and notifies the maintainer that they are ready to be submitted.
+The maintainer will then submit your changes to the repository.

codereview/__init__.py

@@ -1 +0,0 @@
-__version__ = 'v1.0-69-gd1f8508c'

codereview/need_retry_pb2.py

@@ -1,32 +0,0 @@
-#!/usr/bin/python2.4
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-
-from froofle.protobuf import descriptor
-from froofle.protobuf import message
-from froofle.protobuf import reflection
-from froofle.protobuf import service
-from froofle.protobuf import service_reflection
-from froofle.protobuf import descriptor_pb2
-
-
-
-_RETRYREQUESTLATERRESPONSE = descriptor.Descriptor(
-  name='RetryRequestLaterResponse',
-  full_name='codereview.RetryRequestLaterResponse',
-  filename='need_retry.proto',
-  containing_type=None,
-  fields=[
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-
-class RetryRequestLaterResponse(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _RETRYREQUESTLATERRESPONSE
-

codereview/proto_client.py

@@ -1,380 +0,0 @@
-# Copyright 2007, 2008 Google Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import base64
-import cookielib
-import getpass
-import logging
-import md5
-import os
-import random
-import socket
-import sys
-import time
-import urllib
-import urllib2
-import urlparse
-
-from froofle.protobuf.service import RpcChannel
-from froofle.protobuf.service import RpcController
-from need_retry_pb2 import RetryRequestLaterResponse;
-
-_cookie_jars = {}
-
-def _open_jar(path):
-  auth = False
-
-  if path is None:
-    c = cookielib.CookieJar()
-  else:
-    c = _cookie_jars.get(path)
-    if c is None:
-      c = cookielib.MozillaCookieJar(path)
-
-      if os.path.exists(path):
-        try:
-          c.load()
-          auth = True
-        except (cookielib.LoadError, IOError):
-          pass
-
-        if auth:
-          print >>sys.stderr, \
-                'Loaded authentication cookies from %s' \
-                % path
-      else:
-        os.close(os.open(path, os.O_CREAT, 0600))
-      os.chmod(path, 0600)
-      _cookie_jars[path] = c
-    else:
-      auth = True
-  return c, auth
-
-
-class ClientLoginError(urllib2.HTTPError):
-  """Raised to indicate an error authenticating with ClientLogin."""
-
-  def __init__(self, url, code, msg, headers, args):
-    urllib2.HTTPError.__init__(self, url, code, msg, headers, None)
-    self.args = args
-    self.reason = args["Error"]
-
-
-class Proxy(object):
-  class _ResultHolder(object):
-    def __call__(self, result):
-      self._result = result
-
-  class _RemoteController(RpcController):
-    def Reset(self):
-      pass
-  
-    def Failed(self):
-      pass
-  
-    def ErrorText(self):
-      pass
-  
-    def StartCancel(self):
-      pass
-  
-    def SetFailed(self, reason):
-      raise RuntimeError, reason
-  
-    def IsCancelled(self):
-      pass
-  
-    def NotifyOnCancel(self, callback):
-      pass
-  
-  def __init__(self, stub):
-    self._stub = stub
-
-  def __getattr__(self, key):
-    method = getattr(self._stub, key)
-
-    def call(request):
-      done = self._ResultHolder()
-      method(self._RemoteController(), request, done)
-      return done._result
-
-    return call
-
-
-class HttpRpc(RpcChannel):
-  """Simple protobuf over HTTP POST implementation."""
-
-  def __init__(self, host, auth_function,
-               host_override=None,
-               extra_headers={},
-               cookie_file=None):
-    """Creates a new HttpRpc.
-
-    Args:
-      host: The host to send requests to.
-      auth_function: A function that takes no arguments and returns an
-        (email, password) tuple when called. Will be called if authentication
-        is required.
-      host_override: The host header to send to the server (defaults to host).
-      extra_headers: A dict of extra headers to append to every request.
-      cookie_file: If not None, name of the file in ~/ to save the
-        cookie jar into.  Applications are encouraged to set this to
-        '.$appname_cookies' or some otherwise unique name.
-    """
-    self.host = host.lower()
-    self.host_override = host_override
-    self.auth_function = auth_function
-    self.authenticated = False
-    self.extra_headers = extra_headers
-    self.xsrf_token = None
-    if cookie_file is None:
-      self.cookie_file = None
-    else:
-      self.cookie_file = os.path.expanduser("~/%s" % cookie_file)
-    self.opener = self._GetOpener()
-    if self.host_override:
-      logging.info("Server: %s; Host: %s", self.host, self.host_override)
-    else:
-      logging.info("Server: %s", self.host)
-
-  def CallMethod(self, method, controller, request, response_type, done):
-    pat = "application/x-google-protobuf; name=%s"
-
-    url = "/proto/%s/%s" % (method.containing_service.name, method.name)
-    reqbin = request.SerializeToString()
-    reqtyp = pat % request.DESCRIPTOR.full_name
-    reqmd5 = base64.b64encode(md5.new(reqbin).digest())
-
-    start = time.time()
-    while True:
-      t, b = self._Send(url, reqbin, reqtyp, reqmd5)
-      if t == (pat % RetryRequestLaterResponse.DESCRIPTOR.full_name):
-        if time.time() >= (start + 1800):
-          controller.SetFailed("timeout")
-          return
-        s = random.uniform(0.250, 2.000)
-        print "Busy, retrying in %.3f seconds ..." % s
-        time.sleep(s)
-        continue
-
-      if t == (pat % response_type.DESCRIPTOR.full_name):
-        response = response_type()
-        response.ParseFromString(b)
-        done(response)
-      else:
-        controller.SetFailed("Unexpected %s response" % t)
-      break
-
-  def _CreateRequest(self, url, data=None):
-    """Creates a new urllib request."""
-    logging.debug("Creating request for: '%s' with payload:\n%s", url, data)
-    req = urllib2.Request(url, data=data)
-    if self.host_override:
-      req.add_header("Host", self.host_override)
-    for key, value in self.extra_headers.iteritems():
-      req.add_header(key, value)
-    return req
-
-  def _GetAuthToken(self, email, password):
-    """Uses ClientLogin to authenticate the user, returning an auth token.
-
-    Args:
-      email:    The user's email address
-      password: The user's password
-
-    Raises:
-      ClientLoginError: If there was an error authenticating with ClientLogin.
-      HTTPError: If there was some other form of HTTP error.
-
-    Returns:
-      The authentication token returned by ClientLogin.
-    """
-    account_type = 'GOOGLE'
-    if self.host.endswith('.google.com'):
-      account_type = 'HOSTED'
-
-    req = self._CreateRequest(
-        url="https://www.google.com/accounts/ClientLogin",
-        data=urllib.urlencode({
-            "Email": email,
-            "Passwd": password,
-            "service": "ah",
-            "source": "gerrit-codereview-client",
-            "accountType": account_type,
-        })
-    )
-    try:
-      response = self.opener.open(req)
-      response_body = response.read()
-      response_dict = dict(x.split("=")
-                           for x in response_body.split("\n") if x)
-      return response_dict["Auth"]
-    except urllib2.HTTPError, e:
-      if e.code == 403:
-        body = e.read()
-        response_dict = dict(x.split("=", 1) for x in body.split("\n") if x)
-        raise ClientLoginError(req.get_full_url(), e.code, e.msg,
-                               e.headers, response_dict)
-      else:
-        raise
-
-  def _GetAuthCookie(self, auth_token):
-    """Fetches authentication cookies for an authentication token.
-
-    Args:
-      auth_token: The authentication token returned by ClientLogin.
-
-    Raises:
-      HTTPError: If there was an error fetching the authentication cookies.
-    """
-    # This is a dummy value to allow us to identify when we're successful.
-    continue_location = "http://localhost/"
-    args = {"continue": continue_location, "auth": auth_token}
-    req = self._CreateRequest("http://%s/_ah/login?%s" %
-                              (self.host, urllib.urlencode(args)))
-    try:
-      response = self.opener.open(req)
-    except urllib2.HTTPError, e:
-      response = e
-    if (response.code != 302 or
-        response.info()["location"] != continue_location):
-      raise urllib2.HTTPError(req.get_full_url(), response.code, response.msg,
-                              response.headers, response.fp)
-
-  def _GetXsrfToken(self):
-    """Fetches /proto/_token for use in X-XSRF-Token HTTP header.
-
-    Raises:
-      HTTPError: If there was an error fetching a new token.
-    """
-    tries = 0
-    while True:
-      url = "http://%s/proto/_token" % self.host
-      req = self._CreateRequest(url)
-      try:
-        response = self.opener.open(req)
-        self.xsrf_token = response.read()
-        return
-      except urllib2.HTTPError, e:
-        if tries > 3:
-          raise
-        elif e.code == 401:
-          self._Authenticate()
-        else:
-          raise
-
-  def _Authenticate(self):
-    """Authenticates the user.
-
-    The authentication process works as follows:
-     1) We get a username and password from the user
-     2) We use ClientLogin to obtain an AUTH token for the user
-        (see http://code.google.com/apis/accounts/AuthForInstalledApps.html).
-     3) We pass the auth token to /_ah/login on the server to obtain an
-        authentication cookie. If login was successful, it tries to redirect
-        us to the URL we provided.
-
-    If we attempt to access the upload API without first obtaining an
-    authentication cookie, it returns a 401 response and directs us to
-    authenticate ourselves with ClientLogin.
-    """
-    attempts = 0
-    while True:
-      attempts += 1
-      try:
-        cred = self.auth_function()
-        auth_token = self._GetAuthToken(cred[0], cred[1])
-      except ClientLoginError:
-        if attempts < 3:
-          continue
-        raise
-      self._GetAuthCookie(auth_token)
-      self.authenticated = True
-      if self.cookie_file is not None:
-        print >>sys.stderr, \
-              'Saving authentication cookies to %s' \
-              % self.cookie_file
-        self.cookie_jar.save()
-      return
-
-  def _Send(self, request_path, payload, content_type, content_md5):
-    """Sends an RPC and returns the response.
-
-    Args:
-      request_path: The path to send the request to, eg /api/appversion/create.
-      payload: The body of the request, or None to send an empty request.
-      content_type: The Content-Type header to use.
-      content_md5: The Content-MD5 header to use.
-
-    Returns:
-      The content type, as a string.
-      The response body, as a string.
-    """
-    if not self.authenticated:
-      self._Authenticate()
-    if not self.xsrf_token:
-      self._GetXsrfToken()
-
-    old_timeout = socket.getdefaulttimeout()
-    socket.setdefaulttimeout(None)
-    try:
-      tries = 0
-      while True:
-        tries += 1
-        url = "http://%s%s" % (self.host, request_path)
-        req = self._CreateRequest(url=url, data=payload)
-        req.add_header("Content-Type", content_type)
-        req.add_header("Content-MD5", content_md5)
-        req.add_header("X-XSRF-Token", self.xsrf_token)
-        try:
-          f = self.opener.open(req)
-          hdr = f.info()
-          type = hdr.getheader('Content-Type',
-                               'application/octet-stream')
-          response = f.read()
-          f.close()
-          return type, response
-        except urllib2.HTTPError, e:
-          if tries > 3:
-            raise
-          elif e.code == 401:
-            self._Authenticate()
-          elif e.code == 403:
-            if not hasattr(e, 'read'):
-              e.read = lambda self: ''
-            raise RuntimeError, '403\nxsrf: %s\n%s' \
-                  % (self.xsrf_token, e.read())
-          else:
-            raise
-    finally:
-      socket.setdefaulttimeout(old_timeout)
-
-  def _GetOpener(self):
-    """Returns an OpenerDirector that supports cookies and ignores redirects.
-
-    Returns:
-      A urllib2.OpenerDirector object.
-    """
-    opener = urllib2.OpenerDirector()
-    opener.add_handler(urllib2.ProxyHandler())
-    opener.add_handler(urllib2.UnknownHandler())
-    opener.add_handler(urllib2.HTTPHandler())
-    opener.add_handler(urllib2.HTTPDefaultErrorHandler())
-    opener.add_handler(urllib2.HTTPSHandler())
-    opener.add_handler(urllib2.HTTPErrorProcessor())
-
-    self.cookie_jar, \
-    self.authenticated = _open_jar(self.cookie_file)
-    opener.add_handler(urllib2.HTTPCookieProcessor(self.cookie_jar))
-    return opener

codereview/review_pb2.py

@@ -1,48 +0,0 @@
-#!/usr/bin/python2.4
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-
-from froofle.protobuf import descriptor
-from froofle.protobuf import message
-from froofle.protobuf import reflection
-from froofle.protobuf import service
-from froofle.protobuf import service_reflection
-from froofle.protobuf import descriptor_pb2
-
-
-import upload_bundle_pb2
-
-
-
-_REVIEWSERVICE = descriptor.ServiceDescriptor(
-  name='ReviewService',
-  full_name='codereview.ReviewService',
-  index=0,
-  options=None,
-  methods=[
-  descriptor.MethodDescriptor(
-    name='UploadBundle',
-    full_name='codereview.ReviewService.UploadBundle',
-    index=0,
-    containing_service=None,
-    input_type=upload_bundle_pb2._UPLOADBUNDLEREQUEST,
-    output_type=upload_bundle_pb2._UPLOADBUNDLERESPONSE,
-    options=None,
-  ),
-  descriptor.MethodDescriptor(
-    name='ContinueBundle',
-    full_name='codereview.ReviewService.ContinueBundle',
-    index=1,
-    containing_service=None,
-    input_type=upload_bundle_pb2._UPLOADBUNDLECONTINUE,
-    output_type=upload_bundle_pb2._UPLOADBUNDLERESPONSE,
-    options=None,
-  ),
-])
-
-class ReviewService(service.Service):
-  __metaclass__ = service_reflection.GeneratedServiceType
-  DESCRIPTOR = _REVIEWSERVICE
-class ReviewService_Stub(ReviewService):
-  __metaclass__ = service_reflection.GeneratedServiceStubType
-  DESCRIPTOR = _REVIEWSERVICE
-

codereview/upload_bundle_pb2.py

@@ -1,190 +0,0 @@
-#!/usr/bin/python2.4
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-
-from froofle.protobuf import descriptor
-from froofle.protobuf import message
-from froofle.protobuf import reflection
-from froofle.protobuf import service
-from froofle.protobuf import service_reflection
-from froofle.protobuf import descriptor_pb2
-
-
-_UPLOADBUNDLERESPONSE_CODETYPE = descriptor.EnumDescriptor(
-  name='CodeType',
-  full_name='codereview.UploadBundleResponse.CodeType',
-  filename='CodeType',
-  values=[
-    descriptor.EnumValueDescriptor(
-      name='RECEIVED', index=0, number=1,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='CONTINUE', index=1, number=4,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='UNAUTHORIZED_USER', index=2, number=7,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='UNKNOWN_PROJECT', index=3, number=2,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='UNKNOWN_BRANCH', index=4, number=3,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='UNKNOWN_BUNDLE', index=5, number=5,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='NOT_BUNDLE_OWNER', index=6, number=6,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='BUNDLE_CLOSED', index=7, number=8,
-      options=None,
-      type=None),
-  ],
-  options=None,
-)
-
-
-_UPLOADBUNDLEREQUEST = descriptor.Descriptor(
-  name='UploadBundleRequest',
-  full_name='codereview.UploadBundleRequest',
-  filename='upload_bundle.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='dest_project', full_name='codereview.UploadBundleRequest.dest_project', index=0,
-      number=10, type=9, cpp_type=9, label=2,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='dest_branch', full_name='codereview.UploadBundleRequest.dest_branch', index=1,
-      number=11, type=9, cpp_type=9, label=2,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='partial_upload', full_name='codereview.UploadBundleRequest.partial_upload', index=2,
-      number=12, type=8, cpp_type=7, label=2,
-      default_value=False,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='bundle_data', full_name='codereview.UploadBundleRequest.bundle_data', index=3,
-      number=13, type=12, cpp_type=9, label=2,
-      default_value="",
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='contained_object', full_name='codereview.UploadBundleRequest.contained_object', index=4,
-      number=1, type=9, cpp_type=9, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_UPLOADBUNDLERESPONSE = descriptor.Descriptor(
-  name='UploadBundleResponse',
-  full_name='codereview.UploadBundleResponse',
-  filename='upload_bundle.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='status_code', full_name='codereview.UploadBundleResponse.status_code', index=0,
-      number=10, type=14, cpp_type=8, label=2,
-      default_value=1,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='bundle_id', full_name='codereview.UploadBundleResponse.bundle_id', index=1,
-      number=11, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-    _UPLOADBUNDLERESPONSE_CODETYPE,
-  ],
-  options=None)
-
-
-_UPLOADBUNDLECONTINUE = descriptor.Descriptor(
-  name='UploadBundleContinue',
-  full_name='codereview.UploadBundleContinue',
-  filename='upload_bundle.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='bundle_id', full_name='codereview.UploadBundleContinue.bundle_id', index=0,
-      number=10, type=9, cpp_type=9, label=2,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='segment_id', full_name='codereview.UploadBundleContinue.segment_id', index=1,
-      number=11, type=5, cpp_type=1, label=2,
-      default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='partial_upload', full_name='codereview.UploadBundleContinue.partial_upload', index=2,
-      number=12, type=8, cpp_type=7, label=2,
-      default_value=False,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='bundle_data', full_name='codereview.UploadBundleContinue.bundle_data', index=3,
-      number=13, type=12, cpp_type=9, label=1,
-      default_value="",
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_UPLOADBUNDLERESPONSE.fields_by_name['status_code'].enum_type = _UPLOADBUNDLERESPONSE_CODETYPE
-
-class UploadBundleRequest(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _UPLOADBUNDLEREQUEST
-
-class UploadBundleResponse(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _UPLOADBUNDLERESPONSE
-
-class UploadBundleContinue(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _UPLOADBUNDLECONTINUE
-

color.py

@@ -17,78 +17,105 @@ import os
 import sys
 
 import pager
-from git_config import GitConfig
 
-COLORS = {None     :-1,
-          'normal' :-1,
-          'black'  : 0,
-          'red'    : 1,
-          'green'  : 2,
-          'yellow' : 3,
-          'blue'   : 4,
+COLORS = {None: -1,
+          'normal': -1,
+          'black': 0,
+          'red': 1,
+          'green': 2,
+          'yellow': 3,
+          'blue': 4,
           'magenta': 5,
-          'cyan'   : 6,
-          'white'  : 7}
+          'cyan': 6,
+          'white': 7}
 
-ATTRS = {None     :-1,
-         'bold'   : 1,
-         'dim'    : 2,
-         'ul'     : 4,
-         'blink'  : 5,
+ATTRS = {None: -1,
+         'bold': 1,
+         'dim': 2,
+         'ul': 4,
+         'blink': 5,
          'reverse': 7}
 
 RESET = "\033[m"
 
-def is_color(s): return s in COLORS
-def is_attr(s):  return s in ATTRS
 
-def _Color(fg = None, bg = None, attr = None):
-    fg = COLORS[fg]
-    bg = COLORS[bg]
-    attr = ATTRS[attr]
+def is_color(s):
+  return s in COLORS
 
-    if attr >= 0 or fg >= 0 or bg >= 0:
-      need_sep = False
-      code = "\033["
 
-      if attr >= 0:
-        code += chr(ord('0') + attr)
-        need_sep = True
+def is_attr(s):
+  return s in ATTRS
 
-      if fg >= 0:
-        if need_sep:
-          code += ';'
-        need_sep = True
 
-        if fg < 8:
-          code += '3%c' % (ord('0') + fg)
-        else:
-          code += '38;5;%d' % fg
+def _Color(fg=None, bg=None, attr=None):
+  fg = COLORS[fg]
+  bg = COLORS[bg]
+  attr = ATTRS[attr]
 
-      if bg >= 0:
-        if need_sep:
-          code += ';'
-        need_sep = True
+  if attr >= 0 or fg >= 0 or bg >= 0:
+    need_sep = False
+    code = "\033["
 
-        if bg < 8:
-          code += '4%c' % (ord('0') + bg)
-        else:
-          code += '48;5;%d' % bg
-      code += 'm'
-    else:
-      code = ''
-    return code
+    if attr >= 0:
+      code += chr(ord('0') + attr)
+      need_sep = True
+
+    if fg >= 0:
+      if need_sep:
+        code += ';'
+      need_sep = True
+
+      if fg < 8:
+        code += '3%c' % (ord('0') + fg)
+      else:
+        code += '38;5;%d' % fg
+
+    if bg >= 0:
+      if need_sep:
+        code += ';'
+
+      if bg < 8:
+        code += '4%c' % (ord('0') + bg)
+      else:
+        code += '48;5;%d' % bg
+    code += 'm'
+  else:
+    code = ''
+  return code
+
+DEFAULT = None
+
+
+def SetDefaultColoring(state):
+  """Set coloring behavior to |state|.
+
+  This is useful for overriding config options via the command line.
+  """
+  if state is None:
+    # Leave it alone -- return quick!
+    return
+
+  global DEFAULT
+  state = state.lower()
+  if state in ('auto',):
+    DEFAULT = state
+  elif state in ('always', 'yes', 'true', True):
+    DEFAULT = 'always'
+  elif state in ('never', 'no', 'false', False):
+    DEFAULT = 'never'
 
 
 class Coloring(object):
-  def __init__(self, config, type):
-    self._section = 'color.%s' % type
+  def __init__(self, config, section_type):
+    self._section = 'color.%s' % section_type
     self._config = config
     self._out = sys.stdout
 
-    on = self._config.GetString(self._section)
+    on = DEFAULT
     if on is None:
-      on = self._config.GetString('color.ui')
+      on = self._config.GetString(self._section)
+      if on is None:
+        on = self._config.GetString('color.ui')
 
     if on == 'auto':
       if pager.active or os.isatty(1):
@@ -100,6 +127,9 @@ class Coloring(object):
     else:
       self._on = False
 
+  def redirect(self, out):
+    self._out = out
+
   @property
   def is_on(self):
     return self._on
@@ -107,28 +137,54 @@ class Coloring(object):
   def write(self, fmt, *args):
     self._out.write(fmt % args)
 
+  def flush(self):
+    self._out.flush()
+
   def nl(self):
     self._out.write('\n')
 
   def printer(self, opt=None, fg=None, bg=None, attr=None):
     s = self
     c = self.colorer(opt, fg, bg, attr)
+
     def f(fmt, *args):
       s._out.write(c(fmt, *args))
     return f
 
+  def nofmt_printer(self, opt=None, fg=None, bg=None, attr=None):
+    s = self
+    c = self.nofmt_colorer(opt, fg, bg, attr)
+
+    def f(fmt):
+      s._out.write(c(fmt))
+    return f
+
   def colorer(self, opt=None, fg=None, bg=None, attr=None):
     if self._on:
       c = self._parse(opt, fg, bg, attr)
+
       def f(fmt, *args):
-        str = fmt % args
-        return ''.join([c, str, RESET])
+        output = fmt % args
+        return ''.join([c, output, RESET])
       return f
     else:
+
       def f(fmt, *args):
         return fmt % args
       return f
 
+  def nofmt_colorer(self, opt=None, fg=None, bg=None, attr=None):
+    if self._on:
+      c = self._parse(opt, fg, bg, attr)
+
+      def f(fmt):
+        return ''.join([c, fmt, RESET])
+      return f
+    else:
+      def f(fmt):
+        return fmt
+      return f
+
   def _parse(self, opt, fg, bg, attr):
     if not opt:
       return _Color(fg, bg, attr)
@@ -137,7 +193,7 @@ class Coloring(object):
     if v is None:
       return _Color(fg, bg, attr)
 
-    v = v.trim().lowercase()
+    v = v.strip().lower()
     if v == "reset":
       return RESET
     elif v == '':
@@ -146,8 +202,10 @@ class Coloring(object):
     have_fg = False
     for a in v.split(' '):
       if is_color(a):
-        if have_fg: bg = a
-        else:       fg = a
+        if have_fg:
+          bg = a
+        else:
+          fg = a
       elif is_attr(a):
         attr = a
 

command.py

@@ -15,18 +15,48 @@
 
 import os
 import optparse
+import platform
+import re
 import sys
 
+from event_log import EventLog
 from error import NoSuchProjectError
+from error import InvalidProjectGroupsError
+
 
 class Command(object):
   """Base class for any command line action in repo.
   """
 
   common = False
+  event_log = EventLog()
   manifest = None
   _optparse = None
 
+  def WantPager(self, _opt):
+    return False
+
+  def ReadEnvironmentOptions(self, opts):
+    """ Set options from environment variables. """
+
+    env_options = self._RegisteredEnvironmentOptions()
+
+    for env_key, opt_key in env_options.items():
+      # Get the user-set option value if any
+      opt_value = getattr(opts, opt_key)
+
+      # If the value is set, it means the user has passed it as a command
+      # line option, and we should use that.  Otherwise we can try to set it
+      # with the value from the corresponding environment variable.
+      if opt_value is not None:
+        continue
+
+      env_value = os.environ.get(env_key)
+      if env_value is not None:
+        setattr(opts, opt_key, env_value)
+
+    return opts
+
   @property
   def OptionParser(self):
     if self._optparse is None:
@@ -35,7 +65,7 @@ class Command(object):
         usage = self.helpUsage.strip().replace('%prog', me)
       except AttributeError:
         usage = 'repo %s' % self.NAME
-      self._optparse = optparse.OptionParser(usage = usage)
+      self._optparse = optparse.OptionParser(usage=usage)
       self._Options(self._optparse)
     return self._optparse
 
@@ -43,6 +73,24 @@ class Command(object):
     """Initialize the option parser.
     """
 
+  def _RegisteredEnvironmentOptions(self):
+    """Get options that can be set from environment variables.
+
+    Return a dictionary mapping environment variable name
+    to option key name that it can override.
+
+    Example: {'REPO_MY_OPTION': 'my_option'}
+
+    Will allow the option with key value 'my_option' to be set
+    from the value in the environment variable named 'REPO_MY_OPTION'.
+
+    Note: This does not work properly for options that are explicitly
+    set to None by the user, or options that are defined with a
+    default value other than None.
+
+    """
+    return {}
+
   def Usage(self):
     """Display usage and terminate.
     """
@@ -53,64 +101,152 @@ class Command(object):
     """Perform the action, after option parsing is complete.
     """
     raise NotImplementedError
- 
-  def GetProjects(self, args, missing_ok=False):
+
+  def _ResetPathToProjectMap(self, projects):
+    self._by_path = dict((p.worktree, p) for p in projects)
+
+  def _UpdatePathToProjectMap(self, project):
+    self._by_path[project.worktree] = project
+
+  def _GetProjectByPath(self, manifest, path):
+    project = None
+    if os.path.exists(path):
+      oldpath = None
+      while path and \
+            path != oldpath and \
+            path != manifest.topdir:
+        try:
+          project = self._by_path[path]
+          break
+        except KeyError:
+          oldpath = path
+          path = os.path.dirname(path)
+      if not project and path == manifest.topdir:
+        try:
+          project = self._by_path[path]
+        except KeyError:
+          pass
+    else:
+      try:
+        project = self._by_path[path]
+      except KeyError:
+        pass
+    return project
+
+  def GetProjects(self, args, manifest=None, groups='', missing_ok=False,
+                  submodules_ok=False):
     """A list of projects that match the arguments.
     """
-    all = self.manifest.projects
+    if not manifest:
+      manifest = self.manifest
+    all_projects_list = manifest.projects
     result = []
 
+    mp = manifest.manifestProject
+
+    if not groups:
+      groups = mp.config.GetString('manifest.groups')
+    if not groups:
+      groups = 'default,platform-' + platform.system().lower()
+    groups = [x for x in re.split(r'[,\s]+', groups) if x]
+
     if not args:
-      for project in all.values():
-        if missing_ok or project.Exists:
+      derived_projects = {}
+      for project in all_projects_list:
+        if submodules_ok or project.sync_s:
+          derived_projects.update((p.name, p)
+                                  for p in project.GetDerivedSubprojects())
+      all_projects_list.extend(derived_projects.values())
+      for project in all_projects_list:
+        if (missing_ok or project.Exists) and project.MatchesGroups(groups):
           result.append(project)
     else:
-      by_path = None
+      self._ResetPathToProjectMap(all_projects_list)
 
       for arg in args:
-        project = all.get(arg)
+        projects = manifest.GetProjectsWithName(arg)
 
-        if not project:
-          path = os.path.abspath(arg)
+        if not projects:
+          path = os.path.abspath(arg).replace('\\', '/')
+          project = self._GetProjectByPath(manifest, path)
 
-          if not by_path:
-            by_path = dict()
-            for p in all.values():
-              by_path[p.worktree] = p
+          # If it's not a derived project, update path->project mapping and
+          # search again, as arg might actually point to a derived subproject.
+          if (project and not project.Derived and (submodules_ok or
+                                                   project.sync_s)):
+            search_again = False
+            for subproject in project.GetDerivedSubprojects():
+              self._UpdatePathToProjectMap(subproject)
+              search_again = True
+            if search_again:
+              project = self._GetProjectByPath(manifest, path) or project
 
-          if os.path.exists(path):
-            while path \
-              and path != '/' \
-              and path != self.manifest.topdir:
-              try:
-                project = by_path[path]
-                break
-              except KeyError:
-                path = os.path.dirname(path)
-          else:
-            try:
-              project = by_path[path]
-            except KeyError:
-              pass
+          if project:
+            projects = [project]
 
-        if not project:
-          raise NoSuchProjectError(arg)
-        if not missing_ok and not project.Exists:
+        if not projects:
           raise NoSuchProjectError(arg)
 
-        result.append(project)
+        for project in projects:
+          if not missing_ok and not project.Exists:
+            raise NoSuchProjectError(arg)
+          if not project.MatchesGroups(groups):
+            raise InvalidProjectGroupsError(arg)
+
+        result.extend(projects)
 
     def _getpath(x):
       return x.relpath
     result.sort(key=_getpath)
     return result
 
+  def FindProjects(self, args, inverse=False):
+    result = []
+    patterns = [re.compile(r'%s' % a, re.IGNORECASE) for a in args]
+    for project in self.GetProjects(''):
+      for pattern in patterns:
+        match = pattern.search(project.name) or pattern.search(project.relpath)
+        if not inverse and match:
+          result.append(project)
+          break
+        if inverse and match:
+          break
+      else:
+        if inverse:
+          result.append(project)
+    result.sort(key=lambda project: project.relpath)
+    return result
+
+
 class InteractiveCommand(Command):
   """Command which requires user interaction on the tty and
      must not run within a pager, even if the user asks to.
   """
+  def WantPager(self, _opt):
+    return False
+
 
 class PagedCommand(Command):
   """Command which defaults to output in a pager, as its
      display tends to be larger than one screen full.
   """
+  def WantPager(self, _opt):
+    return True
+
+
+class MirrorSafeCommand(object):
+  """Command permits itself to run within a mirror,
+     and does not require a working directory.
+  """
+
+
+class GitcAvailableCommand(object):
+  """Command that requires GITC to be available, but does
+     not require the local client to be a GITC client.
+  """
+
+
+class GitcClientCommand(object):
+  """Command that requires the local client to be a GITC
+     client.
+  """

docs/manifest-format.md

@@ -0,0 +1,385 @@
+# repo Manifest Format
+
+A repo manifest describes the structure of a repo client; that is
+the directories that are visible and where they should be obtained
+from with git.
+
+The basic structure of a manifest is a bare Git repository holding
+a single `default.xml` XML file in the top level directory.
+
+Manifests are inherently version controlled, since they are kept
+within a Git repository.  Updates to manifests are automatically
+obtained by clients during `repo sync`.
+
+[TOC]
+
+
+## XML File Format
+
+A manifest XML file (e.g. `default.xml`) roughly conforms to the
+following DTD:
+
+```xml
+<!DOCTYPE manifest [
+  <!ELEMENT manifest (notice?,
+                      remote*,
+                      default?,
+                      manifest-server?,
+                      remove-project*,
+                      project*,
+                      extend-project*,
+                      repo-hooks?,
+                      include*)>
+
+  <!ELEMENT notice (#PCDATA)>
+
+  <!ELEMENT remote EMPTY>
+  <!ATTLIST remote name         ID    #REQUIRED>
+  <!ATTLIST remote alias        CDATA #IMPLIED>
+  <!ATTLIST remote fetch        CDATA #REQUIRED>
+  <!ATTLIST remote pushurl      CDATA #IMPLIED>
+  <!ATTLIST remote review       CDATA #IMPLIED>
+  <!ATTLIST remote revision     CDATA #IMPLIED>
+
+  <!ELEMENT default EMPTY>
+  <!ATTLIST default remote      IDREF #IMPLIED>
+  <!ATTLIST default revision    CDATA #IMPLIED>
+  <!ATTLIST default dest-branch CDATA #IMPLIED>
+  <!ATTLIST default upstream    CDATA #IMPLIED>
+  <!ATTLIST default sync-j      CDATA #IMPLIED>
+  <!ATTLIST default sync-c      CDATA #IMPLIED>
+  <!ATTLIST default sync-s      CDATA #IMPLIED>
+  <!ATTLIST default sync-tags   CDATA #IMPLIED>
+
+  <!ELEMENT manifest-server EMPTY>
+  <!ATTLIST manifest-server url CDATA #REQUIRED>
+
+  <!ELEMENT project (annotation*,
+                     project*,
+                     copyfile*,
+                     linkfile*)>
+  <!ATTLIST project name        CDATA #REQUIRED>
+  <!ATTLIST project path        CDATA #IMPLIED>
+  <!ATTLIST project remote      IDREF #IMPLIED>
+  <!ATTLIST project revision    CDATA #IMPLIED>
+  <!ATTLIST project dest-branch CDATA #IMPLIED>
+  <!ATTLIST project groups      CDATA #IMPLIED>
+  <!ATTLIST project sync-c      CDATA #IMPLIED>
+  <!ATTLIST project sync-s      CDATA #IMPLIED>
+  <!ATTLIST default sync-tags   CDATA #IMPLIED>
+  <!ATTLIST project upstream CDATA #IMPLIED>
+  <!ATTLIST project clone-depth CDATA #IMPLIED>
+  <!ATTLIST project force-path CDATA #IMPLIED>
+
+  <!ELEMENT annotation EMPTY>
+  <!ATTLIST annotation name  CDATA #REQUIRED>
+  <!ATTLIST annotation value CDATA #REQUIRED>
+  <!ATTLIST annotation keep  CDATA "true">
+
+  <!ELEMENT copyfile EMPTY>
+  <!ATTLIST copyfile src  CDATA #REQUIRED>
+  <!ATTLIST copyfile dest CDATA #REQUIRED>
+
+  <!ELEMENT linkfile EMPTY>
+  <!ATTLIST linkfile src CDATA #REQUIRED>
+  <!ATTLIST linkfile dest CDATA #REQUIRED>
+
+  <!ELEMENT extend-project EMPTY>
+  <!ATTLIST extend-project name CDATA #REQUIRED>
+  <!ATTLIST extend-project path CDATA #IMPLIED>
+  <!ATTLIST extend-project groups CDATA #IMPLIED>
+  <!ATTLIST extend-project revision CDATA #IMPLIED>
+
+  <!ELEMENT remove-project EMPTY>
+  <!ATTLIST remove-project name  CDATA #REQUIRED>
+
+  <!ELEMENT repo-hooks EMPTY>
+  <!ATTLIST repo-hooks in-project CDATA #REQUIRED>
+  <!ATTLIST repo-hooks enabled-list CDATA #REQUIRED>
+
+  <!ELEMENT include EMPTY>
+  <!ATTLIST include name CDATA #REQUIRED>
+]>
+```
+
+A description of the elements and their attributes follows.
+
+
+### Element manifest
+
+The root element of the file.
+
+
+### Element remote
+
+One or more remote elements may be specified.  Each remote element
+specifies a Git URL shared by one or more projects and (optionally)
+the Gerrit review server those projects upload changes through.
+
+Attribute `name`: A short name unique to this manifest file.  The
+name specified here is used as the remote name in each project's
+.git/config, and is therefore automatically available to commands
+like `git fetch`, `git remote`, `git pull` and `git push`.
+
+Attribute `alias`: The alias, if specified, is used to override
+`name` to be set as the remote name in each project's .git/config.
+Its value can be duplicated while attribute `name` has to be unique
+in the manifest file. This helps each project to be able to have
+same remote name which actually points to different remote url.
+
+Attribute `fetch`: The Git URL prefix for all projects which use
+this remote.  Each project's name is appended to this prefix to
+form the actual URL used to clone the project.
+
+Attribute `pushurl`: The Git "push" URL prefix for all projects
+which use this remote.  Each project's name is appended to this
+prefix to form the actual URL used to "git push" the project.
+This attribute is optional; if not specified then "git push"
+will use the same URL as the `fetch` attribute.
+
+Attribute `review`: Hostname of the Gerrit server where reviews
+are uploaded to by `repo upload`.  This attribute is optional;
+if not specified then `repo upload` will not function.
+
+Attribute `revision`: Name of a Git branch (e.g. `master` or
+`refs/heads/master`). Remotes with their own revision will override
+the default revision.
+
+### Element default
+
+At most one default element may be specified.  Its remote and
+revision attributes are used when a project element does not
+specify its own remote or revision attribute.
+
+Attribute `remote`: Name of a previously defined remote element.
+Project elements lacking a remote attribute of their own will use
+this remote.
+
+Attribute `revision`: Name of a Git branch (e.g. `master` or
+`refs/heads/master`).  Project elements lacking their own
+revision attribute will use this revision.
+
+Attribute `dest-branch`: Name of a Git branch (e.g. `master`).
+Project elements not setting their own `dest-branch` will inherit
+this value. If this value is not set, projects will use `revision`
+by default instead.
+
+Attribute `upstream`: Name of the Git ref in which a sha1
+can be found.  Used when syncing a revision locked manifest in
+-c mode to avoid having to sync the entire ref space. Project elements
+not setting their own `upstream` will inherit this value.
+
+Attribute `sync-j`: Number of parallel jobs to use when synching.
+
+Attribute `sync-c`: Set to true to only sync the given Git
+branch (specified in the `revision` attribute) rather than the
+whole ref space.  Project elements lacking a sync-c element of
+their own will use this value.
+
+Attribute `sync-s`: Set to true to also sync sub-projects.
+
+Attribute `sync-tags`: Set to false to only sync the given Git
+branch (specified in the `revision` attribute) rather than
+the other ref tags.
+
+
+### Element manifest-server
+
+At most one manifest-server may be specified. The url attribute
+is used to specify the URL of a manifest server, which is an
+XML RPC service.
+
+The manifest server should implement the following RPC methods:
+
+    GetApprovedManifest(branch, target)
+
+Return a manifest in which each project is pegged to a known good revision
+for the current branch and target. This is used by repo sync when the
+--smart-sync option is given.
+
+The target to use is defined by environment variables TARGET_PRODUCT
+and TARGET_BUILD_VARIANT. These variables are used to create a string
+of the form $TARGET_PRODUCT-$TARGET_BUILD_VARIANT, e.g. passion-userdebug.
+If one of those variables or both are not present, the program will call
+GetApprovedManifest without the target parameter and the manifest server
+should choose a reasonable default target.
+
+    GetManifest(tag)
+
+Return a manifest in which each project is pegged to the revision at
+the specified tag. This is used by repo sync when the --smart-tag option
+is given.
+
+
+### Element project
+
+One or more project elements may be specified.  Each element
+describes a single Git repository to be cloned into the repo
+client workspace.  You may specify Git-submodules by creating a
+nested project.  Git-submodules will be automatically
+recognized and inherit their parent's attributes, but those
+may be overridden by an explicitly specified project element.
+
+Attribute `name`: A unique name for this project.  The project's
+name is appended onto its remote's fetch URL to generate the actual
+URL to configure the Git remote with.  The URL gets formed as:
+
+    ${remote_fetch}/${project_name}.git
+
+where ${remote_fetch} is the remote's fetch attribute and
+${project_name} is the project's name attribute.  The suffix ".git"
+is always appended as repo assumes the upstream is a forest of
+bare Git repositories.  If the project has a parent element, its
+name will be prefixed by the parent's.
+
+The project name must match the name Gerrit knows, if Gerrit is
+being used for code reviews.
+
+Attribute `path`: An optional path relative to the top directory
+of the repo client where the Git working directory for this project
+should be placed.  If not supplied the project name is used.
+If the project has a parent element, its path will be prefixed
+by the parent's.
+
+Attribute `remote`: Name of a previously defined remote element.
+If not supplied the remote given by the default element is used.
+
+Attribute `revision`: Name of the Git branch the manifest wants
+to track for this project.  Names can be relative to refs/heads
+(e.g. just "master") or absolute (e.g. "refs/heads/master").
+Tags and/or explicit SHA-1s should work in theory, but have not
+been extensively tested.  If not supplied the revision given by
+the remote element is used if applicable, else the default
+element is used.
+
+Attribute `dest-branch`: Name of a Git branch (e.g. `master`).
+When using `repo upload`, changes will be submitted for code
+review on this branch. If unspecified both here and in the
+default element, `revision` is used instead.
+
+Attribute `groups`: List of groups to which this project belongs,
+whitespace or comma separated.  All projects belong to the group
+"all", and each project automatically belongs to a group of
+its name:`name` and path:`path`.  E.g. for
+<project name="monkeys" path="barrel-of"/>, that project
+definition is implicitly in the following manifest groups:
+default, name:monkeys, and path:barrel-of.  If you place a project in the
+group "notdefault", it will not be automatically downloaded by repo.
+If the project has a parent element, the `name` and `path` here
+are the prefixed ones.
+
+Attribute `sync-c`: Set to true to only sync the given Git
+branch (specified in the `revision` attribute) rather than the
+whole ref space.
+
+Attribute `sync-s`: Set to true to also sync sub-projects.
+
+Attribute `upstream`: Name of the Git ref in which a sha1
+can be found.  Used when syncing a revision locked manifest in
+-c mode to avoid having to sync the entire ref space.
+
+Attribute `clone-depth`: Set the depth to use when fetching this
+project.  If specified, this value will override any value given
+to repo init with the --depth option on the command line.
+
+Attribute `force-path`: Set to true to force this project to create the
+local mirror repository according to its `path` attribute (if supplied)
+rather than the `name` attribute.  This attribute only applies to the
+local mirrors syncing, it will be ignored when syncing the projects in a
+client working directory.
+
+### Element extend-project
+
+Modify the attributes of the named project.
+
+This element is mostly useful in a local manifest file, to modify the
+attributes of an existing project without completely replacing the
+existing project definition.  This makes the local manifest more robust
+against changes to the original manifest.
+
+Attribute `path`: If specified, limit the change to projects checked out
+at the specified path, rather than all projects with the given name.
+
+Attribute `groups`: List of additional groups to which this project
+belongs.  Same syntax as the corresponding element of `project`.
+
+Attribute `revision`: If specified, overrides the revision of the original
+project.  Same syntax as the corresponding element of `project`.
+
+### Element annotation
+
+Zero or more annotation elements may be specified as children of a
+project element. Each element describes a name-value pair that will be
+exported into each project's environment during a 'forall' command,
+prefixed with REPO__.  In addition, there is an optional attribute
+"keep" which accepts the case insensitive values "true" (default) or
+"false".  This attribute determines whether or not the annotation will
+be kept when exported with the manifest subcommand.
+
+### Element copyfile
+
+Zero or more copyfile elements may be specified as children of a
+project element. Each element describes a src-dest pair of files;
+the "src" file will be copied to the "dest" place during `repo sync`
+command.
+"src" is project relative, "dest" is relative to the top of the tree.
+
+### Element linkfile
+
+It's just like copyfile and runs at the same time as copyfile but
+instead of copying it creates a symlink.
+
+### Element remove-project
+
+Deletes the named project from the internal manifest table, possibly
+allowing a subsequent project element in the same manifest file to
+replace the project with a different source.
+
+This element is mostly useful in a local manifest file, where
+the user can remove a project, and possibly replace it with their
+own definition.
+
+### Element include
+
+This element provides the capability of including another manifest
+file into the originating manifest.  Normal rules apply for the
+target manifest to include - it must be a usable manifest on its own.
+
+Attribute `name`: the manifest to include, specified relative to
+the manifest repository's root.
+
+
+## Local Manifests
+
+Additional remotes and projects may be added through local manifest
+files stored in `$TOP_DIR/.repo/local_manifests/*.xml`.
+
+For example:
+
+    $ ls .repo/local_manifests
+    local_manifest.xml
+    another_local_manifest.xml
+
+    $ cat .repo/local_manifests/local_manifest.xml
+    <?xml version="1.0" encoding="UTF-8"?>
+    <manifest>
+      <project path="manifest"
+               name="tools/manifest" />
+      <project path="platform-manifest"
+               name="platform/manifest" />
+    </manifest>
+
+Users may add projects to the local manifest(s) prior to a `repo sync`
+invocation, instructing repo to automatically download and manage
+these extra projects.
+
+Manifest files stored in `$TOP_DIR/.repo/local_manifests/*.xml` will
+be loaded in alphabetical order.
+
+Additional remotes and projects may also be added through a local
+manifest, stored in `$TOP_DIR/.repo/local_manifest.xml`. This method
+is deprecated in favor of using multiple manifest files as mentioned
+above.
+
+If `$TOP_DIR/.repo/local_manifest.xml` exists, it will be loaded before
+any manifest files stored in `$TOP_DIR/.repo/local_manifests/*.xml`.

docs/manifest-format.txt

@@ -1,184 +0,0 @@
-repo Manifest Format
-====================
-
-A repo manifest describes the structure of a repo client; that is
-the directories that are visible and where they should be obtained
-from with git.
-
-The basic structure of a manifest is a bare Git repository holding
-a single 'default.xml' XML file in the top level directory.
-
-Manifests are inherently version controlled, since they are kept
-within a Git repository.  Updates to manifests are automatically
-obtained by clients during `repo sync`.
-
-
-XML File Format
----------------
-
-A manifest XML file (e.g. 'default.xml') roughly conforms to the
-following DTD:
-
-<!DOCTYPE manifest [
-  <!ELEMENT manifest (remote*,
-                      default?,
-                      project*,
-                      add-remote*)>
-
-  <!ELEMENT remote (EMPTY)>
-  <!ATTLIST remote name         ID    #REQUIRED>
-  <!ATTLIST remote fetch        CDATA #REQUIRED>
-  <!ATTLIST remote review       CDATA #IMPLIED>
-  <!ATTLIST remote project-name CDATA #IMPLIED>
-
-  <!ELEMENT default (EMPTY)>
-  <!ATTLIST default remote   IDREF #IMPLIED>
-  <!ATTLIST default revision CDATA #IMPLIED>
-
-  <!ELEMENT project (remote*)>
-  <!ATTLIST project name     CDATA #REQUIRED>
-  <!ATTLIST project path     CDATA #IMPLIED>
-  <!ATTLIST project remote   IDREF #IMPLIED>
-  <!ATTLIST project revision CDATA #IMPLIED>
-
-  <!ELEMENT add-remote (EMPTY)>
-  <!ATTLIST add-remote to-project   ID    #REQUIRED>
-  <!ATTLIST add-remote name         ID    #REQUIRED>
-  <!ATTLIST add-remote fetch        CDATA #REQUIRED>
-  <!ATTLIST add-remote review       CDATA #IMPLIED>
-  <!ATTLIST add-remote project-name CDATA #IMPLIED>
-]>
-
-A description of the elements and their attributes follows.
-
-
-Element manifest
-----------------
-
-The root element of the file.
-
-
-Element remote
---------------
-
-One or more remote elements may be specified.  Each remote element
-specifies a Git URL shared by one or more projects and (optionally)
-the Gerrit review server those projects upload changes through.
-
-Attribute `name`: A short name unique to this manifest file.  The
-name specified here is used as the remote name in each project's
-.git/config, and is therefore automatically available to commands
-like `git fetch`, `git remote`, `git pull` and `git push`.
-
-Attribute `fetch`: The Git URL prefix for all projects which use
-this remote.  Each project's name is appended to this prefix to
-form the actual URL used to clone the project.
-
-Attribute `review`: Hostname of the Gerrit server where reviews
-are uploaded to by `repo upload`.  This attribute is optional;
-if not specified then `repo upload` will not function.
-
-Attribute `project-name`: Specifies the name of this project used
-by the review server given in the review attribute of this element.
-Only permitted when the remote element is nested inside of a project
-element (see below).  If not given, defaults to the name supplied
-in the project's name attribute.
-
-Element add-remote
-------------------
-
-Adds a remote to an existing project, whose name is given by the
-to-project attribute.  This is functionally equivalent to nesting
-a remote element under the project, but has the advantage that it
-can be specified in the uesr's `local_manifest.xml` to add a remote
-to a project declared by the normal manifest.
-
-The element can be used to add a fork of an existing project that
-the user needs to work with.
-
-
-Element default
----------------
-
-At most one default element may be specified.  Its remote and
-revision attributes are used when a project element does not
-specify its own remote or revision attribute.
-
-Attribute `remote`: Name of a previously defined remote element.
-Project elements lacking a remote attribute of their own will use
-this remote.
-
-Attribute `revision`: Name of a Git branch (e.g. `master` or
-`refs/heads/master`).  Project elements lacking their own
-revision attribute will use this revision.
-
-
-Element project
----------------
-
-One or more project elements may be specified.  Each element
-describes a single Git repository to be cloned into the repo
-client workspace.
-
-Attribute `name`: A unique name for this project.  The project's
-name is appended onto its remote's fetch URL to generate the actual
-URL to configure the Git remote with.  The URL gets formed as:
-
-  ${remote_fetch}/${project_name}.git
-
-where ${remote_fetch} is the remote's fetch attribute and
-${project_name} is the project's name attribute.  The suffix ".git"
-is always appended as repo assumes the upstream is a forrest of
-bare Git repositories.
-
-The project name must match the name Gerrit knows, if Gerrit is
-being used for code reviews.
-
-Attribute `path`: An optional path relative to the top directory
-of the repo client where the Git working directory for this project
-should be placed.  If not supplied the project name is used.
-
-Attribute `remote`: Name of a previously defined remote element.
-If not supplied the remote given by the default element is used.
-
-Attribute `revision`: Name of the Git branch the manifest wants
-to track for this project.  Names can be relative to refs/heads
-(e.g. just "master") or absolute (e.g. "refs/heads/master").
-Tags and/or explicit SHA-1s should work in theory, but have not
-been extensively tested.  If not supplied the revision given by
-the default element is used.
-
-Child element `remote`: Described like the top-level remote element,
-but adds an additional remote to only this project.  These additional
-remotes are fetched from first on the initial `repo sync`, causing
-the majority of the project's object database to be obtained through
-these additional remotes.
-
-
-Local Manifest
-==============
-
-Additional remotes and projects may be added through a local
-manifest, stored in `$TOP_DIR/.repo/local_manifest.xml`.
-
-For example:
-
-----
-	$ cat .repo/local_manifest.xml
-	<?xml version="1.0" encoding="UTF-8"?>
-	<manifest>
-	  <project path="manifest"
-			   name="tools/manifest" />
-	  <project path="platform-manifest"
-			   name="platform/manifest" />
-	</manifest>
-----
-
-Users may add projects to the local manifest prior to a `repo sync`
-invocation, instructing repo to automatically download and manage
-these extra projects.
-
-Currently the only supported feature of a local manifest is to
-add new remotes and/or projects.  In the future a local manifest
-may support picking different revisions of a project, or deleting
-projects specified in the default manifest.

docs/repo-hooks.md

@@ -0,0 +1,110 @@
+# repo hooks
+
+[TOC]
+
+Repo provides a mechanism to hook specific stages of the runtime with custom
+python modules.  All the hooks live in one git project which is checked out by
+the manifest (specified during `repo init`), and the manifest itself defines
+which hooks are registered.
+
+These are useful to run linters, check formatting, and run quick unittests
+before allowing a step to proceed (e.g. before uploading a commit to Gerrit).
+
+A complete example can be found in the Android project.  It can be easily
+re-used by any repo based project and is not specific to Android.<br>
+https://android.googlesource.com/platform/tools/repohooks
+
+## Approvals
+
+When a hook is processed the first time, the user is prompted for approval.
+We don't want to execute arbitrary code without explicit consent.  For manifests
+fetched via secure protocols (e.g. https://), the user is prompted once.  For
+insecure protocols (e.g. http://), the user is prompted whenever the registered
+repohooks project is updated and a hook is triggered.
+
+## Manifest Settings
+
+For the full syntax, see the [repo manifest format](./manifest-format.md).
+
+Here's a short example from
+[Android](https://android.googlesource.com/platform/manifest/+/master/default.xml).
+The `<project>` line checks out the repohooks git repo to the local
+`tools/repohooks/` path.  The `<repo-hooks>` line says to look in the project
+with the name `platform/tools/repohooks` for hooks to run during the
+`pre-upload` phase.
+
+```xml
+<project path="tools/repohooks" name="platform/tools/repohooks" />
+<repo-hooks in-project="platform/tools/repohooks" enabled-list="pre-upload" />
+```
+
+## Source Layout
+
+The repohooks git repo should have a python file with the same name as the hook.
+So if you want to support the `pre-upload` hook, you'll need to create a file
+named `pre-upload.py`.  Repo will dynamically load that module when processing
+the hook and then call the `main` function in it.
+
+Hooks should have their `main` accept `**kwargs` for future compatibility.
+
+## Runtime
+
+Hook return values are ignored.
+
+Any uncaught exceptions from the hook will cause the step to fail.  This is
+intended as a fallback safety check though rather than the normal flow.  If
+you want your hook to trigger a failure, it should call `sys.exit()` (after
+displaying relevant diagnostics).
+
+Output (stdout & stderr) are not filtered in any way.  Hooks should generally
+not be too verbose.  A short summary is nice, and some status information when
+long running operations occur, but long/verbose output should be used only if
+the hook ultimately fails.
+
+The hook runs from the top level of the repo client where the operation is
+started.
+For example, if the repo client is under `~/tree/`, then that is where the hook
+runs, even if you ran repo in a git repository at `~/tree/src/foo/`, or in a
+subdirectory of that git repository in `~/tree/src/foo/bar/`.
+Hooks frequently start off by doing a `os.chdir` to the specific project they're
+called on (see below) and then changing back to the original dir when they're
+finished.
+
+Python's `sys.path` is modified so that the top of repohooks directory comes
+first.  This should help simplify the hook logic to easily allow importing of
+local modules.
+
+Repo does not modify the state of the git checkout.  This means that the hooks
+might be running in a dirty git repo with many commits and checked out to the
+latest one.  If the hook wants to operate on specific git commits, it needs to
+manually discover the list of pending commits, extract the diff/commit, and
+then check it directly.  Hooks should not normally modify the active git repo
+(such as checking out a specific commit to run checks) without first prompting
+the user.  Although user interaction is discouraged in the common case, it can
+be useful when deploying automatic fixes.
+
+## Hooks
+
+Here are all the points available for hooking.
+
+### pre-upload
+
+This hook runs when people run `repo upload`.
+
+The `pre-upload.py` file should be defined like:
+
+```py
+def main(project_list, worktree_list=None, **kwargs):
+    """Main function invoked directly by repo.
+
+    We must use the name "main" as that is what repo requires.
+
+    Args:
+      project_list: List of projects to run on.
+      worktree_list: A list of directories.  It should be the same length as
+          project_list, so that each entry in project_list matches with a
+          directory in worktree_list.  If None, we will attempt to calculate
+          the directories automatically.
+      kwargs: Leave this here for forward-compatibility.
+    """
+```

editor.py

@@ -13,12 +13,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
 import os
+import re
 import sys
 import subprocess
 import tempfile
 
 from error import EditorError
+import platform_utils
 
 class Editor(object):
   """Manages the user's preferred text editor."""
@@ -38,9 +41,10 @@ class Editor(object):
     if e:
       return e
 
-    e = cls.globalConfig.GetString('core.editor')
-    if e:
-      return e
+    if cls.globalConfig:
+      e = cls.globalConfig.GetString('core.editor')
+      if e:
+        return e
 
     e = os.getenv('VISUAL')
     if e:
@@ -51,10 +55,10 @@ class Editor(object):
       return e
 
     if os.getenv('TERM') == 'dumb':
-      print >>sys.stderr,\
+      print(
 """No editor specified in GIT_EDITOR, core.editor, VISUAL or EDITOR.
 Tried to fall back to vi but terminal is dumb.  Please configure at
-least one of these before using this command."""
+least one of these before using this command.""", file=sys.stderr)
       sys.exit(1)
 
     return 'vi'
@@ -65,21 +69,48 @@ least one of these before using this command."""
 
        Args:
          data        : the text to edit
-  
+
       Returns:
         new value of edited text; None if editing did not succeed
     """
     editor = cls._GetEditor()
+    if editor == ':':
+      return data
+
     fd, path = tempfile.mkstemp()
     try:
       os.write(fd, data)
       os.close(fd)
       fd = None
 
-      if subprocess.Popen([editor, path]).wait() != 0:
-        raise EditorError()
-      return open(path).read()
+      if platform_utils.isWindows():
+        # Split on spaces, respecting quoted strings
+        import shlex
+        args = shlex.split(editor)
+        shell = False
+      elif re.compile("^.*[$ \t'].*$").match(editor):
+        args = [editor + ' "$@"', 'sh']
+        shell = True
+      else:
+        args = [editor]
+        shell = False
+      args.append(path)
+
+      try:
+        rc = subprocess.Popen(args, shell=shell).wait()
+      except OSError as e:
+        raise EditorError('editor failed, %s: %s %s'
+          % (str(e), editor, path))
+      if rc != 0:
+        raise EditorError('editor failed with exit status %d: %s %s'
+          % (rc, editor, path))
+
+      fd2 = open(path)
+      try:
+        return fd2.read()
+      finally:
+        fd2.close()
     finally:
       if fd:
         os.close(fd)
-      os.remove(path)
+      platform_utils.remove(path)

error.py

@@ -17,32 +17,56 @@ class ManifestParseError(Exception):
   """Failed to parse the manifest file.
   """
 
-class EditorError(Exception):
-  """Unspecified error from the user's text editor.
+class ManifestInvalidRevisionError(Exception):
+  """The revision value in a project is incorrect.
   """
 
-class GitError(Exception):
-  """Unspecified internal error from git.
+class NoManifestException(Exception):
+  """The required manifest does not exist.
   """
-  def __init__(self, command):
-    self.command = command
-
-  def __str__(self):
-    return self.command
-
-class ImportError(Exception):
-  """An import from a non-Git format cannot be performed.
-  """
-  def __init__(self, reason):
+  def __init__(self, path, reason):
+    super(NoManifestException, self).__init__()
+    self.path = path
     self.reason = reason
 
   def __str__(self):
     return self.reason
 
+class EditorError(Exception):
+  """Unspecified error from the user's text editor.
+  """
+  def __init__(self, reason):
+    super(EditorError, self).__init__()
+    self.reason = reason
+
+  def __str__(self):
+    return self.reason
+
+class GitError(Exception):
+  """Unspecified internal error from git.
+  """
+  def __init__(self, command):
+    super(GitError, self).__init__()
+    self.command = command
+
+  def __str__(self):
+    return self.command
+
 class UploadError(Exception):
   """A bundle upload to Gerrit did not succeed.
   """
   def __init__(self, reason):
+    super(UploadError, self).__init__()
+    self.reason = reason
+
+  def __str__(self):
+    return self.reason
+
+class DownloadError(Exception):
+  """Cannot download a repository.
+  """
+  def __init__(self, reason):
+    super(DownloadError, self).__init__()
     self.reason = reason
 
   def __str__(self):
@@ -52,10 +76,24 @@ class NoSuchProjectError(Exception):
   """A specified project does not exist in the work tree.
   """
   def __init__(self, name=None):
+    super(NoSuchProjectError, self).__init__()
     self.name = name
 
   def __str__(self):
-    if self.Name is None:
+    if self.name is None:
+      return 'in current directory'
+    return self.name
+
+
+class InvalidProjectGroupsError(Exception):
+  """A specified project is not suitable for the specified groups
+  """
+  def __init__(self, name=None):
+    super(InvalidProjectGroupsError, self).__init__()
+    self.name = name
+
+  def __str__(self):
+    if self.name is None:
       return 'in current directory'
     return self.name
 
@@ -64,5 +102,12 @@ class RepoChangedException(Exception):
      repo or manifest repositories.  In this special case we must
      use exec to re-execute repo with the new code and manifest.
   """
-  def __init__(self, extra_args=[]):
-    self.extra_args = extra_args
+  def __init__(self, extra_args=None):
+    super(RepoChangedException, self).__init__()
+    self.extra_args = extra_args or []
+
+class HookError(Exception):
+  """Thrown if a 'repo-hook' could not be run.
+
+  The common case is that the file wasn't present when we tried to run it.
+  """

event_log.py

@@ -0,0 +1,176 @@
+#
+# Copyright (C) 2017 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+
+import json
+import multiprocessing
+
+TASK_COMMAND = 'command'
+TASK_SYNC_NETWORK = 'sync-network'
+TASK_SYNC_LOCAL = 'sync-local'
+
+class EventLog(object):
+  """Event log that records events that occurred during a repo invocation.
+
+  Events are written to the log as a consecutive JSON entries, one per line.
+  Each entry contains the following keys:
+  - id: A ('RepoOp', ID) tuple, suitable for storing in a datastore.
+        The ID is only unique for the invocation of the repo command.
+  - name: Name of the object being operated upon.
+  - task_name: The task that was performed.
+  - start: Timestamp of when the operation started.
+  - finish: Timestamp of when the operation finished.
+  - success: Boolean indicating if the operation was successful.
+  - try_count: A counter indicating the try count of this task.
+
+  Optionally:
+  - parent: A ('RepoOp', ID) tuple indicating the parent event for nested
+            events.
+
+  Valid task_names include:
+  - command: The invocation of a subcommand.
+  - sync-network: The network component of a sync command.
+  - sync-local: The local component of a sync command.
+
+  Specific tasks may include additional informational properties.
+  """
+
+  def __init__(self):
+    """Initializes the event log."""
+    self._log = []
+    self._parent = None
+
+  def Add(self, name, task_name, start, finish=None, success=None,
+          try_count=1, kind='RepoOp'):
+    """Add an event to the log.
+
+    Args:
+      name: Name of the object being operated upon.
+      task_name: A sub-task that was performed for name.
+      start: Timestamp of when the operation started.
+      finish: Timestamp of when the operation finished.
+      success: Boolean indicating if the operation was successful.
+      try_count: A counter indicating the try count of this task.
+      kind: The kind of the object for the unique identifier.
+
+    Returns:
+      A dictionary of the event added to the log.
+    """
+    event = {
+        'id': (kind, _NextEventId()),
+        'name': name,
+        'task_name': task_name,
+        'start_time': start,
+        'try': try_count,
+    }
+
+    if self._parent:
+      event['parent'] = self._parent['id']
+
+    if success is not None or finish is not None:
+        self.FinishEvent(event, finish, success)
+
+    self._log.append(event)
+    return event
+
+  def AddSync(self, project, task_name, start, finish, success):
+    """Add a event to the log for a sync command.
+
+    Args:
+      project: Project being synced.
+      task_name: A sub-task that was performed for name.
+                 One of (TASK_SYNC_NETWORK, TASK_SYNC_LOCAL)
+      start: Timestamp of when the operation started.
+      finish: Timestamp of when the operation finished.
+      success: Boolean indicating if the operation was successful.
+
+    Returns:
+      A dictionary of the event added to the log.
+    """
+    event = self.Add(project.relpath, task_name, start, finish, success)
+    if event is not None:
+      event['project'] = project.name
+      if project.revisionExpr:
+        event['revision'] = project.revisionExpr
+      if project.remote.url:
+        event['project_url'] = project.remote.url
+      if project.remote.fetchUrl:
+        event['remote_url'] = project.remote.fetchUrl
+      try:
+        event['git_hash'] = project.GetCommitRevisionId()
+      except Exception:
+        pass
+    return event
+
+  def GetStatusString(self, success):
+    """Converst a boolean success to a status string.
+
+    Args:
+      success: Boolean indicating if the operation was successful.
+
+    Returns:
+      status string.
+    """
+    return 'pass' if success else 'fail'
+
+  def FinishEvent(self, event, finish, success):
+    """Finishes an incomplete event.
+
+    Args:
+      event: An event that has been added to the log.
+      finish: Timestamp of when the operation finished.
+      success: Boolean indicating if the operation was successful.
+
+    Returns:
+      A dictionary of the event added to the log.
+    """
+    event['status'] =  self.GetStatusString(success)
+    event['finish_time'] = finish
+    return event
+
+  def SetParent(self, event):
+    """Set a parent event for all new entities.
+
+    Args:
+      event: The event to use as a parent.
+    """
+    self._parent = event
+
+  def Write(self, filename):
+    """Writes the log out to a file.
+
+    Args:
+      filename: The file to write the log to.
+    """
+    with open(filename, 'w+') as f:
+      for e in self._log:
+        json.dump(e, f, sort_keys=True)
+        f.write('\n')
+
+
+# An integer id that is unique across this invocation of the program.
+_EVENT_ID = multiprocessing.Value('i', 1)
+
+def _NextEventId():
+  """Helper function for grabbing the next unique id.
+
+  Returns:
+    A unique, to this invocation of the program, integer id.
+  """
+  with _EVENT_ID.get_lock():
+    val = _EVENT_ID.value
+    _EVENT_ID.value += 1
+  return val

froofle/protobuf/descriptor.py

@@ -1,433 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-# TODO(robinson): We probably need to provide deep-copy methods for
-# descriptor types.  When a FieldDescriptor is passed into
-# Descriptor.__init__(), we should make a deep copy and then set
-# containing_type on it.  Alternatively, we could just get
-# rid of containing_type (iit's not needed for reflection.py, at least).
-#
-# TODO(robinson): Print method?
-#
-# TODO(robinson): Useful __repr__?
-
-"""Descriptors essentially contain exactly the information found in a .proto
-file, in types that make this information accessible in Python.
-"""
-
-__author__ = 'robinson@google.com (Will Robinson)'
-
-class DescriptorBase(object):
-
-  """Descriptors base class.
-
-  This class is the base of all descriptor classes. It provides common options
-  related functionaility.
-  """
-
-  def __init__(self, options, options_class_name):
-    """Initialize the descriptor given its options message and the name of the
-    class of the options message. The name of the class is required in case
-    the options message is None and has to be created.
-    """
-    self._options = options
-    self._options_class_name = options_class_name
-
-  def GetOptions(self):
-    """Retrieves descriptor options.
-
-    This method returns the options set or creates the default options for the
-    descriptor.
-    """
-    if self._options:
-      return self._options
-    from froofle.protobuf import descriptor_pb2
-    try:
-      options_class = getattr(descriptor_pb2, self._options_class_name)
-    except AttributeError:
-      raise RuntimeError('Unknown options class name %s!' %
-                         (self._options_class_name))
-    self._options = options_class()
-    return self._options
-
-
-class Descriptor(DescriptorBase):
-
-  """Descriptor for a protocol message type.
-
-  A Descriptor instance has the following attributes:
-
-    name: (str) Name of this protocol message type.
-    full_name: (str) Fully-qualified name of this protocol message type,
-      which will include protocol "package" name and the name of any
-      enclosing types.
-
-    filename: (str) Name of the .proto file containing this message.
-
-    containing_type: (Descriptor) Reference to the descriptor of the
-      type containing us, or None if we have no containing type.
-
-    fields: (list of FieldDescriptors) Field descriptors for all
-      fields in this type.
-    fields_by_number: (dict int -> FieldDescriptor) Same FieldDescriptor
-      objects as in |fields|, but indexed by "number" attribute in each
-      FieldDescriptor.
-    fields_by_name: (dict str -> FieldDescriptor) Same FieldDescriptor
-      objects as in |fields|, but indexed by "name" attribute in each
-      FieldDescriptor.
-
-    nested_types: (list of Descriptors) Descriptor references
-      for all protocol message types nested within this one.
-    nested_types_by_name: (dict str -> Descriptor) Same Descriptor
-      objects as in |nested_types|, but indexed by "name" attribute
-      in each Descriptor.
-
-    enum_types: (list of EnumDescriptors) EnumDescriptor references
-      for all enums contained within this type.
-    enum_types_by_name: (dict str ->EnumDescriptor) Same EnumDescriptor
-      objects as in |enum_types|, but indexed by "name" attribute
-      in each EnumDescriptor.
-    enum_values_by_name: (dict str -> EnumValueDescriptor) Dict mapping
-      from enum value name to EnumValueDescriptor for that value.
-
-    extensions: (list of FieldDescriptor) All extensions defined directly
-      within this message type (NOT within a nested type).
-    extensions_by_name: (dict, string -> FieldDescriptor) Same FieldDescriptor
-      objects as |extensions|, but indexed by "name" attribute of each
-      FieldDescriptor.
-
-    options: (descriptor_pb2.MessageOptions) Protocol message options or None
-      to use default message options.
-  """
-
-  def __init__(self, name, full_name, filename, containing_type,
-               fields, nested_types, enum_types, extensions, options=None):
-    """Arguments to __init__() are as described in the description
-    of Descriptor fields above.
-    """
-    super(Descriptor, self).__init__(options, 'MessageOptions')
-    self.name = name
-    self.full_name = full_name
-    self.filename = filename
-    self.containing_type = containing_type
-
-    # We have fields in addition to fields_by_name and fields_by_number,
-    # so that:
-    #   1. Clients can index fields by "order in which they're listed."
-    #   2. Clients can easily iterate over all fields with the terse
-    #      syntax: for f in descriptor.fields: ...
-    self.fields = fields
-    for field in self.fields:
-      field.containing_type = self
-    self.fields_by_number = dict((f.number, f) for f in fields)
-    self.fields_by_name = dict((f.name, f) for f in fields)
-
-    self.nested_types = nested_types
-    self.nested_types_by_name = dict((t.name, t) for t in nested_types)
-
-    self.enum_types = enum_types
-    for enum_type in self.enum_types:
-      enum_type.containing_type = self
-    self.enum_types_by_name = dict((t.name, t) for t in enum_types)
-    self.enum_values_by_name = dict(
-        (v.name, v) for t in enum_types for v in t.values)
-
-    self.extensions = extensions
-    for extension in self.extensions:
-      extension.extension_scope = self
-    self.extensions_by_name = dict((f.name, f) for f in extensions)
-
-
-# TODO(robinson): We should have aggressive checking here,
-# for example:
-#   * If you specify a repeated field, you should not be allowed
-#     to specify a default value.
-#   * [Other examples here as needed].
-#
-# TODO(robinson): for this and other *Descriptor classes, we
-# might also want to lock things down aggressively (e.g.,
-# prevent clients from setting the attributes).  Having
-# stronger invariants here in general will reduce the number
-# of runtime checks we must do in reflection.py...
-class FieldDescriptor(DescriptorBase):
-
-  """Descriptor for a single field in a .proto file.
-
-  A FieldDescriptor instance has the following attriubtes:
-
-    name: (str) Name of this field, exactly as it appears in .proto.
-    full_name: (str) Name of this field, including containing scope.  This is
-      particularly relevant for extensions.
-    index: (int) Dense, 0-indexed index giving the order that this
-      field textually appears within its message in the .proto file.
-    number: (int) Tag number declared for this field in the .proto file.
-
-    type: (One of the TYPE_* constants below) Declared type.
-    cpp_type: (One of the CPPTYPE_* constants below) C++ type used to
-      represent this field.
-
-    label: (One of the LABEL_* constants below) Tells whether this
-      field is optional, required, or repeated.
-    default_value: (Varies) Default value of this field.  Only
-      meaningful for non-repeated scalar fields.  Repeated fields
-      should always set this to [], and non-repeated composite
-      fields should always set this to None.
-
-    containing_type: (Descriptor) Descriptor of the protocol message
-      type that contains this field.  Set by the Descriptor constructor
-      if we're passed into one.
-      Somewhat confusingly, for extension fields, this is the
-      descriptor of the EXTENDED message, not the descriptor
-      of the message containing this field.  (See is_extension and
-      extension_scope below).
-    message_type: (Descriptor) If a composite field, a descriptor
-      of the message type contained in this field.  Otherwise, this is None.
-    enum_type: (EnumDescriptor) If this field contains an enum, a
-      descriptor of that enum.  Otherwise, this is None.
-
-    is_extension: True iff this describes an extension field.
-    extension_scope: (Descriptor) Only meaningful if is_extension is True.
-      Gives the message that immediately contains this extension field.
-      Will be None iff we're a top-level (file-level) extension field.
-
-    options: (descriptor_pb2.FieldOptions) Protocol message field options or
-      None to use default field options.
-  """
-
-  # Must be consistent with C++ FieldDescriptor::Type enum in
-  # descriptor.h.
-  #
-  # TODO(robinson): Find a way to eliminate this repetition.
-  TYPE_DOUBLE         = 1
-  TYPE_FLOAT          = 2
-  TYPE_INT64          = 3
-  TYPE_UINT64         = 4
-  TYPE_INT32          = 5
-  TYPE_FIXED64        = 6
-  TYPE_FIXED32        = 7
-  TYPE_BOOL           = 8
-  TYPE_STRING         = 9
-  TYPE_GROUP          = 10
-  TYPE_MESSAGE        = 11
-  TYPE_BYTES          = 12
-  TYPE_UINT32         = 13
-  TYPE_ENUM           = 14
-  TYPE_SFIXED32       = 15
-  TYPE_SFIXED64       = 16
-  TYPE_SINT32         = 17
-  TYPE_SINT64         = 18
-  MAX_TYPE            = 18
-
-  # Must be consistent with C++ FieldDescriptor::CppType enum in
-  # descriptor.h.
-  #
-  # TODO(robinson): Find a way to eliminate this repetition.
-  CPPTYPE_INT32       = 1
-  CPPTYPE_INT64       = 2
-  CPPTYPE_UINT32      = 3
-  CPPTYPE_UINT64      = 4
-  CPPTYPE_DOUBLE      = 5
-  CPPTYPE_FLOAT       = 6
-  CPPTYPE_BOOL        = 7
-  CPPTYPE_ENUM        = 8
-  CPPTYPE_STRING      = 9
-  CPPTYPE_MESSAGE     = 10
-  MAX_CPPTYPE         = 10
-
-  # Must be consistent with C++ FieldDescriptor::Label enum in
-  # descriptor.h.
-  #
-  # TODO(robinson): Find a way to eliminate this repetition.
-  LABEL_OPTIONAL      = 1
-  LABEL_REQUIRED      = 2
-  LABEL_REPEATED      = 3
-  MAX_LABEL           = 3
-
-  def __init__(self, name, full_name, index, number, type, cpp_type, label,
-               default_value, message_type, enum_type, containing_type,
-               is_extension, extension_scope, options=None):
-    """The arguments are as described in the description of FieldDescriptor
-    attributes above.
-
-    Note that containing_type may be None, and may be set later if necessary
-    (to deal with circular references between message types, for example).
-    Likewise for extension_scope.
-    """
-    super(FieldDescriptor, self).__init__(options, 'FieldOptions')
-    self.name = name
-    self.full_name = full_name
-    self.index = index
-    self.number = number
-    self.type = type
-    self.cpp_type = cpp_type
-    self.label = label
-    self.default_value = default_value
-    self.containing_type = containing_type
-    self.message_type = message_type
-    self.enum_type = enum_type
-    self.is_extension = is_extension
-    self.extension_scope = extension_scope
-
-
-class EnumDescriptor(DescriptorBase):
-
-  """Descriptor for an enum defined in a .proto file.
-
-  An EnumDescriptor instance has the following attributes:
-
-    name: (str) Name of the enum type.
-    full_name: (str) Full name of the type, including package name
-      and any enclosing type(s).
-    filename: (str) Name of the .proto file in which this appears.
-
-    values: (list of EnumValueDescriptors) List of the values
-      in this enum.
-    values_by_name: (dict str -> EnumValueDescriptor) Same as |values|,
-      but indexed by the "name" field of each EnumValueDescriptor.
-    values_by_number: (dict int -> EnumValueDescriptor) Same as |values|,
-      but indexed by the "number" field of each EnumValueDescriptor.
-    containing_type: (Descriptor) Descriptor of the immediate containing
-      type of this enum, or None if this is an enum defined at the
-      top level in a .proto file.  Set by Descriptor's constructor
-      if we're passed into one.
-    options: (descriptor_pb2.EnumOptions) Enum options message or
-      None to use default enum options.
-  """
-
-  def __init__(self, name, full_name, filename, values,
-               containing_type=None, options=None):
-    """Arguments are as described in the attribute description above."""
-    super(EnumDescriptor, self).__init__(options, 'EnumOptions')
-    self.name = name
-    self.full_name = full_name
-    self.filename = filename
-    self.values = values
-    for value in self.values:
-      value.type = self
-    self.values_by_name = dict((v.name, v) for v in values)
-    self.values_by_number = dict((v.number, v) for v in values)
-    self.containing_type = containing_type
-
-
-class EnumValueDescriptor(DescriptorBase):
-
-  """Descriptor for a single value within an enum.
-
-    name: (str) Name of this value.
-    index: (int) Dense, 0-indexed index giving the order that this
-      value appears textually within its enum in the .proto file.
-    number: (int) Actual number assigned to this enum value.
-    type: (EnumDescriptor) EnumDescriptor to which this value
-      belongs.  Set by EnumDescriptor's constructor if we're
-      passed into one.
-    options: (descriptor_pb2.EnumValueOptions) Enum value options message or
-      None to use default enum value options options.
-  """
-
-  def __init__(self, name, index, number, type=None, options=None):
-    """Arguments are as described in the attribute description above."""
-    super(EnumValueDescriptor, self).__init__(options, 'EnumValueOptions')
-    self.name = name
-    self.index = index
-    self.number = number
-    self.type = type
-
-
-class ServiceDescriptor(DescriptorBase):
-
-  """Descriptor for a service.
-
-    name: (str) Name of the service.
-    full_name: (str) Full name of the service, including package name.
-    index: (int) 0-indexed index giving the order that this services
-      definition appears withing the .proto file.
-    methods: (list of MethodDescriptor) List of methods provided by this
-      service.
-    options: (descriptor_pb2.ServiceOptions) Service options message or
-      None to use default service options.
-  """
-
-  def __init__(self, name, full_name, index, methods, options=None):
-    super(ServiceDescriptor, self).__init__(options, 'ServiceOptions')
-    self.name = name
-    self.full_name = full_name
-    self.index = index
-    self.methods = methods
-    # Set the containing service for each method in this service.
-    for method in self.methods:
-      method.containing_service = self
-
-  def FindMethodByName(self, name):
-    """Searches for the specified method, and returns its descriptor."""
-    for method in self.methods:
-      if name == method.name:
-        return method
-    return None
-
-
-class MethodDescriptor(DescriptorBase):
-
-  """Descriptor for a method in a service.
-
-  name: (str) Name of the method within the service.
-  full_name: (str) Full name of method.
-  index: (int) 0-indexed index of the method inside the service.
-  containing_service: (ServiceDescriptor) The service that contains this
-    method.
-  input_type: The descriptor of the message that this method accepts.
-  output_type: The descriptor of the message that this method returns.
-  options: (descriptor_pb2.MethodOptions) Method options message or
-    None to use default method options.
-  """
-
-  def __init__(self, name, full_name, index, containing_service,
-               input_type, output_type, options=None):
-    """The arguments are as described in the description of MethodDescriptor
-    attributes above.
-
-    Note that containing_service may be None, and may be set later if necessary.
-    """
-    super(MethodDescriptor, self).__init__(options, 'MethodOptions')
-    self.name = name
-    self.full_name = full_name
-    self.index = index
-    self.containing_service = containing_service
-    self.input_type = input_type
-    self.output_type = output_type
-
-
-def _ParseOptions(message, string):
-  """Parses serialized options.
-
-  This helper function is used to parse serialized options in generated
-  proto2 files. It must not be used outside proto2.
-  """
-  message.ParseFromString(string)
-  return message;

froofle/protobuf/descriptor_pb2.py

@@ -1,950 +0,0 @@
-#!/usr/bin/python2.4
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-
-from froofle.protobuf import descriptor
-from froofle.protobuf import message
-from froofle.protobuf import reflection
-from froofle.protobuf import service
-from froofle.protobuf import service_reflection
-
-
-_FIELDDESCRIPTORPROTO_TYPE = descriptor.EnumDescriptor(
-  name='Type',
-  full_name='froofle.protobuf.FieldDescriptorProto.Type',
-  filename='Type',
-  values=[
-    descriptor.EnumValueDescriptor(
-      name='TYPE_DOUBLE', index=0, number=1,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_FLOAT', index=1, number=2,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_INT64', index=2, number=3,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_UINT64', index=3, number=4,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_INT32', index=4, number=5,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_FIXED64', index=5, number=6,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_FIXED32', index=6, number=7,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_BOOL', index=7, number=8,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_STRING', index=8, number=9,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_GROUP', index=9, number=10,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_MESSAGE', index=10, number=11,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_BYTES', index=11, number=12,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_UINT32', index=12, number=13,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_ENUM', index=13, number=14,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_SFIXED32', index=14, number=15,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_SFIXED64', index=15, number=16,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_SINT32', index=16, number=17,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='TYPE_SINT64', index=17, number=18,
-      options=None,
-      type=None),
-  ],
-  options=None,
-)
-
-_FIELDDESCRIPTORPROTO_LABEL = descriptor.EnumDescriptor(
-  name='Label',
-  full_name='froofle.protobuf.FieldDescriptorProto.Label',
-  filename='Label',
-  values=[
-    descriptor.EnumValueDescriptor(
-      name='LABEL_OPTIONAL', index=0, number=1,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='LABEL_REQUIRED', index=1, number=2,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='LABEL_REPEATED', index=2, number=3,
-      options=None,
-      type=None),
-  ],
-  options=None,
-)
-
-_FILEOPTIONS_OPTIMIZEMODE = descriptor.EnumDescriptor(
-  name='OptimizeMode',
-  full_name='froofle.protobuf.FileOptions.OptimizeMode',
-  filename='OptimizeMode',
-  values=[
-    descriptor.EnumValueDescriptor(
-      name='SPEED', index=0, number=1,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='CODE_SIZE', index=1, number=2,
-      options=None,
-      type=None),
-  ],
-  options=None,
-)
-
-_FIELDOPTIONS_CTYPE = descriptor.EnumDescriptor(
-  name='CType',
-  full_name='froofle.protobuf.FieldOptions.CType',
-  filename='CType',
-  values=[
-    descriptor.EnumValueDescriptor(
-      name='CORD', index=0, number=1,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='STRING_PIECE', index=1, number=2,
-      options=None,
-      type=None),
-  ],
-  options=None,
-)
-
-
-_FILEDESCRIPTORSET = descriptor.Descriptor(
-  name='FileDescriptorSet',
-  full_name='froofle.protobuf.FileDescriptorSet',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='file', full_name='froofle.protobuf.FileDescriptorSet.file', index=0,
-      number=1, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_FILEDESCRIPTORPROTO = descriptor.Descriptor(
-  name='FileDescriptorProto',
-  full_name='froofle.protobuf.FileDescriptorProto',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name', full_name='froofle.protobuf.FileDescriptorProto.name', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='package', full_name='froofle.protobuf.FileDescriptorProto.package', index=1,
-      number=2, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='dependency', full_name='froofle.protobuf.FileDescriptorProto.dependency', index=2,
-      number=3, type=9, cpp_type=9, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='message_type', full_name='froofle.protobuf.FileDescriptorProto.message_type', index=3,
-      number=4, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='enum_type', full_name='froofle.protobuf.FileDescriptorProto.enum_type', index=4,
-      number=5, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='service', full_name='froofle.protobuf.FileDescriptorProto.service', index=5,
-      number=6, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='extension', full_name='froofle.protobuf.FileDescriptorProto.extension', index=6,
-      number=7, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='options', full_name='froofle.protobuf.FileDescriptorProto.options', index=7,
-      number=8, type=11, cpp_type=10, label=1,
-      default_value=None,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_DESCRIPTORPROTO_EXTENSIONRANGE = descriptor.Descriptor(
-  name='ExtensionRange',
-  full_name='froofle.protobuf.DescriptorProto.ExtensionRange',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='start', full_name='froofle.protobuf.DescriptorProto.ExtensionRange.start', index=0,
-      number=1, type=5, cpp_type=1, label=1,
-      default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='end', full_name='froofle.protobuf.DescriptorProto.ExtensionRange.end', index=1,
-      number=2, type=5, cpp_type=1, label=1,
-      default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-_DESCRIPTORPROTO = descriptor.Descriptor(
-  name='DescriptorProto',
-  full_name='froofle.protobuf.DescriptorProto',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name', full_name='froofle.protobuf.DescriptorProto.name', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='field', full_name='froofle.protobuf.DescriptorProto.field', index=1,
-      number=2, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='extension', full_name='froofle.protobuf.DescriptorProto.extension', index=2,
-      number=6, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='nested_type', full_name='froofle.protobuf.DescriptorProto.nested_type', index=3,
-      number=3, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='enum_type', full_name='froofle.protobuf.DescriptorProto.enum_type', index=4,
-      number=4, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='extension_range', full_name='froofle.protobuf.DescriptorProto.extension_range', index=5,
-      number=5, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='options', full_name='froofle.protobuf.DescriptorProto.options', index=6,
-      number=7, type=11, cpp_type=10, label=1,
-      default_value=None,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_FIELDDESCRIPTORPROTO = descriptor.Descriptor(
-  name='FieldDescriptorProto',
-  full_name='froofle.protobuf.FieldDescriptorProto',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name', full_name='froofle.protobuf.FieldDescriptorProto.name', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='number', full_name='froofle.protobuf.FieldDescriptorProto.number', index=1,
-      number=3, type=5, cpp_type=1, label=1,
-      default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='label', full_name='froofle.protobuf.FieldDescriptorProto.label', index=2,
-      number=4, type=14, cpp_type=8, label=1,
-      default_value=1,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='type', full_name='froofle.protobuf.FieldDescriptorProto.type', index=3,
-      number=5, type=14, cpp_type=8, label=1,
-      default_value=1,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='type_name', full_name='froofle.protobuf.FieldDescriptorProto.type_name', index=4,
-      number=6, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='extendee', full_name='froofle.protobuf.FieldDescriptorProto.extendee', index=5,
-      number=2, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='default_value', full_name='froofle.protobuf.FieldDescriptorProto.default_value', index=6,
-      number=7, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='options', full_name='froofle.protobuf.FieldDescriptorProto.options', index=7,
-      number=8, type=11, cpp_type=10, label=1,
-      default_value=None,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-    _FIELDDESCRIPTORPROTO_TYPE,
-    _FIELDDESCRIPTORPROTO_LABEL,
-  ],
-  options=None)
-
-
-_ENUMDESCRIPTORPROTO = descriptor.Descriptor(
-  name='EnumDescriptorProto',
-  full_name='froofle.protobuf.EnumDescriptorProto',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name', full_name='froofle.protobuf.EnumDescriptorProto.name', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='value', full_name='froofle.protobuf.EnumDescriptorProto.value', index=1,
-      number=2, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='options', full_name='froofle.protobuf.EnumDescriptorProto.options', index=2,
-      number=3, type=11, cpp_type=10, label=1,
-      default_value=None,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_ENUMVALUEDESCRIPTORPROTO = descriptor.Descriptor(
-  name='EnumValueDescriptorProto',
-  full_name='froofle.protobuf.EnumValueDescriptorProto',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name', full_name='froofle.protobuf.EnumValueDescriptorProto.name', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='number', full_name='froofle.protobuf.EnumValueDescriptorProto.number', index=1,
-      number=2, type=5, cpp_type=1, label=1,
-      default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='options', full_name='froofle.protobuf.EnumValueDescriptorProto.options', index=2,
-      number=3, type=11, cpp_type=10, label=1,
-      default_value=None,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_SERVICEDESCRIPTORPROTO = descriptor.Descriptor(
-  name='ServiceDescriptorProto',
-  full_name='froofle.protobuf.ServiceDescriptorProto',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name', full_name='froofle.protobuf.ServiceDescriptorProto.name', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='method', full_name='froofle.protobuf.ServiceDescriptorProto.method', index=1,
-      number=2, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='options', full_name='froofle.protobuf.ServiceDescriptorProto.options', index=2,
-      number=3, type=11, cpp_type=10, label=1,
-      default_value=None,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_METHODDESCRIPTORPROTO = descriptor.Descriptor(
-  name='MethodDescriptorProto',
-  full_name='froofle.protobuf.MethodDescriptorProto',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name', full_name='froofle.protobuf.MethodDescriptorProto.name', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='input_type', full_name='froofle.protobuf.MethodDescriptorProto.input_type', index=1,
-      number=2, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='output_type', full_name='froofle.protobuf.MethodDescriptorProto.output_type', index=2,
-      number=3, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='options', full_name='froofle.protobuf.MethodDescriptorProto.options', index=3,
-      number=4, type=11, cpp_type=10, label=1,
-      default_value=None,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_FILEOPTIONS = descriptor.Descriptor(
-  name='FileOptions',
-  full_name='froofle.protobuf.FileOptions',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='java_package', full_name='froofle.protobuf.FileOptions.java_package', index=0,
-      number=1, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='java_outer_classname', full_name='froofle.protobuf.FileOptions.java_outer_classname', index=1,
-      number=8, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='java_multiple_files', full_name='froofle.protobuf.FileOptions.java_multiple_files', index=2,
-      number=10, type=8, cpp_type=7, label=1,
-      default_value=False,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='optimize_for', full_name='froofle.protobuf.FileOptions.optimize_for', index=3,
-      number=9, type=14, cpp_type=8, label=1,
-      default_value=2,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='uninterpreted_option', full_name='froofle.protobuf.FileOptions.uninterpreted_option', index=4,
-      number=999, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-    _FILEOPTIONS_OPTIMIZEMODE,
-  ],
-  options=None)
-
-
-_MESSAGEOPTIONS = descriptor.Descriptor(
-  name='MessageOptions',
-  full_name='froofle.protobuf.MessageOptions',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='message_set_wire_format', full_name='froofle.protobuf.MessageOptions.message_set_wire_format', index=0,
-      number=1, type=8, cpp_type=7, label=1,
-      default_value=False,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='uninterpreted_option', full_name='froofle.protobuf.MessageOptions.uninterpreted_option', index=1,
-      number=999, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_FIELDOPTIONS = descriptor.Descriptor(
-  name='FieldOptions',
-  full_name='froofle.protobuf.FieldOptions',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='ctype', full_name='froofle.protobuf.FieldOptions.ctype', index=0,
-      number=1, type=14, cpp_type=8, label=1,
-      default_value=1,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='experimental_map_key', full_name='froofle.protobuf.FieldOptions.experimental_map_key', index=1,
-      number=9, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='uninterpreted_option', full_name='froofle.protobuf.FieldOptions.uninterpreted_option', index=2,
-      number=999, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-    _FIELDOPTIONS_CTYPE,
-  ],
-  options=None)
-
-
-_ENUMOPTIONS = descriptor.Descriptor(
-  name='EnumOptions',
-  full_name='froofle.protobuf.EnumOptions',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='uninterpreted_option', full_name='froofle.protobuf.EnumOptions.uninterpreted_option', index=0,
-      number=999, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_ENUMVALUEOPTIONS = descriptor.Descriptor(
-  name='EnumValueOptions',
-  full_name='froofle.protobuf.EnumValueOptions',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='uninterpreted_option', full_name='froofle.protobuf.EnumValueOptions.uninterpreted_option', index=0,
-      number=999, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_SERVICEOPTIONS = descriptor.Descriptor(
-  name='ServiceOptions',
-  full_name='froofle.protobuf.ServiceOptions',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='uninterpreted_option', full_name='froofle.protobuf.ServiceOptions.uninterpreted_option', index=0,
-      number=999, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_METHODOPTIONS = descriptor.Descriptor(
-  name='MethodOptions',
-  full_name='froofle.protobuf.MethodOptions',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='uninterpreted_option', full_name='froofle.protobuf.MethodOptions.uninterpreted_option', index=0,
-      number=999, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_UNINTERPRETEDOPTION_NAMEPART = descriptor.Descriptor(
-  name='NamePart',
-  full_name='froofle.protobuf.UninterpretedOption.NamePart',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name_part', full_name='froofle.protobuf.UninterpretedOption.NamePart.name_part', index=0,
-      number=1, type=9, cpp_type=9, label=2,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='is_extension', full_name='froofle.protobuf.UninterpretedOption.NamePart.is_extension', index=1,
-      number=2, type=8, cpp_type=7, label=2,
-      default_value=False,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-_UNINTERPRETEDOPTION = descriptor.Descriptor(
-  name='UninterpretedOption',
-  full_name='froofle.protobuf.UninterpretedOption',
-  filename='froofle/protobuf/descriptor.proto',
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='name', full_name='froofle.protobuf.UninterpretedOption.name', index=0,
-      number=2, type=11, cpp_type=10, label=3,
-      default_value=[],
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='identifier_value', full_name='froofle.protobuf.UninterpretedOption.identifier_value', index=1,
-      number=3, type=9, cpp_type=9, label=1,
-      default_value=unicode("", "utf-8"),
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='positive_int_value', full_name='froofle.protobuf.UninterpretedOption.positive_int_value', index=2,
-      number=4, type=4, cpp_type=4, label=1,
-      default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='negative_int_value', full_name='froofle.protobuf.UninterpretedOption.negative_int_value', index=3,
-      number=5, type=3, cpp_type=2, label=1,
-      default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='double_value', full_name='froofle.protobuf.UninterpretedOption.double_value', index=4,
-      number=6, type=1, cpp_type=5, label=1,
-      default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='string_value', full_name='froofle.protobuf.UninterpretedOption.string_value', index=5,
-      number=7, type=12, cpp_type=9, label=1,
-      default_value="",
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],  # TODO(robinson): Implement.
-  enum_types=[
-  ],
-  options=None)
-
-
-_FILEDESCRIPTORSET.fields_by_name['file'].message_type = _FILEDESCRIPTORPROTO
-_FILEDESCRIPTORPROTO.fields_by_name['message_type'].message_type = _DESCRIPTORPROTO
-_FILEDESCRIPTORPROTO.fields_by_name['enum_type'].message_type = _ENUMDESCRIPTORPROTO
-_FILEDESCRIPTORPROTO.fields_by_name['service'].message_type = _SERVICEDESCRIPTORPROTO
-_FILEDESCRIPTORPROTO.fields_by_name['extension'].message_type = _FIELDDESCRIPTORPROTO
-_FILEDESCRIPTORPROTO.fields_by_name['options'].message_type = _FILEOPTIONS
-_DESCRIPTORPROTO.fields_by_name['field'].message_type = _FIELDDESCRIPTORPROTO
-_DESCRIPTORPROTO.fields_by_name['extension'].message_type = _FIELDDESCRIPTORPROTO
-_DESCRIPTORPROTO.fields_by_name['nested_type'].message_type = _DESCRIPTORPROTO
-_DESCRIPTORPROTO.fields_by_name['enum_type'].message_type = _ENUMDESCRIPTORPROTO
-_DESCRIPTORPROTO.fields_by_name['extension_range'].message_type = _DESCRIPTORPROTO_EXTENSIONRANGE
-_DESCRIPTORPROTO.fields_by_name['options'].message_type = _MESSAGEOPTIONS
-_FIELDDESCRIPTORPROTO.fields_by_name['label'].enum_type = _FIELDDESCRIPTORPROTO_LABEL
-_FIELDDESCRIPTORPROTO.fields_by_name['type'].enum_type = _FIELDDESCRIPTORPROTO_TYPE
-_FIELDDESCRIPTORPROTO.fields_by_name['options'].message_type = _FIELDOPTIONS
-_ENUMDESCRIPTORPROTO.fields_by_name['value'].message_type = _ENUMVALUEDESCRIPTORPROTO
-_ENUMDESCRIPTORPROTO.fields_by_name['options'].message_type = _ENUMOPTIONS
-_ENUMVALUEDESCRIPTORPROTO.fields_by_name['options'].message_type = _ENUMVALUEOPTIONS
-_SERVICEDESCRIPTORPROTO.fields_by_name['method'].message_type = _METHODDESCRIPTORPROTO
-_SERVICEDESCRIPTORPROTO.fields_by_name['options'].message_type = _SERVICEOPTIONS
-_METHODDESCRIPTORPROTO.fields_by_name['options'].message_type = _METHODOPTIONS
-_FILEOPTIONS.fields_by_name['optimize_for'].enum_type = _FILEOPTIONS_OPTIMIZEMODE
-_FILEOPTIONS.fields_by_name['uninterpreted_option'].message_type = _UNINTERPRETEDOPTION
-_MESSAGEOPTIONS.fields_by_name['uninterpreted_option'].message_type = _UNINTERPRETEDOPTION
-_FIELDOPTIONS.fields_by_name['ctype'].enum_type = _FIELDOPTIONS_CTYPE
-_FIELDOPTIONS.fields_by_name['uninterpreted_option'].message_type = _UNINTERPRETEDOPTION
-_ENUMOPTIONS.fields_by_name['uninterpreted_option'].message_type = _UNINTERPRETEDOPTION
-_ENUMVALUEOPTIONS.fields_by_name['uninterpreted_option'].message_type = _UNINTERPRETEDOPTION
-_SERVICEOPTIONS.fields_by_name['uninterpreted_option'].message_type = _UNINTERPRETEDOPTION
-_METHODOPTIONS.fields_by_name['uninterpreted_option'].message_type = _UNINTERPRETEDOPTION
-_UNINTERPRETEDOPTION.fields_by_name['name'].message_type = _UNINTERPRETEDOPTION_NAMEPART
-
-class FileDescriptorSet(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _FILEDESCRIPTORSET
-
-class FileDescriptorProto(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _FILEDESCRIPTORPROTO
-
-class DescriptorProto(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  
-  class ExtensionRange(message.Message):
-    __metaclass__ = reflection.GeneratedProtocolMessageType
-    DESCRIPTOR = _DESCRIPTORPROTO_EXTENSIONRANGE
-  DESCRIPTOR = _DESCRIPTORPROTO
-
-class FieldDescriptorProto(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _FIELDDESCRIPTORPROTO
-
-class EnumDescriptorProto(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _ENUMDESCRIPTORPROTO
-
-class EnumValueDescriptorProto(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _ENUMVALUEDESCRIPTORPROTO
-
-class ServiceDescriptorProto(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _SERVICEDESCRIPTORPROTO
-
-class MethodDescriptorProto(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _METHODDESCRIPTORPROTO
-
-class FileOptions(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _FILEOPTIONS
-
-class MessageOptions(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _MESSAGEOPTIONS
-
-class FieldOptions(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _FIELDOPTIONS
-
-class EnumOptions(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _ENUMOPTIONS
-
-class EnumValueOptions(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _ENUMVALUEOPTIONS
-
-class ServiceOptions(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _SERVICEOPTIONS
-
-class MethodOptions(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _METHODOPTIONS
-
-class UninterpretedOption(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  
-  class NamePart(message.Message):
-    __metaclass__ = reflection.GeneratedProtocolMessageType
-    DESCRIPTOR = _UNINTERPRETEDOPTION_NAMEPART
-  DESCRIPTOR = _UNINTERPRETEDOPTION
-

froofle/protobuf/internal/decoder.py

@@ -1,209 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""Class for decoding protocol buffer primitives.
-
-Contains the logic for decoding every logical protocol field type
-from one of the 5 physical wire types.
-"""
-
-__author__ = 'robinson@google.com (Will Robinson)'
-
-import struct
-from froofle.protobuf import message
-from froofle.protobuf.internal import input_stream
-from froofle.protobuf.internal import wire_format
-
-
-
-# Note that much of this code is ported from //net/proto/ProtocolBuffer, and
-# that the interface is strongly inspired by WireFormat from the C++ proto2
-# implementation.
-
-
-class Decoder(object):
-
-  """Decodes logical protocol buffer fields from the wire."""
-
-  def __init__(self, s):
-    """Initializes the decoder to read from s.
-
-    Args:
-      s: An immutable sequence of bytes, which must be accessible
-        via the Python buffer() primitive (i.e., buffer(s)).
-    """
-    self._stream = input_stream.InputStream(s)
-
-  def EndOfStream(self):
-    """Returns true iff we've reached the end of the bytes we're reading."""
-    return self._stream.EndOfStream()
-
-  def Position(self):
-    """Returns the 0-indexed position in |s|."""
-    return self._stream.Position()
-
-  def ReadFieldNumberAndWireType(self):
-    """Reads a tag from the wire. Returns a (field_number, wire_type) pair."""
-    tag_and_type = self.ReadUInt32()
-    return wire_format.UnpackTag(tag_and_type)
-
-  def SkipBytes(self, bytes):
-    """Skips the specified number of bytes on the wire."""
-    self._stream.SkipBytes(bytes)
-
-  # Note that the Read*() methods below are not exactly symmetrical with the
-  # corresponding Encoder.Append*() methods.  Those Encoder methods first
-  # encode a tag, but the Read*() methods below assume that the tag has already
-  # been read, and that the client wishes to read a field of the specified type
-  # starting at the current position.
-
-  def ReadInt32(self):
-    """Reads and returns a signed, varint-encoded, 32-bit integer."""
-    return self._stream.ReadVarint32()
-
-  def ReadInt64(self):
-    """Reads and returns a signed, varint-encoded, 64-bit integer."""
-    return self._stream.ReadVarint64()
-
-  def ReadUInt32(self):
-    """Reads and returns an signed, varint-encoded, 32-bit integer."""
-    return self._stream.ReadVarUInt32()
-
-  def ReadUInt64(self):
-    """Reads and returns an signed, varint-encoded,64-bit integer."""
-    return self._stream.ReadVarUInt64()
-
-  def ReadSInt32(self):
-    """Reads and returns a signed, zigzag-encoded, varint-encoded,
-    32-bit integer."""
-    return wire_format.ZigZagDecode(self._stream.ReadVarUInt32())
-
-  def ReadSInt64(self):
-    """Reads and returns a signed, zigzag-encoded, varint-encoded,
-    64-bit integer."""
-    return wire_format.ZigZagDecode(self._stream.ReadVarUInt64())
-
-  def ReadFixed32(self):
-    """Reads and returns an unsigned, fixed-width, 32-bit integer."""
-    return self._stream.ReadLittleEndian32()
-
-  def ReadFixed64(self):
-    """Reads and returns an unsigned, fixed-width, 64-bit integer."""
-    return self._stream.ReadLittleEndian64()
-
-  def ReadSFixed32(self):
-    """Reads and returns a signed, fixed-width, 32-bit integer."""
-    value = self._stream.ReadLittleEndian32()
-    if value >= (1 << 31):
-      value -= (1 << 32)
-    return value
-
-  def ReadSFixed64(self):
-    """Reads and returns a signed, fixed-width, 64-bit integer."""
-    value = self._stream.ReadLittleEndian64()
-    if value >= (1 << 63):
-      value -= (1 << 64)
-    return value
-
-  def ReadFloat(self):
-    """Reads and returns a 4-byte floating-point number."""
-    serialized = self._stream.ReadBytes(4)
-    return struct.unpack('f', serialized)[0]
-
-  def ReadDouble(self):
-    """Reads and returns an 8-byte floating-point number."""
-    serialized = self._stream.ReadBytes(8)
-    return struct.unpack('d', serialized)[0]
-
-  def ReadBool(self):
-    """Reads and returns a bool."""
-    i = self._stream.ReadVarUInt32()
-    return bool(i)
-
-  def ReadEnum(self):
-    """Reads and returns an enum value."""
-    return self._stream.ReadVarUInt32()
-
-  def ReadString(self):
-    """Reads and returns a length-delimited string."""
-    bytes = self.ReadBytes()
-    return unicode(bytes, 'utf-8')
-
-  def ReadBytes(self):
-    """Reads and returns a length-delimited byte sequence."""
-    length = self._stream.ReadVarUInt32()
-    return self._stream.ReadBytes(length)
-
-  def ReadMessageInto(self, msg):
-    """Calls msg.MergeFromString() to merge
-    length-delimited serialized message data into |msg|.
-
-    REQUIRES: The decoder must be positioned at the serialized "length"
-      prefix to a length-delmiited serialized message.
-
-    POSTCONDITION: The decoder is positioned just after the
-      serialized message, and we have merged those serialized
-      contents into |msg|.
-    """
-    length = self._stream.ReadVarUInt32()
-    sub_buffer = self._stream.GetSubBuffer(length)
-    num_bytes_used = msg.MergeFromString(sub_buffer)
-    if num_bytes_used != length:
-      raise message.DecodeError(
-          'Submessage told to deserialize from %d-byte encoding, '
-          'but used only %d bytes' % (length, num_bytes_used))
-    self._stream.SkipBytes(num_bytes_used)
-
-  def ReadGroupInto(self, expected_field_number, group):
-    """Calls group.MergeFromString() to merge
-    END_GROUP-delimited serialized message data into |group|.
-    We'll raise an exception if we don't find an END_GROUP
-    tag immediately after the serialized message contents.
-
-    REQUIRES: The decoder is positioned just after the START_GROUP
-      tag for this group.
-
-    POSTCONDITION: The decoder is positioned just after the
-      END_GROUP tag for this group, and we have merged
-      the contents of the group into |group|.
-    """
-    sub_buffer = self._stream.GetSubBuffer()  # No a priori length limit.
-    num_bytes_used = group.MergeFromString(sub_buffer)
-    if num_bytes_used < 0:
-      raise message.DecodeError('Group message reported negative bytes read.')
-    self._stream.SkipBytes(num_bytes_used)
-    field_number, field_type = self.ReadFieldNumberAndWireType()
-    if field_type != wire_format.WIRETYPE_END_GROUP:
-      raise message.DecodeError('Group message did not end with an END_GROUP.')
-    if field_number != expected_field_number:
-      raise message.DecodeError('END_GROUP tag had field '
-                                'number %d, was expecting field number %d' % (
-          field_number, expected_field_number))
-    # We're now positioned just after the END_GROUP tag.  Perfect.

froofle/protobuf/internal/encoder.py

@@ -1,206 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""Class for encoding protocol message primitives.
-
-Contains the logic for encoding every logical protocol field type
-into one of the 5 physical wire types.
-"""
-
-__author__ = 'robinson@google.com (Will Robinson)'
-
-import struct
-from froofle.protobuf import message
-from froofle.protobuf.internal import wire_format
-from froofle.protobuf.internal import output_stream
-
-
-# Note that much of this code is ported from //net/proto/ProtocolBuffer, and
-# that the interface is strongly inspired by WireFormat from the C++ proto2
-# implementation.
-
-
-class Encoder(object):
-
-  """Encodes logical protocol buffer fields to the wire format."""
-
-  def __init__(self):
-    self._stream = output_stream.OutputStream()
-
-  def ToString(self):
-    """Returns all values encoded in this object as a string."""
-    return self._stream.ToString()
-
-  # All the Append*() methods below first append a tag+type pair to the buffer
-  # before appending the specified value.
-
-  def AppendInt32(self, field_number, value):
-    """Appends a 32-bit integer to our buffer, varint-encoded."""
-    self._AppendTag(field_number, wire_format.WIRETYPE_VARINT)
-    self._stream.AppendVarint32(value)
-
-  def AppendInt64(self, field_number, value):
-    """Appends a 64-bit integer to our buffer, varint-encoded."""
-    self._AppendTag(field_number, wire_format.WIRETYPE_VARINT)
-    self._stream.AppendVarint64(value)
-
-  def AppendUInt32(self, field_number, unsigned_value):
-    """Appends an unsigned 32-bit integer to our buffer, varint-encoded."""
-    self._AppendTag(field_number, wire_format.WIRETYPE_VARINT)
-    self._stream.AppendVarUInt32(unsigned_value)
-
-  def AppendUInt64(self, field_number, unsigned_value):
-    """Appends an unsigned 64-bit integer to our buffer, varint-encoded."""
-    self._AppendTag(field_number, wire_format.WIRETYPE_VARINT)
-    self._stream.AppendVarUInt64(unsigned_value)
-
-  def AppendSInt32(self, field_number, value):
-    """Appends a 32-bit integer to our buffer, zigzag-encoded and then
-    varint-encoded.
-    """
-    self._AppendTag(field_number, wire_format.WIRETYPE_VARINT)
-    zigzag_value = wire_format.ZigZagEncode(value)
-    self._stream.AppendVarUInt32(zigzag_value)
-
-  def AppendSInt64(self, field_number, value):
-    """Appends a 64-bit integer to our buffer, zigzag-encoded and then
-    varint-encoded.
-    """
-    self._AppendTag(field_number, wire_format.WIRETYPE_VARINT)
-    zigzag_value = wire_format.ZigZagEncode(value)
-    self._stream.AppendVarUInt64(zigzag_value)
-
-  def AppendFixed32(self, field_number, unsigned_value):
-    """Appends an unsigned 32-bit integer to our buffer, in little-endian
-    byte-order.
-    """
-    self._AppendTag(field_number, wire_format.WIRETYPE_FIXED32)
-    self._stream.AppendLittleEndian32(unsigned_value)
-
-  def AppendFixed64(self, field_number, unsigned_value):
-    """Appends an unsigned 64-bit integer to our buffer, in little-endian
-    byte-order.
-    """
-    self._AppendTag(field_number, wire_format.WIRETYPE_FIXED64)
-    self._stream.AppendLittleEndian64(unsigned_value)
-
-  def AppendSFixed32(self, field_number, value):
-    """Appends a signed 32-bit integer to our buffer, in little-endian
-    byte-order.
-    """
-    sign = (value & 0x80000000) and -1 or 0
-    if value >> 32 != sign:
-      raise message.EncodeError('SFixed32 out of range: %d' % value)
-    self._AppendTag(field_number, wire_format.WIRETYPE_FIXED32)
-    self._stream.AppendLittleEndian32(value & 0xffffffff)
-
-  def AppendSFixed64(self, field_number, value):
-    """Appends a signed 64-bit integer to our buffer, in little-endian
-    byte-order.
-    """
-    sign = (value & 0x8000000000000000) and -1 or 0
-    if value >> 64 != sign:
-      raise message.EncodeError('SFixed64 out of range: %d' % value)
-    self._AppendTag(field_number, wire_format.WIRETYPE_FIXED64)
-    self._stream.AppendLittleEndian64(value & 0xffffffffffffffff)
-
-  def AppendFloat(self, field_number, value):
-    """Appends a floating-point number to our buffer."""
-    self._AppendTag(field_number, wire_format.WIRETYPE_FIXED32)
-    self._stream.AppendRawBytes(struct.pack('f', value))
-
-  def AppendDouble(self, field_number, value):
-    """Appends a double-precision floating-point number to our buffer."""
-    self._AppendTag(field_number, wire_format.WIRETYPE_FIXED64)
-    self._stream.AppendRawBytes(struct.pack('d', value))
-
-  def AppendBool(self, field_number, value):
-    """Appends a boolean to our buffer."""
-    self.AppendInt32(field_number, value)
-
-  def AppendEnum(self, field_number, value):
-    """Appends an enum value to our buffer."""
-    self.AppendInt32(field_number, value)
-
-  def AppendString(self, field_number, value):
-    """Appends a length-prefixed unicode string, encoded as UTF-8 to our buffer,
-    with the length varint-encoded.
-    """
-    self.AppendBytes(field_number, value.encode('utf-8'))
-
-  def AppendBytes(self, field_number, value):
-    """Appends a length-prefixed sequence of bytes to our buffer, with the
-    length varint-encoded.
-    """
-    self._AppendTag(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
-    self._stream.AppendVarUInt32(len(value))
-    self._stream.AppendRawBytes(value)
-
-  # TODO(robinson): For AppendGroup() and AppendMessage(), we'd really like to
-  # avoid the extra string copy here.  We can do so if we widen the Message
-  # interface to be able to serialize to a stream in addition to a string.  The
-  # challenge when thinking ahead to the Python/C API implementation of Message
-  # is finding a stream-like Python thing to which we can write raw bytes
-  # from C.  I'm not sure such a thing exists(?).  (array.array is pretty much
-  # what we want, but it's not directly exposed in the Python/C API).
-
-  def AppendGroup(self, field_number, group):
-    """Appends a group to our buffer.
-    """
-    self._AppendTag(field_number, wire_format.WIRETYPE_START_GROUP)
-    self._stream.AppendRawBytes(group.SerializeToString())
-    self._AppendTag(field_number, wire_format.WIRETYPE_END_GROUP)
-
-  def AppendMessage(self, field_number, msg):
-    """Appends a nested message to our buffer.
-    """
-    self._AppendTag(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
-    self._stream.AppendVarUInt32(msg.ByteSize())
-    self._stream.AppendRawBytes(msg.SerializeToString())
-
-  def AppendMessageSetItem(self, field_number, msg):
-    """Appends an item using the message set wire format.
-
-    The message set message looks like this:
-      message MessageSet {
-        repeated group Item = 1 {
-          required int32 type_id = 2;
-          required string message = 3;
-        }
-      }
-    """
-    self._AppendTag(1, wire_format.WIRETYPE_START_GROUP)
-    self.AppendInt32(2, field_number)
-    self.AppendMessage(3, msg)
-    self._AppendTag(1, wire_format.WIRETYPE_END_GROUP)
-
-  def _AppendTag(self, field_number, wire_type):
-    """Appends a tag containing field number and wire type information."""
-    self._stream.AppendVarUInt32(wire_format.PackTag(field_number, wire_type))

froofle/protobuf/internal/input_stream.py

@@ -1,326 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""InputStream is the primitive interface for reading bits from the wire.
-
-All protocol buffer deserialization can be expressed in terms of
-the InputStream primitives provided here.
-"""
-
-__author__ = 'robinson@google.com (Will Robinson)'
-
-import struct
-from array import array
-from froofle.protobuf import message
-from froofle.protobuf.internal import wire_format
-
-
-# Note that much of this code is ported from //net/proto/ProtocolBuffer, and
-# that the interface is strongly inspired by CodedInputStream from the C++
-# proto2 implementation.
-
-
-class InputStreamBuffer(object):
-
-  """Contains all logic for reading bits, and dealing with stream position.
-
-  If an InputStream method ever raises an exception, the stream is left
-  in an indeterminate state and is not safe for further use.
-  """
-
-  def __init__(self, s):
-    # What we really want is something like array('B', s), where elements we
-    # read from the array are already given to us as one-byte integers.  BUT
-    # using array() instead of buffer() would force full string copies to result
-    # from each GetSubBuffer() call.
-    #
-    # So, if the N serialized bytes of a single protocol buffer object are
-    # split evenly between 2 child messages, and so on recursively, using
-    # array('B', s) instead of buffer() would incur an additional N*logN bytes
-    # copied during deserialization.
-    #
-    # The higher constant overhead of having to ord() for every byte we read
-    # from the buffer in _ReadVarintHelper() could definitely lead to worse
-    # performance in many real-world scenarios, even if the asymptotic
-    # complexity is better.  However, our real answer is that the mythical
-    # Python/C extension module output mode for the protocol compiler will
-    # be blazing-fast and will eliminate most use of this class anyway.
-    self._buffer = buffer(s)
-    self._pos = 0
-
-  def EndOfStream(self):
-    """Returns true iff we're at the end of the stream.
-    If this returns true, then a call to any other InputStream method
-    will raise an exception.
-    """
-    return self._pos >= len(self._buffer)
-
-  def Position(self):
-    """Returns the current position in the stream, or equivalently, the
-    number of bytes read so far.
-    """
-    return self._pos
-
-  def GetSubBuffer(self, size=None):
-    """Returns a sequence-like object that represents a portion of our
-    underlying sequence.
-
-    Position 0 in the returned object corresponds to self.Position()
-    in this stream.
-
-    If size is specified, then the returned object ends after the
-    next "size" bytes in this stream.  If size is not specified,
-    then the returned object ends at the end of this stream.
-
-    We guarantee that the returned object R supports the Python buffer
-    interface (and thus that the call buffer(R) will work).
-
-    Note that the returned buffer is read-only.
-
-    The intended use for this method is for nested-message and nested-group
-    deserialization, where we want to make a recursive MergeFromString()
-    call on the portion of the original sequence that contains the serialized
-    nested message.  (And we'd like to do so without making unnecessary string
-    copies).
-
-    REQUIRES: size is nonnegative.
-    """
-    # Note that buffer() doesn't perform any actual string copy.
-    if size is None:
-      return buffer(self._buffer, self._pos)
-    else:
-      if size < 0:
-        raise message.DecodeError('Negative size %d' % size)
-      return buffer(self._buffer, self._pos, size)
-
-  def SkipBytes(self, num_bytes):
-    """Skip num_bytes bytes ahead, or go to the end of the stream, whichever
-    comes first.
-
-    REQUIRES: num_bytes is nonnegative.
-    """
-    if num_bytes < 0:
-      raise message.DecodeError('Negative num_bytes %d' % num_bytes)
-    self._pos += num_bytes
-    self._pos = min(self._pos, len(self._buffer))
-
-  def ReadBytes(self, size):
-    """Reads up to 'size' bytes from the stream, stopping early
-    only if we reach the end of the stream.  Returns the bytes read
-    as a string.
-    """
-    if size < 0:
-      raise message.DecodeError('Negative size %d' % size)
-    s = (self._buffer[self._pos : self._pos + size])
-    self._pos += len(s)  # Only advance by the number of bytes actually read.
-    return s
-
-  def ReadLittleEndian32(self):
-    """Interprets the next 4 bytes of the stream as a little-endian
-    encoded, unsiged 32-bit integer, and returns that integer.
-    """
-    try:
-      i = struct.unpack(wire_format.FORMAT_UINT32_LITTLE_ENDIAN,
-                        self._buffer[self._pos : self._pos + 4])
-      self._pos += 4
-      return i[0]  # unpack() result is a 1-element tuple.
-    except struct.error, e:
-      raise message.DecodeError(e)
-
-  def ReadLittleEndian64(self):
-    """Interprets the next 8 bytes of the stream as a little-endian
-    encoded, unsiged 64-bit integer, and returns that integer.
-    """
-    try:
-      i = struct.unpack(wire_format.FORMAT_UINT64_LITTLE_ENDIAN,
-                        self._buffer[self._pos : self._pos + 8])
-      self._pos += 8
-      return i[0]  # unpack() result is a 1-element tuple.
-    except struct.error, e:
-      raise message.DecodeError(e)
-
-  def ReadVarint32(self):
-    """Reads a varint from the stream, interprets this varint
-    as a signed, 32-bit integer, and returns the integer.
-    """
-    i = self.ReadVarint64()
-    if not wire_format.INT32_MIN <= i <= wire_format.INT32_MAX:
-      raise message.DecodeError('Value out of range for int32: %d' % i)
-    return int(i)
-
-  def ReadVarUInt32(self):
-    """Reads a varint from the stream, interprets this varint
-    as an unsigned, 32-bit integer, and returns the integer.
-    """
-    i = self.ReadVarUInt64()
-    if i > wire_format.UINT32_MAX:
-      raise message.DecodeError('Value out of range for uint32: %d' % i)
-    return i
-
-  def ReadVarint64(self):
-    """Reads a varint from the stream, interprets this varint
-    as a signed, 64-bit integer, and returns the integer.
-    """
-    i = self.ReadVarUInt64()
-    if i > wire_format.INT64_MAX:
-      i -= (1 << 64)
-    return i
-
-  def ReadVarUInt64(self):
-    """Reads a varint from the stream, interprets this varint
-    as an unsigned, 64-bit integer, and returns the integer.
-    """
-    i = self._ReadVarintHelper()
-    if not 0 <= i <= wire_format.UINT64_MAX:
-      raise message.DecodeError('Value out of range for uint64: %d' % i)
-    return i
-
-  def _ReadVarintHelper(self):
-    """Helper for the various varint-reading methods above.
-    Reads an unsigned, varint-encoded integer from the stream and
-    returns this integer.
-
-    Does no bounds checking except to ensure that we read at most as many bytes
-    as could possibly be present in a varint-encoded 64-bit number.
-    """
-    result = 0
-    shift = 0
-    while 1:
-      if shift >= 64:
-        raise message.DecodeError('Too many bytes when decoding varint.')
-      try:
-        b = ord(self._buffer[self._pos])
-      except IndexError:
-        raise message.DecodeError('Truncated varint.')
-      self._pos += 1
-      result |= ((b & 0x7f) << shift)
-      shift += 7
-      if not (b & 0x80):
-        return result
-
-class InputStreamArray(object):
-  def __init__(self, s):
-    self._buffer = array('B', s)
-    self._pos = 0
-
-  def EndOfStream(self):
-    return self._pos >= len(self._buffer)
-
-  def Position(self):
-    return self._pos
-
-  def GetSubBuffer(self, size=None):
-    if size is None:
-      return self._buffer[self._pos : ].tostring()
-    else:
-      if size < 0:
-        raise message.DecodeError('Negative size %d' % size)
-      return self._buffer[self._pos : self._pos + size].tostring()
-
-  def SkipBytes(self, num_bytes):
-    if num_bytes < 0:
-      raise message.DecodeError('Negative num_bytes %d' % num_bytes)
-    self._pos += num_bytes
-    self._pos = min(self._pos, len(self._buffer))
-
-  def ReadBytes(self, size):
-    if size < 0:
-      raise message.DecodeError('Negative size %d' % size)
-    s = self._buffer[self._pos : self._pos + size].tostring()
-    self._pos += len(s)  # Only advance by the number of bytes actually read.
-    return s
-
-  def ReadLittleEndian32(self):
-    try:
-      i = struct.unpack(wire_format.FORMAT_UINT32_LITTLE_ENDIAN,
-                        self._buffer[self._pos : self._pos + 4])
-      self._pos += 4
-      return i[0]  # unpack() result is a 1-element tuple.
-    except struct.error, e:
-      raise message.DecodeError(e)
-
-  def ReadLittleEndian64(self):
-    try:
-      i = struct.unpack(wire_format.FORMAT_UINT64_LITTLE_ENDIAN,
-                        self._buffer[self._pos : self._pos + 8])
-      self._pos += 8
-      return i[0]  # unpack() result is a 1-element tuple.
-    except struct.error, e:
-      raise message.DecodeError(e)
-
-  def ReadVarint32(self):
-    i = self.ReadVarint64()
-    if not wire_format.INT32_MIN <= i <= wire_format.INT32_MAX:
-      raise message.DecodeError('Value out of range for int32: %d' % i)
-    return int(i)
-
-  def ReadVarUInt32(self):
-    i = self.ReadVarUInt64()
-    if i > wire_format.UINT32_MAX:
-      raise message.DecodeError('Value out of range for uint32: %d' % i)
-    return i
-
-  def ReadVarint64(self):
-    i = self.ReadVarUInt64()
-    if i > wire_format.INT64_MAX:
-      i -= (1 << 64)
-    return i
-
-  def ReadVarUInt64(self):
-    i = self._ReadVarintHelper()
-    if not 0 <= i <= wire_format.UINT64_MAX:
-      raise message.DecodeError('Value out of range for uint64: %d' % i)
-    return i
-
-  def _ReadVarintHelper(self):
-    result = 0
-    shift = 0
-    while 1:
-      if shift >= 64:
-        raise message.DecodeError('Too many bytes when decoding varint.')
-      try:
-        b = self._buffer[self._pos]
-      except IndexError:
-        raise message.DecodeError('Truncated varint.')
-      self._pos += 1
-      result |= ((b & 0x7f) << shift)
-      shift += 7
-      if not (b & 0x80):
-        return result
-
-try:
-  buffer("")
-  InputStream = InputStreamBuffer
-except NotImplementedError:
-  # Google App Engine: dev_appserver.py
-  InputStream = InputStreamArray
-except RuntimeError:
-  # Google App Engine: production
-  InputStream = InputStreamArray

froofle/protobuf/internal/message_listener.py

@@ -1,69 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""Defines a listener interface for observing certain
-state transitions on Message objects.
-
-Also defines a null implementation of this interface.
-"""
-
-__author__ = 'robinson@google.com (Will Robinson)'
-
-
-class MessageListener(object):
-
-  """Listens for transitions to nonempty and for invalidations of cached
-  byte sizes.  Meant to be registered via Message._SetListener().
-  """
-
-  def TransitionToNonempty(self):
-    """Called the *first* time that this message becomes nonempty.
-    Implementations are free (but not required) to call this method multiple
-    times after the message has become nonempty.
-    """
-    raise NotImplementedError
-
-  def ByteSizeDirty(self):
-    """Called *every* time the cached byte size value
-    for this object is invalidated (transitions from being
-    "clean" to "dirty").
-    """
-    raise NotImplementedError
-
-
-class NullMessageListener(object):
-
-  """No-op MessageListener implementation."""
-
-  def TransitionToNonempty(self):
-    pass
-
-  def ByteSizeDirty(self):
-    pass

froofle/protobuf/internal/output_stream.py

@@ -1,125 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""OutputStream is the primitive interface for sticking bits on the wire.
-
-All protocol buffer serialization can be expressed in terms of
-the OutputStream primitives provided here.
-"""
-
-__author__ = 'robinson@google.com (Will Robinson)'
-
-import array
-import struct
-from froofle.protobuf import message
-from froofle.protobuf.internal import wire_format
-
-
-
-# Note that much of this code is ported from //net/proto/ProtocolBuffer, and
-# that the interface is strongly inspired by CodedOutputStream from the C++
-# proto2 implementation.
-
-
-class OutputStream(object):
-
-  """Contains all logic for writing bits, and ToString() to get the result."""
-
-  def __init__(self):
-    self._buffer = array.array('B')
-
-  def AppendRawBytes(self, raw_bytes):
-    """Appends raw_bytes to our internal buffer."""
-    self._buffer.fromstring(raw_bytes)
-
-  def AppendLittleEndian32(self, unsigned_value):
-    """Appends an unsigned 32-bit integer to the internal buffer,
-    in little-endian byte order.
-    """
-    if not 0 <= unsigned_value <= wire_format.UINT32_MAX:
-      raise message.EncodeError(
-          'Unsigned 32-bit out of range: %d' % unsigned_value)
-    self._buffer.fromstring(struct.pack(
-        wire_format.FORMAT_UINT32_LITTLE_ENDIAN, unsigned_value))
-
-  def AppendLittleEndian64(self, unsigned_value):
-    """Appends an unsigned 64-bit integer to the internal buffer,
-    in little-endian byte order.
-    """
-    if not 0 <= unsigned_value <= wire_format.UINT64_MAX:
-      raise message.EncodeError(
-          'Unsigned 64-bit out of range: %d' % unsigned_value)
-    self._buffer.fromstring(struct.pack(
-        wire_format.FORMAT_UINT64_LITTLE_ENDIAN, unsigned_value))
-
-  def AppendVarint32(self, value):
-    """Appends a signed 32-bit integer to the internal buffer,
-    encoded as a varint.  (Note that a negative varint32 will
-    always require 10 bytes of space.)
-    """
-    if not wire_format.INT32_MIN <= value <= wire_format.INT32_MAX:
-      raise message.EncodeError('Value out of range: %d' % value)
-    self.AppendVarint64(value)
-
-  def AppendVarUInt32(self, value):
-    """Appends an unsigned 32-bit integer to the internal buffer,
-    encoded as a varint.
-    """
-    if not 0 <= value <= wire_format.UINT32_MAX:
-      raise message.EncodeError('Value out of range: %d' % value)
-    self.AppendVarUInt64(value)
-
-  def AppendVarint64(self, value):
-    """Appends a signed 64-bit integer to the internal buffer,
-    encoded as a varint.
-    """
-    if not wire_format.INT64_MIN <= value <= wire_format.INT64_MAX:
-      raise message.EncodeError('Value out of range: %d' % value)
-    if value < 0:
-      value += (1 << 64)
-    self.AppendVarUInt64(value)
-
-  def AppendVarUInt64(self, unsigned_value):
-    """Appends an unsigned 64-bit integer to the internal buffer,
-    encoded as a varint.
-    """
-    if not 0 <= unsigned_value <= wire_format.UINT64_MAX:
-      raise message.EncodeError('Value out of range: %d' % unsigned_value)
-    while True:
-      bits = unsigned_value & 0x7f
-      unsigned_value >>= 7
-      if not unsigned_value:
-        self._buffer.append(bits)
-        break
-      self._buffer.append(0x80|bits)
-
-  def ToString(self):
-    """Returns a string containing the bytes in our internal buffer."""
-    return self._buffer.tostring()

froofle/protobuf/internal/type_checkers.py

@@ -1,268 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""Provides type checking routines.
-
-This module defines type checking utilities in the forms of dictionaries:
-
-VALUE_CHECKERS: A dictionary of field types and a value validation object.
-TYPE_TO_BYTE_SIZE_FN: A dictionary with field types and a size computing
-  function.
-TYPE_TO_SERIALIZE_METHOD: A dictionary with field types and serialization
-  function.
-FIELD_TYPE_TO_WIRE_TYPE: A dictionary with field typed and their
-  coresponding wire types.
-TYPE_TO_DESERIALIZE_METHOD: A dictionary with field types and deserialization
-  function.
-"""
-
-__author__ = 'robinson@google.com (Will Robinson)'
-
-from froofle.protobuf.internal import decoder
-from froofle.protobuf.internal import encoder
-from froofle.protobuf.internal import wire_format
-from froofle.protobuf import descriptor
-
-_FieldDescriptor = descriptor.FieldDescriptor
-
-
-def GetTypeChecker(cpp_type, field_type):
-  """Returns a type checker for a message field of the specified types.
-
-  Args:
-    cpp_type: C++ type of the field (see descriptor.py).
-    field_type: Protocol message field type (see descriptor.py).
-
-  Returns:
-    An instance of TypeChecker which can be used to verify the types
-    of values assigned to a field of the specified type.
-  """
-  if (cpp_type == _FieldDescriptor.CPPTYPE_STRING and
-      field_type == _FieldDescriptor.TYPE_STRING):
-    return UnicodeValueChecker()
-  return _VALUE_CHECKERS[cpp_type]
-
-
-# None of the typecheckers below make any attempt to guard against people
-# subclassing builtin types and doing weird things.  We're not trying to
-# protect against malicious clients here, just people accidentally shooting
-# themselves in the foot in obvious ways.
-
-class TypeChecker(object):
-
-  """Type checker used to catch type errors as early as possible
-  when the client is setting scalar fields in protocol messages.
-  """
-
-  def __init__(self, *acceptable_types):
-    self._acceptable_types = acceptable_types
-
-  def CheckValue(self, proposed_value):
-    if not isinstance(proposed_value, self._acceptable_types):
-      message = ('%.1024r has type %s, but expected one of: %s' %
-                 (proposed_value, type(proposed_value), self._acceptable_types))
-      raise TypeError(message)
-
-
-# IntValueChecker and its subclasses perform integer type-checks
-# and bounds-checks.
-class IntValueChecker(object):
-
-  """Checker used for integer fields.  Performs type-check and range check."""
-
-  def CheckValue(self, proposed_value):
-    if not isinstance(proposed_value, (int, long)):
-      message = ('%.1024r has type %s, but expected one of: %s' %
-                 (proposed_value, type(proposed_value), (int, long)))
-      raise TypeError(message)
-    if not self._MIN <= proposed_value <= self._MAX:
-      raise ValueError('Value out of range: %d' % proposed_value)
-
-
-class UnicodeValueChecker(object):
-
-  """Checker used for string fields."""
-
-  def CheckValue(self, proposed_value):
-    if not isinstance(proposed_value, (str, unicode)):
-      message = ('%.1024r has type %s, but expected one of: %s' %
-                 (proposed_value, type(proposed_value), (str, unicode)))
-      raise TypeError(message)
-
-    # If the value is of type 'str' make sure that it is in 7-bit ASCII
-    # encoding.
-    if isinstance(proposed_value, str):
-      try:
-        unicode(proposed_value, 'ascii')
-      except UnicodeDecodeError:
-        raise ValueError('%.1024r isn\'t in 7-bit ASCII encoding.'
-                         % (proposed_value))
-
-
-class Int32ValueChecker(IntValueChecker):
-  # We're sure to use ints instead of longs here since comparison may be more
-  # efficient.
-  _MIN = -2147483648
-  _MAX = 2147483647
-
-
-class Uint32ValueChecker(IntValueChecker):
-  _MIN = 0
-  _MAX = (1 << 32) - 1
-
-
-class Int64ValueChecker(IntValueChecker):
-  _MIN = -(1 << 63)
-  _MAX = (1 << 63) - 1
-
-
-class Uint64ValueChecker(IntValueChecker):
-  _MIN = 0
-  _MAX = (1 << 64) - 1
-
-
-# Type-checkers for all scalar CPPTYPEs.
-_VALUE_CHECKERS = {
-    _FieldDescriptor.CPPTYPE_INT32: Int32ValueChecker(),
-    _FieldDescriptor.CPPTYPE_INT64: Int64ValueChecker(),
-    _FieldDescriptor.CPPTYPE_UINT32: Uint32ValueChecker(),
-    _FieldDescriptor.CPPTYPE_UINT64: Uint64ValueChecker(),
-    _FieldDescriptor.CPPTYPE_DOUBLE: TypeChecker(
-        float, int, long),
-    _FieldDescriptor.CPPTYPE_FLOAT: TypeChecker(
-        float, int, long),
-    _FieldDescriptor.CPPTYPE_BOOL: TypeChecker(bool, int),
-    _FieldDescriptor.CPPTYPE_ENUM: Int32ValueChecker(),
-    _FieldDescriptor.CPPTYPE_STRING: TypeChecker(str),
-    }
-
-
-# Map from field type to a function F, such that F(field_num, value)
-# gives the total byte size for a value of the given type.  This
-# byte size includes tag information and any other additional space
-# associated with serializing "value".
-TYPE_TO_BYTE_SIZE_FN = {
-    _FieldDescriptor.TYPE_DOUBLE: wire_format.DoubleByteSize,
-    _FieldDescriptor.TYPE_FLOAT: wire_format.FloatByteSize,
-    _FieldDescriptor.TYPE_INT64: wire_format.Int64ByteSize,
-    _FieldDescriptor.TYPE_UINT64: wire_format.UInt64ByteSize,
-    _FieldDescriptor.TYPE_INT32: wire_format.Int32ByteSize,
-    _FieldDescriptor.TYPE_FIXED64: wire_format.Fixed64ByteSize,
-    _FieldDescriptor.TYPE_FIXED32: wire_format.Fixed32ByteSize,
-    _FieldDescriptor.TYPE_BOOL: wire_format.BoolByteSize,
-    _FieldDescriptor.TYPE_STRING: wire_format.StringByteSize,
-    _FieldDescriptor.TYPE_GROUP: wire_format.GroupByteSize,
-    _FieldDescriptor.TYPE_MESSAGE: wire_format.MessageByteSize,
-    _FieldDescriptor.TYPE_BYTES: wire_format.BytesByteSize,
-    _FieldDescriptor.TYPE_UINT32: wire_format.UInt32ByteSize,
-    _FieldDescriptor.TYPE_ENUM: wire_format.EnumByteSize,
-    _FieldDescriptor.TYPE_SFIXED32: wire_format.SFixed32ByteSize,
-    _FieldDescriptor.TYPE_SFIXED64: wire_format.SFixed64ByteSize,
-    _FieldDescriptor.TYPE_SINT32: wire_format.SInt32ByteSize,
-    _FieldDescriptor.TYPE_SINT64: wire_format.SInt64ByteSize
-    }
-
-
-# Maps from field type to an unbound Encoder method F, such that
-# F(encoder, field_number, value) will append the serialization
-# of a value of this type to the encoder.
-_Encoder = encoder.Encoder
-TYPE_TO_SERIALIZE_METHOD = {
-    _FieldDescriptor.TYPE_DOUBLE: _Encoder.AppendDouble,
-    _FieldDescriptor.TYPE_FLOAT: _Encoder.AppendFloat,
-    _FieldDescriptor.TYPE_INT64: _Encoder.AppendInt64,
-    _FieldDescriptor.TYPE_UINT64: _Encoder.AppendUInt64,
-    _FieldDescriptor.TYPE_INT32: _Encoder.AppendInt32,
-    _FieldDescriptor.TYPE_FIXED64: _Encoder.AppendFixed64,
-    _FieldDescriptor.TYPE_FIXED32: _Encoder.AppendFixed32,
-    _FieldDescriptor.TYPE_BOOL: _Encoder.AppendBool,
-    _FieldDescriptor.TYPE_STRING: _Encoder.AppendString,
-    _FieldDescriptor.TYPE_GROUP: _Encoder.AppendGroup,
-    _FieldDescriptor.TYPE_MESSAGE: _Encoder.AppendMessage,
-    _FieldDescriptor.TYPE_BYTES: _Encoder.AppendBytes,
-    _FieldDescriptor.TYPE_UINT32: _Encoder.AppendUInt32,
-    _FieldDescriptor.TYPE_ENUM: _Encoder.AppendEnum,
-    _FieldDescriptor.TYPE_SFIXED32: _Encoder.AppendSFixed32,
-    _FieldDescriptor.TYPE_SFIXED64: _Encoder.AppendSFixed64,
-    _FieldDescriptor.TYPE_SINT32: _Encoder.AppendSInt32,
-    _FieldDescriptor.TYPE_SINT64: _Encoder.AppendSInt64,
-    }
-
-
-# Maps from field type to expected wiretype.
-FIELD_TYPE_TO_WIRE_TYPE = {
-    _FieldDescriptor.TYPE_DOUBLE: wire_format.WIRETYPE_FIXED64,
-    _FieldDescriptor.TYPE_FLOAT: wire_format.WIRETYPE_FIXED32,
-    _FieldDescriptor.TYPE_INT64: wire_format.WIRETYPE_VARINT,
-    _FieldDescriptor.TYPE_UINT64: wire_format.WIRETYPE_VARINT,
-    _FieldDescriptor.TYPE_INT32: wire_format.WIRETYPE_VARINT,
-    _FieldDescriptor.TYPE_FIXED64: wire_format.WIRETYPE_FIXED64,
-    _FieldDescriptor.TYPE_FIXED32: wire_format.WIRETYPE_FIXED32,
-    _FieldDescriptor.TYPE_BOOL: wire_format.WIRETYPE_VARINT,
-    _FieldDescriptor.TYPE_STRING:
-      wire_format.WIRETYPE_LENGTH_DELIMITED,
-    _FieldDescriptor.TYPE_GROUP: wire_format.WIRETYPE_START_GROUP,
-    _FieldDescriptor.TYPE_MESSAGE:
-      wire_format.WIRETYPE_LENGTH_DELIMITED,
-    _FieldDescriptor.TYPE_BYTES:
-      wire_format.WIRETYPE_LENGTH_DELIMITED,
-    _FieldDescriptor.TYPE_UINT32: wire_format.WIRETYPE_VARINT,
-    _FieldDescriptor.TYPE_ENUM: wire_format.WIRETYPE_VARINT,
-    _FieldDescriptor.TYPE_SFIXED32: wire_format.WIRETYPE_FIXED32,
-    _FieldDescriptor.TYPE_SFIXED64: wire_format.WIRETYPE_FIXED64,
-    _FieldDescriptor.TYPE_SINT32: wire_format.WIRETYPE_VARINT,
-    _FieldDescriptor.TYPE_SINT64: wire_format.WIRETYPE_VARINT,
-    }
-
-
-# Maps from field type to an unbound Decoder method F,
-# such that F(decoder) will read a field of the requested type.
-#
-# Note that Message and Group are intentionally missing here.
-# They're handled by _RecursivelyMerge().
-_Decoder = decoder.Decoder
-TYPE_TO_DESERIALIZE_METHOD = {
-    _FieldDescriptor.TYPE_DOUBLE: _Decoder.ReadDouble,
-    _FieldDescriptor.TYPE_FLOAT: _Decoder.ReadFloat,
-    _FieldDescriptor.TYPE_INT64: _Decoder.ReadInt64,
-    _FieldDescriptor.TYPE_UINT64: _Decoder.ReadUInt64,
-    _FieldDescriptor.TYPE_INT32: _Decoder.ReadInt32,
-    _FieldDescriptor.TYPE_FIXED64: _Decoder.ReadFixed64,
-    _FieldDescriptor.TYPE_FIXED32: _Decoder.ReadFixed32,
-    _FieldDescriptor.TYPE_BOOL: _Decoder.ReadBool,
-    _FieldDescriptor.TYPE_STRING: _Decoder.ReadString,
-    _FieldDescriptor.TYPE_BYTES: _Decoder.ReadBytes,
-    _FieldDescriptor.TYPE_UINT32: _Decoder.ReadUInt32,
-    _FieldDescriptor.TYPE_ENUM: _Decoder.ReadEnum,
-    _FieldDescriptor.TYPE_SFIXED32: _Decoder.ReadSFixed32,
-    _FieldDescriptor.TYPE_SFIXED64: _Decoder.ReadSFixed64,
-    _FieldDescriptor.TYPE_SINT32: _Decoder.ReadSInt32,
-    _FieldDescriptor.TYPE_SINT64: _Decoder.ReadSInt64,
-    }

froofle/protobuf/internal/wire_format.py

@@ -1,236 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""Constants and static functions to support protocol buffer wire format."""
-
-__author__ = 'robinson@google.com (Will Robinson)'
-
-import struct
-from froofle.protobuf import message
-
-
-TAG_TYPE_BITS = 3  # Number of bits used to hold type info in a proto tag.
-_TAG_TYPE_MASK = (1 << TAG_TYPE_BITS) - 1  # 0x7
-
-# These numbers identify the wire type of a protocol buffer value.
-# We use the least-significant TAG_TYPE_BITS bits of the varint-encoded
-# tag-and-type to store one of these WIRETYPE_* constants.
-# These values must match WireType enum in //net/proto2/public/wire_format.h.
-WIRETYPE_VARINT = 0
-WIRETYPE_FIXED64 = 1
-WIRETYPE_LENGTH_DELIMITED = 2
-WIRETYPE_START_GROUP = 3
-WIRETYPE_END_GROUP = 4
-WIRETYPE_FIXED32 = 5
-_WIRETYPE_MAX = 5
-
-
-# Bounds for various integer types.
-INT32_MAX = int((1 << 31) - 1)
-INT32_MIN = int(-(1 << 31))
-UINT32_MAX = (1 << 32) - 1
-
-INT64_MAX = (1 << 63) - 1
-INT64_MIN = -(1 << 63)
-UINT64_MAX = (1 << 64) - 1
-
-# "struct" format strings that will encode/decode the specified formats.
-FORMAT_UINT32_LITTLE_ENDIAN = '<I'
-FORMAT_UINT64_LITTLE_ENDIAN = '<Q'
-
-
-# We'll have to provide alternate implementations of AppendLittleEndian*() on
-# any architectures where these checks fail.
-if struct.calcsize(FORMAT_UINT32_LITTLE_ENDIAN) != 4:
-  raise AssertionError('Format "I" is not a 32-bit number.')
-if struct.calcsize(FORMAT_UINT64_LITTLE_ENDIAN) != 8:
-  raise AssertionError('Format "Q" is not a 64-bit number.')
-
-
-def PackTag(field_number, wire_type):
-  """Returns an unsigned 32-bit integer that encodes the field number and
-  wire type information in standard protocol message wire format.
-
-  Args:
-    field_number: Expected to be an integer in the range [1, 1 << 29)
-    wire_type: One of the WIRETYPE_* constants.
-  """
-  if not 0 <= wire_type <= _WIRETYPE_MAX:
-    raise message.EncodeError('Unknown wire type: %d' % wire_type)
-  return (field_number << TAG_TYPE_BITS) | wire_type
-
-
-def UnpackTag(tag):
-  """The inverse of PackTag().  Given an unsigned 32-bit number,
-  returns a (field_number, wire_type) tuple.
-  """
-  return (tag >> TAG_TYPE_BITS), (tag & _TAG_TYPE_MASK)
-
-
-def ZigZagEncode(value):
-  """ZigZag Transform:  Encodes signed integers so that they can be
-  effectively used with varint encoding.  See wire_format.h for
-  more details.
-  """
-  if value >= 0:
-    return value << 1
-  return (value << 1) ^ (~0)
-
-
-def ZigZagDecode(value):
-  """Inverse of ZigZagEncode()."""
-  if not value & 0x1:
-    return value >> 1
-  return (value >> 1) ^ (~0)
-
-
-
-# The *ByteSize() functions below return the number of bytes required to
-# serialize "field number + type" information and then serialize the value.
-
-
-def Int32ByteSize(field_number, int32):
-  return Int64ByteSize(field_number, int32)
-
-
-def Int64ByteSize(field_number, int64):
-  # Have to convert to uint before calling UInt64ByteSize().
-  return UInt64ByteSize(field_number, 0xffffffffffffffff & int64)
-
-
-def UInt32ByteSize(field_number, uint32):
-  return UInt64ByteSize(field_number, uint32)
-
-
-def UInt64ByteSize(field_number, uint64):
-  return _TagByteSize(field_number) + _VarUInt64ByteSizeNoTag(uint64)
-
-
-def SInt32ByteSize(field_number, int32):
-  return UInt32ByteSize(field_number, ZigZagEncode(int32))
-
-
-def SInt64ByteSize(field_number, int64):
-  return UInt64ByteSize(field_number, ZigZagEncode(int64))
-
-
-def Fixed32ByteSize(field_number, fixed32):
-  return _TagByteSize(field_number) + 4
-
-
-def Fixed64ByteSize(field_number, fixed64):
-  return _TagByteSize(field_number) + 8
-
-
-def SFixed32ByteSize(field_number, sfixed32):
-  return _TagByteSize(field_number) + 4
-
-
-def SFixed64ByteSize(field_number, sfixed64):
-  return _TagByteSize(field_number) + 8
-
-
-def FloatByteSize(field_number, flt):
-  return _TagByteSize(field_number) + 4
-
-
-def DoubleByteSize(field_number, double):
-  return _TagByteSize(field_number) + 8
-
-
-def BoolByteSize(field_number, b):
-  return _TagByteSize(field_number) + 1
-
-
-def EnumByteSize(field_number, enum):
-  return UInt32ByteSize(field_number, enum)
-
-
-def StringByteSize(field_number, string):
-  return BytesByteSize(field_number, string.encode('utf-8'))
-
-
-def BytesByteSize(field_number, b):
-  return (_TagByteSize(field_number)
-          + _VarUInt64ByteSizeNoTag(len(b))
-          + len(b))
-
-
-def GroupByteSize(field_number, message):
-  return (2 * _TagByteSize(field_number)  # START and END group.
-          + message.ByteSize())
-
-
-def MessageByteSize(field_number, message):
-  return (_TagByteSize(field_number)
-          + _VarUInt64ByteSizeNoTag(message.ByteSize())
-          + message.ByteSize())
-
-
-def MessageSetItemByteSize(field_number, msg):
-  # First compute the sizes of the tags.
-  # There are 2 tags for the beginning and ending of the repeated group, that
-  # is field number 1, one with field number 2 (type_id) and one with field
-  # number 3 (message).
-  total_size = (2 * _TagByteSize(1) + _TagByteSize(2) + _TagByteSize(3))
-
-  # Add the number of bytes for type_id.
-  total_size += _VarUInt64ByteSizeNoTag(field_number)
-
-  message_size = msg.ByteSize()
-
-  # The number of bytes for encoding the length of the message.
-  total_size += _VarUInt64ByteSizeNoTag(message_size)
-
-  # The size of the message.
-  total_size += message_size
-  return total_size
-
-
-# Private helper functions for the *ByteSize() functions above.
-
-
-def _TagByteSize(field_number):
-  """Returns the bytes required to serialize a tag with this field number."""
-  # Just pass in type 0, since the type won't affect the tag+type size.
-  return _VarUInt64ByteSizeNoTag(PackTag(field_number, 0))
-
-
-def _VarUInt64ByteSizeNoTag(uint64):
-  """Returns the bytes required to serialize a single varint.
-  uint64 must be unsigned.
-  """
-  if uint64 > UINT64_MAX:
-    raise message.EncodeError('Value out of range: %d' % uint64)
-  bytes = 1
-  while uint64 > 0x7f:
-    bytes += 1
-    uint64 >>= 7
-  return bytes

froofle/protobuf/message.py

@@ -1,246 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-# TODO(robinson): We should just make these methods all "pure-virtual" and move
-# all implementation out, into reflection.py for now.
-
-
-"""Contains an abstract base class for protocol messages."""
-
-__author__ = 'robinson@google.com (Will Robinson)'
-
-from froofle.protobuf import text_format
-
-class Error(Exception): pass
-class DecodeError(Error): pass
-class EncodeError(Error): pass
-
-
-class Message(object):
-
-  """Abstract base class for protocol messages.
-
-  Protocol message classes are almost always generated by the protocol
-  compiler.  These generated types subclass Message and implement the methods
-  shown below.
-
-  TODO(robinson): Link to an HTML document here.
-
-  TODO(robinson): Document that instances of this class will also
-  have an Extensions attribute with __getitem__ and __setitem__.
-  Again, not sure how to best convey this.
-
-  TODO(robinson): Document that the class must also have a static
-    RegisterExtension(extension_field) method.
-    Not sure how to best express at this point.
-  """
-
-  # TODO(robinson): Document these fields and methods.
-
-  __slots__ = []
-
-  DESCRIPTOR = None
-
-  def __eq__(self, other_msg):
-    raise NotImplementedError
-
-  def __ne__(self, other_msg):
-    # Can't just say self != other_msg, since that would infinitely recurse. :)
-    return not self == other_msg
-
-  def __str__(self):
-    return text_format.MessageToString(self)
-
-  def MergeFrom(self, other_msg):
-    """Merges the contents of the specified message into current message.
-
-    This method merges the contents of the specified message into the current
-    message. Singular fields that are set in the specified message overwrite
-    the corresponding fields in the current message. Repeated fields are
-    appended. Singular sub-messages and groups are recursively merged.
-
-    Args:
-      other_msg: Message to merge into the current message.
-    """
-    raise NotImplementedError
-
-  def CopyFrom(self, other_msg):
-    """Copies the content of the specified message into the current message.
-
-    The method clears the current message and then merges the specified
-    message using MergeFrom.
-
-    Args:
-      other_msg: Message to copy into the current one.
-    """
-    if self == other_msg:
-      return
-    self.Clear()
-    self.MergeFrom(other_msg)
-
-  def Clear(self):
-    """Clears all data that was set in the message."""
-    raise NotImplementedError
-
-  def IsInitialized(self):
-    """Checks if the message is initialized.
-
-    Returns:
-      The method returns True if the message is initialized (i.e. all of its
-      required fields are set).
-    """
-    raise NotImplementedError
-
-  # TODO(robinson): MergeFromString() should probably return None and be
-  # implemented in terms of a helper that returns the # of bytes read.  Our
-  # deserialization routines would use the helper when recursively
-  # deserializing, but the end user would almost always just want the no-return
-  # MergeFromString().
-
-  def MergeFromString(self, serialized):
-    """Merges serialized protocol buffer data into this message.
-
-    When we find a field in |serialized| that is already present
-    in this message:
-      - If it's a "repeated" field, we append to the end of our list.
-      - Else, if it's a scalar, we overwrite our field.
-      - Else, (it's a nonrepeated composite), we recursively merge
-        into the existing composite.
-
-    TODO(robinson): Document handling of unknown fields.
-
-    Args:
-      serialized: Any object that allows us to call buffer(serialized)
-        to access a string of bytes using the buffer interface.
-
-    TODO(robinson): When we switch to a helper, this will return None.
-
-    Returns:
-      The number of bytes read from |serialized|.
-      For non-group messages, this will always be len(serialized),
-      but for messages which are actually groups, this will
-      generally be less than len(serialized), since we must
-      stop when we reach an END_GROUP tag.  Note that if
-      we *do* stop because of an END_GROUP tag, the number
-      of bytes returned does not include the bytes
-      for the END_GROUP tag information.
-    """
-    raise NotImplementedError
-
-  def ParseFromString(self, serialized):
-    """Like MergeFromString(), except we clear the object first."""
-    self.Clear()
-    self.MergeFromString(serialized)
-
-  def SerializeToString(self):
-    """Serializes the protocol message to a binary string.
-
-    Returns:
-      A binary string representation of the message if all of the required
-      fields in the message are set (i.e. the message is initialized).
-
-    Raises:
-      message.EncodeError if the message isn't initialized.
-    """
-    raise NotImplementedError
-
-  def SerializePartialToString(self):
-    """Serializes the protocol message to a binary string.
-
-    This method is similar to SerializeToString but doesn't check if the
-    message is initialized.
-
-    Returns:
-      A string representation of the partial message.
-    """
-    raise NotImplementedError
-
-  # TODO(robinson): Decide whether we like these better
-  # than auto-generated has_foo() and clear_foo() methods
-  # on the instances themselves.  This way is less consistent
-  # with C++, but it makes reflection-type access easier and
-  # reduces the number of magically autogenerated things.
-  #
-  # TODO(robinson): Be sure to document (and test) exactly
-  # which field names are accepted here.  Are we case-sensitive?
-  # What do we do with fields that share names with Python keywords
-  # like 'lambda' and 'yield'?
-  #
-  # nnorwitz says:
-  # """
-  # Typically (in python), an underscore is appended to names that are
-  # keywords. So they would become lambda_ or yield_.
-  # """
-  def ListFields(self, field_name):
-    """Returns a list of (FieldDescriptor, value) tuples for all
-    fields in the message which are not empty.  A singular field is non-empty
-    if HasField() would return true, and a repeated field is non-empty if
-    it contains at least one element.  The fields are ordered by field
-    number"""
-    raise NotImplementedError
-
-  def HasField(self, field_name):
-    raise NotImplementedError
-
-  def ClearField(self, field_name):
-    raise NotImplementedError
-
-  def HasExtension(self, extension_handle):
-    raise NotImplementedError
-
-  def ClearExtension(self, extension_handle):
-    raise NotImplementedError
-
-  def ByteSize(self):
-    """Returns the serialized size of this message.
-    Recursively calls ByteSize() on all contained messages.
-    """
-    raise NotImplementedError
-
-  def _SetListener(self, message_listener):
-    """Internal method used by the protocol message implementation.
-    Clients should not call this directly.
-
-    Sets a listener that this message will call on certain state transitions.
-
-    The purpose of this method is to register back-edges from children to
-    parents at runtime, for the purpose of setting "has" bits and
-    byte-size-dirty bits in the parent and ancestor objects whenever a child or
-    descendant object is modified.
-
-    If the client wants to disconnect this Message from the object tree, she
-    explicitly sets callback to None.
-
-    If message_listener is None, unregisters any existing listener.  Otherwise,
-    message_listener must implement the MessageListener interface in
-    internal/message_listener.py, and we discard any listener registered
-    via a previous _SetListener() call.
-    """
-    raise NotImplementedError

froofle/protobuf/service.py

@@ -1,208 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""Declares the RPC service interfaces.
-
-This module declares the abstract interfaces underlying proto2 RPC
-services.  These are intented to be independent of any particular RPC
-implementation, so that proto2 services can be used on top of a variety
-of implementations.
-"""
-
-__author__ = 'petar@google.com (Petar Petrov)'
-
-
-class Service(object):
-
-  """Abstract base interface for protocol-buffer-based RPC services.
-
-  Services themselves are abstract classes (implemented either by servers or as
-  stubs), but they subclass this base interface. The methods of this
-  interface can be used to call the methods of the service without knowing
-  its exact type at compile time (analogous to the Message interface).
-  """
-
-  def GetDescriptor(self):
-    """Retrieves this service's descriptor."""
-    raise NotImplementedError
-
-  def CallMethod(self, method_descriptor, rpc_controller,
-                 request, done):
-    """Calls a method of the service specified by method_descriptor.
-
-    Preconditions:
-    * method_descriptor.service == GetDescriptor
-    * request is of the exact same classes as returned by
-      GetRequestClass(method).
-    * After the call has started, the request must not be modified.
-    * "rpc_controller" is of the correct type for the RPC implementation being
-      used by this Service.  For stubs, the "correct type" depends on the
-      RpcChannel which the stub is using.
-
-    Postconditions:
-    * "done" will be called when the method is complete.  This may be
-      before CallMethod() returns or it may be at some point in the future.
-    """
-    raise NotImplementedError
-
-  def GetRequestClass(self, method_descriptor):
-    """Returns the class of the request message for the specified method.
-
-    CallMethod() requires that the request is of a particular subclass of
-    Message. GetRequestClass() gets the default instance of this required
-    type.
-
-    Example:
-      method = service.GetDescriptor().FindMethodByName("Foo")
-      request = stub.GetRequestClass(method)()
-      request.ParseFromString(input)
-      service.CallMethod(method, request, callback)
-    """
-    raise NotImplementedError
-
-  def GetResponseClass(self, method_descriptor):
-    """Returns the class of the response message for the specified method.
-
-    This method isn't really needed, as the RpcChannel's CallMethod constructs
-    the response protocol message. It's provided anyway in case it is useful
-    for the caller to know the response type in advance.
-    """
-    raise NotImplementedError
-
-
-class RpcController(object):
-
-  """An RpcController mediates a single method call.
-
-  The primary purpose of the controller is to provide a way to manipulate
-  settings specific to the RPC implementation and to find out about RPC-level
-  errors. The methods provided by the RpcController interface are intended
-  to be a "least common denominator" set of features which we expect all
-  implementations to support.  Specific implementations may provide more
-  advanced features (e.g. deadline propagation).
-  """
-
-  # Client-side methods below
-
-  def Reset(self):
-    """Resets the RpcController to its initial state.
-
-    After the RpcController has been reset, it may be reused in
-    a new call. Must not be called while an RPC is in progress.
-    """
-    raise NotImplementedError
-
-  def Failed(self):
-    """Returns true if the call failed.
-
-    After a call has finished, returns true if the call failed.  The possible
-    reasons for failure depend on the RPC implementation.  Failed() must not
-    be called before a call has finished.  If Failed() returns true, the
-    contents of the response message are undefined.
-    """
-    raise NotImplementedError
-
-  def ErrorText(self):
-    """If Failed is true, returns a human-readable description of the error."""
-    raise NotImplementedError
-
-  def StartCancel(self):
-    """Initiate cancellation.
-
-    Advises the RPC system that the caller desires that the RPC call be
-    canceled.  The RPC system may cancel it immediately, may wait awhile and
-    then cancel it, or may not even cancel the call at all.  If the call is
-    canceled, the "done" callback will still be called and the RpcController
-    will indicate that the call failed at that time.
-    """
-    raise NotImplementedError
-
-  # Server-side methods below
-
-  def SetFailed(self, reason):
-    """Sets a failure reason.
-
-    Causes Failed() to return true on the client side.  "reason" will be
-    incorporated into the message returned by ErrorText().  If you find
-    you need to return machine-readable information about failures, you
-    should incorporate it into your response protocol buffer and should
-    NOT call SetFailed().
-    """
-    raise NotImplementedError
-
-  def IsCanceled(self):
-    """Checks if the client cancelled the RPC.
-
-    If true, indicates that the client canceled the RPC, so the server may
-    as well give up on replying to it.  The server should still call the
-    final "done" callback.
-    """
-    raise NotImplementedError
-
-  def NotifyOnCancel(self, callback):
-    """Sets a callback to invoke on cancel.
-
-    Asks that the given callback be called when the RPC is canceled.  The
-    callback will always be called exactly once.  If the RPC completes without
-    being canceled, the callback will be called after completion.  If the RPC
-    has already been canceled when NotifyOnCancel() is called, the callback
-    will be called immediately.
-
-    NotifyOnCancel() must be called no more than once per request.
-    """
-    raise NotImplementedError
-
-
-class RpcChannel(object):
-
-  """Abstract interface for an RPC channel.
-
-  An RpcChannel represents a communication line to a service which can be used
-  to call that service's methods.  The service may be running on another
-  machine. Normally, you should not use an RpcChannel directly, but instead
-  construct a stub {@link Service} wrapping it.  Example:
-
-  Example:
-    RpcChannel channel = rpcImpl.Channel("remotehost.example.com:1234")
-    RpcController controller = rpcImpl.Controller()
-    MyService service = MyService_Stub(channel)
-    service.MyMethod(controller, request, callback)
-  """
-
-  def CallMethod(self, method_descriptor, rpc_controller,
-                 request, response_class, done):
-    """Calls the method identified by the descriptor.
-
-    Call the given method of the remote service.  The signature of this
-    procedure looks the same as Service.CallMethod(), but the requirements
-    are less strict in one important way:  the request object doesn't have to
-    be of any specific class as long as its descriptor is method.input_type.
-    """
-    raise NotImplementedError

froofle/protobuf/service_reflection.py

@@ -1,289 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""Contains metaclasses used to create protocol service and service stub
-classes from ServiceDescriptor objects at runtime.
-
-The GeneratedServiceType and GeneratedServiceStubType metaclasses are used to
-inject all useful functionality into the classes output by the protocol
-compiler at compile-time.
-"""
-
-__author__ = 'petar@google.com (Petar Petrov)'
-
-
-class GeneratedServiceType(type):
-
-  """Metaclass for service classes created at runtime from ServiceDescriptors.
-
-  Implementations for all methods described in the Service class are added here
-  by this class. We also create properties to allow getting/setting all fields
-  in the protocol message.
-
-  The protocol compiler currently uses this metaclass to create protocol service
-  classes at runtime. Clients can also manually create their own classes at
-  runtime, as in this example:
-
-  mydescriptor = ServiceDescriptor(.....)
-  class MyProtoService(service.Service):
-    __metaclass__ = GeneratedServiceType
-    DESCRIPTOR = mydescriptor
-  myservice_instance = MyProtoService()
-  ...
-  """
-
-  _DESCRIPTOR_KEY = 'DESCRIPTOR'
-
-  def __init__(cls, name, bases, dictionary):
-    """Creates a message service class.
-
-    Args:
-      name: Name of the class (ignored, but required by the metaclass
-        protocol).
-      bases: Base classes of the class being constructed.
-      dictionary: The class dictionary of the class being constructed.
-        dictionary[_DESCRIPTOR_KEY] must contain a ServiceDescriptor object
-        describing this protocol service type.
-    """
-    # Don't do anything if this class doesn't have a descriptor. This happens
-    # when a service class is subclassed.
-    if GeneratedServiceType._DESCRIPTOR_KEY not in dictionary:
-      return
-    descriptor = dictionary[GeneratedServiceType._DESCRIPTOR_KEY]
-    service_builder = _ServiceBuilder(descriptor)
-    service_builder.BuildService(cls)
-
-
-class GeneratedServiceStubType(GeneratedServiceType):
-
-  """Metaclass for service stubs created at runtime from ServiceDescriptors.
-
-  This class has similar responsibilities as GeneratedServiceType, except that
-  it creates the service stub classes.
-  """
-
-  _DESCRIPTOR_KEY = 'DESCRIPTOR'
-
-  def __init__(cls, name, bases, dictionary):
-    """Creates a message service stub class.
-
-    Args:
-      name: Name of the class (ignored, here).
-      bases: Base classes of the class being constructed.
-      dictionary: The class dictionary of the class being constructed.
-        dictionary[_DESCRIPTOR_KEY] must contain a ServiceDescriptor object
-        describing this protocol service type.
-    """
-    super(GeneratedServiceStubType, cls).__init__(name, bases, dictionary)
-    # Don't do anything if this class doesn't have a descriptor. This happens
-    # when a service stub is subclassed.
-    if GeneratedServiceStubType._DESCRIPTOR_KEY not in dictionary:
-      return
-    descriptor = dictionary[GeneratedServiceStubType._DESCRIPTOR_KEY]
-    service_stub_builder = _ServiceStubBuilder(descriptor)
-    service_stub_builder.BuildServiceStub(cls)
-
-
-class _ServiceBuilder(object):
-
-  """This class constructs a protocol service class using a service descriptor.
-
-  Given a service descriptor, this class constructs a class that represents
-  the specified service descriptor. One service builder instance constructs
-  exactly one service class. That means all instances of that class share the
-  same builder.
-  """
-
-  def __init__(self, service_descriptor):
-    """Initializes an instance of the service class builder.
-
-    Args:
-      service_descriptor: ServiceDescriptor to use when constructing the
-        service class.
-    """
-    self.descriptor = service_descriptor
-
-  def BuildService(self, cls):
-    """Constructs the service class.
-
-    Args:
-      cls: The class that will be constructed.
-    """
-
-    # CallMethod needs to operate with an instance of the Service class. This
-    # internal wrapper function exists only to be able to pass the service
-    # instance to the method that does the real CallMethod work.
-    def _WrapCallMethod(srvc, method_descriptor,
-                        rpc_controller, request, callback):
-      self._CallMethod(srvc, method_descriptor,
-                       rpc_controller, request, callback)
-    self.cls = cls
-    cls.CallMethod = _WrapCallMethod
-    cls.GetDescriptor = self._GetDescriptor
-    cls.GetRequestClass = self._GetRequestClass
-    cls.GetResponseClass = self._GetResponseClass
-    for method in self.descriptor.methods:
-      setattr(cls, method.name, self._GenerateNonImplementedMethod(method))
-
-  def _GetDescriptor(self):
-    """Retrieves the service descriptor.
-
-    Returns:
-      The descriptor of the service (of type ServiceDescriptor).
-    """
-    return self.descriptor
-
-  def _CallMethod(self, srvc, method_descriptor,
-                  rpc_controller, request, callback):
-    """Calls the method described by a given method descriptor.
-
-    Args:
-      srvc: Instance of the service for which this method is called.
-      method_descriptor: Descriptor that represent the method to call.
-      rpc_controller: RPC controller to use for this method's execution.
-      request: Request protocol message.
-      callback: A callback to invoke after the method has completed.
-    """
-    if method_descriptor.containing_service != self.descriptor:
-      raise RuntimeError(
-          'CallMethod() given method descriptor for wrong service type.')
-    method = getattr(srvc, method_descriptor.name)
-    method(rpc_controller, request, callback)
-
-  def _GetRequestClass(self, method_descriptor):
-    """Returns the class of the request protocol message.
-
-    Args:
-      method_descriptor: Descriptor of the method for which to return the
-        request protocol message class.
-
-    Returns:
-      A class that represents the input protocol message of the specified
-      method.
-    """
-    if method_descriptor.containing_service != self.descriptor:
-      raise RuntimeError(
-          'GetRequestClass() given method descriptor for wrong service type.')
-    return method_descriptor.input_type._concrete_class
-
-  def _GetResponseClass(self, method_descriptor):
-    """Returns the class of the response protocol message.
-
-    Args:
-      method_descriptor: Descriptor of the method for which to return the
-        response protocol message class.
-
-    Returns:
-      A class that represents the output protocol message of the specified
-      method.
-    """
-    if method_descriptor.containing_service != self.descriptor:
-      raise RuntimeError(
-          'GetResponseClass() given method descriptor for wrong service type.')
-    return method_descriptor.output_type._concrete_class
-
-  def _GenerateNonImplementedMethod(self, method):
-    """Generates and returns a method that can be set for a service methods.
-
-    Args:
-      method: Descriptor of the service method for which a method is to be
-        generated.
-
-    Returns:
-      A method that can be added to the service class.
-    """
-    return lambda inst, rpc_controller, request, callback: (
-        self._NonImplementedMethod(method.name, rpc_controller, callback))
-
-  def _NonImplementedMethod(self, method_name, rpc_controller, callback):
-    """The body of all methods in the generated service class.
-
-    Args:
-      method_name: Name of the method being executed.
-      rpc_controller: RPC controller used to execute this method.
-      callback: A callback which will be invoked when the method finishes.
-    """
-    rpc_controller.SetFailed('Method %s not implemented.' % method_name)
-    callback(None)
-
-
-class _ServiceStubBuilder(object):
-
-  """Constructs a protocol service stub class using a service descriptor.
-
-  Given a service descriptor, this class constructs a suitable stub class.
-  A stub is just a type-safe wrapper around an RpcChannel which emulates a
-  local implementation of the service.
-
-  One service stub builder instance constructs exactly one class. It means all
-  instances of that class share the same service stub builder.
-  """
-
-  def __init__(self, service_descriptor):
-    """Initializes an instance of the service stub class builder.
-
-    Args:
-      service_descriptor: ServiceDescriptor to use when constructing the
-        stub class.
-    """
-    self.descriptor = service_descriptor
-
-  def BuildServiceStub(self, cls):
-    """Constructs the stub class.
-
-    Args:
-      cls: The class that will be constructed.
-    """
-
-    def _ServiceStubInit(stub, rpc_channel):
-      stub.rpc_channel = rpc_channel
-    self.cls = cls
-    cls.__init__ = _ServiceStubInit
-    for method in self.descriptor.methods:
-      setattr(cls, method.name, self._GenerateStubMethod(method))
-
-  def _GenerateStubMethod(self, method):
-    return lambda inst, rpc_controller, request, callback: self._StubMethod(
-        inst, method, rpc_controller, request, callback)
-
-  def _StubMethod(self, stub, method_descriptor,
-                  rpc_controller, request, callback):
-    """The body of all service methods in the generated stub class.
-
-    Args:
-      stub: Stub instance.
-      method_descriptor: Descriptor of the invoked method.
-      rpc_controller: Rpc controller to execute the method.
-      request: Request protocol message.
-      callback: A callback to execute when the method finishes.
-    """
-    stub.rpc_channel.CallMethod(
-        method_descriptor, rpc_controller, request,
-        method_descriptor.output_type._concrete_class, callback)

froofle/protobuf/text_format.py

@@ -1,125 +0,0 @@
-# Protocol Buffers - Google's data interchange format
-# Copyright 2008 Google Inc.  All rights reserved.
-# http://code.google.com/p/protobuf/
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-#     * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-#     * Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following disclaimer
-# in the documentation and/or other materials provided with the
-# distribution.
-#     * Neither the name of Google Inc. nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""Contains routines for printing protocol messages in text format."""
-
-__author__ = 'kenton@google.com (Kenton Varda)'
-
-import cStringIO
-
-from froofle.protobuf import descriptor
-
-__all__ = [ 'MessageToString', 'PrintMessage', 'PrintField', 'PrintFieldValue' ]
-
-def MessageToString(message):
-  out = cStringIO.StringIO()
-  PrintMessage(message, out)
-  result = out.getvalue()
-  out.close()
-  return result
-
-def PrintMessage(message, out, indent = 0):
-  for field, value in message.ListFields():
-    if field.label == descriptor.FieldDescriptor.LABEL_REPEATED:
-      for element in value:
-        PrintField(field, element, out, indent)
-    else:
-      PrintField(field, value, out, indent)
-
-def PrintField(field, value, out, indent = 0):
-  """Print a single field name/value pair.  For repeated fields, the value
-  should be a single element."""
-
-  out.write(' ' * indent);
-  if field.is_extension:
-    out.write('[')
-    if (field.containing_type.GetOptions().message_set_wire_format and
-        field.type == descriptor.FieldDescriptor.TYPE_MESSAGE and
-        field.message_type == field.extension_scope and
-        field.label == descriptor.FieldDescriptor.LABEL_OPTIONAL):
-      out.write(field.message_type.full_name)
-    else:
-      out.write(field.full_name)
-    out.write(']')
-  elif field.type == descriptor.FieldDescriptor.TYPE_GROUP:
-    # For groups, use the capitalized name.
-    out.write(field.message_type.name)
-  else:
-    out.write(field.name)
-
-  if field.cpp_type != descriptor.FieldDescriptor.CPPTYPE_MESSAGE:
-    # The colon is optional in this case, but our cross-language golden files
-    # don't include it.
-    out.write(': ')
-
-  PrintFieldValue(field, value, out, indent)
-  out.write('\n')
-
-def PrintFieldValue(field, value, out, indent = 0):
-  """Print a single field value (not including name).  For repeated fields,
-  the value should be a single element."""
-
-  if field.cpp_type == descriptor.FieldDescriptor.CPPTYPE_MESSAGE:
-    out.write(' {\n')
-    PrintMessage(value, out, indent + 2)
-    out.write(' ' * indent + '}')
-  elif field.cpp_type == descriptor.FieldDescriptor.CPPTYPE_ENUM:
-    out.write(field.enum_type.values_by_number[value].name)
-  elif field.cpp_type == descriptor.FieldDescriptor.CPPTYPE_STRING:
-    out.write('\"')
-    out.write(_CEscape(value))
-    out.write('\"')
-  elif field.cpp_type == descriptor.FieldDescriptor.CPPTYPE_BOOL:
-    if value:
-      out.write("true")
-    else:
-      out.write("false")
-  else:
-    out.write(str(value))
-
-# text.encode('string_escape') does not seem to satisfy our needs as it
-# encodes unprintable characters using two-digit hex escapes whereas our
-# C++ unescaping function allows hex escapes to be any length.  So,
-# "\0011".encode('string_escape') ends up being "\\x011", which will be
-# decoded in C++ as a single-character string with char code 0x11.
-def _CEscape(text):
-  def escape(c):
-    o = ord(c)
-    if o == 10: return r"\n"   # optional escape
-    if o == 13: return r"\r"   # optional escape
-    if o ==  9: return r"\t"   # optional escape
-    if o == 39: return r"\'"   # optional escape
-
-    if o == 34: return r'\"'   # necessary escape
-    if o == 92: return r"\\"   # necessary escape
-
-    if o >= 127 or o < 32: return "\\%03o" % o # necessary escapes
-    return c
-  return "".join([escape(c) for c in text])

gerrit_upload.py

@@ -1,155 +0,0 @@
-#
-# Copyright (C) 2008 The Android Open Source Project
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import getpass
-import os
-import sys
-from tempfile import mkstemp
-
-from codereview.proto_client import HttpRpc, Proxy
-from codereview.review_pb2 import ReviewService_Stub
-from codereview.upload_bundle_pb2 import *
-from git_command import GitCommand
-from error import UploadError
-
-try:
-  import readline
-except ImportError:
-  pass
-
-MAX_SEGMENT_SIZE = 1020 * 1024
-
-def _GetRpcServer(email, server, save_cookies):
-  """Returns an RpcServer.
-
-  Returns:
-    A new RpcServer, on which RPC calls can be made.
-  """
-
-  def GetUserCredentials():
-    """Prompts the user for a username and password."""
-    e = email
-    if e is None:
-      e = raw_input("Email: ").strip()
-    password = getpass.getpass("Password for %s: " % e)
-    return (e, password)
-
-  # If this is the dev_appserver, use fake authentication.
-  lc_server = server.lower()
-  if lc_server == "localhost" or lc_server.startswith("localhost:"):
-    if email is None:
-      email = "test@example.com"
-    server = HttpRpc(
-        server,
-        lambda: (email, "password"),
-        extra_headers={"Cookie":
-                       'dev_appserver_login="%s:False"' % email})
-    # Don't try to talk to ClientLogin.
-    server.authenticated = True
-    return server
-
-  if save_cookies:
-    cookie_file = ".gerrit_cookies"
-  else:
-    cookie_file = None
-
-  return HttpRpc(server, GetUserCredentials,
-                 cookie_file=cookie_file)
-
-def UploadBundle(project,
-                 server,
-                 email,
-                 dest_project,
-                 dest_branch,
-                 src_branch,
-                 bases,
-                 save_cookies=True):
-
-  srv = _GetRpcServer(email, server, save_cookies)
-  review = Proxy(ReviewService_Stub(srv))
-  tmp_fd, tmp_bundle = mkstemp(".bundle", ".gpq")
-  os.close(tmp_fd)
-
-  srcid = project.bare_git.rev_parse(src_branch)
-  revlist = project._revlist(src_branch, *bases)
-
-  if srcid not in revlist:
-    # This can happen if src_branch is an annotated tag
-    #
-    revlist.append(srcid)
-  revlist_size = len(revlist) * 42
-
-  try:
-    cmd = ['bundle', 'create', tmp_bundle, src_branch]
-    cmd.extend(bases)
-    if GitCommand(project, cmd).Wait() != 0:
-      raise UploadError('cannot create bundle')
-    fd = open(tmp_bundle, "rb")
-
-    bundle_id = None
-    segment_id = 0
-    next_data = fd.read(MAX_SEGMENT_SIZE - revlist_size)
-
-    while True:
-      this_data = next_data
-      next_data = fd.read(MAX_SEGMENT_SIZE)
-      segment_id += 1
-
-      if bundle_id is None:
-        req = UploadBundleRequest()
-        req.dest_project = str(dest_project)
-        req.dest_branch = str(dest_branch)
-        for c in revlist:
-          req.contained_object.append(c)
-      else:
-        req = UploadBundleContinue()
-        req.bundle_id = bundle_id
-        req.segment_id = segment_id
-
-      req.bundle_data = this_data
-      if len(next_data) > 0:
-        req.partial_upload = True
-      else:
-        req.partial_upload = False
-
-      if bundle_id is None:
-        rsp = review.UploadBundle(req)
-      else:
-        rsp = review.ContinueBundle(req)
-
-      if rsp.status_code == UploadBundleResponse.CONTINUE:
-        bundle_id = rsp.bundle_id
-      elif rsp.status_code == UploadBundleResponse.RECEIVED:
-        bundle_id = rsp.bundle_id
-        return bundle_id
-      else:
-        if rsp.status_code == UploadBundleResponse.UNKNOWN_PROJECT:
-          reason = 'unknown project "%s"' % dest_project
-        elif rsp.status_code == UploadBundleResponse.UNKNOWN_BRANCH:
-          reason = 'unknown branch "%s"' % dest_branch
-        elif rsp.status_code == UploadBundleResponse.UNKNOWN_BUNDLE:
-          reason = 'unknown bundle'
-        elif rsp.status_code == UploadBundleResponse.NOT_BUNDLE_OWNER:
-          reason = 'not bundle owner'
-        elif rsp.status_code == UploadBundleResponse.BUNDLE_CLOSED:
-          reason = 'bundle closed'
-        elif rsp.status_code == UploadBundleResponse.UNAUTHORIZED_USER:
-          reason = ('Unauthorized user.  Visit http://%s/hello to sign up.'
-                    % server)
-        else:
-          reason = 'unknown error ' + str(rsp.status_code)
-        raise UploadError(reason)
-  finally:
-    os.unlink(tmp_bundle)

git_command.py

@@ -13,31 +13,91 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
 import os
 import sys
 import subprocess
+import tempfile
+from signal import SIGTERM
+
 from error import GitError
+import platform_utils
+from trace import REPO_TRACE, IsTrace, Trace
+from wrapper import Wrapper
 
 GIT = 'git'
 MIN_GIT_VERSION = (1, 5, 4)
 GIT_DIR = 'GIT_DIR'
-REPO_TRACE = 'REPO_TRACE'
 
 LAST_GITDIR = None
 LAST_CWD = None
-try:
-  TRACE = os.environ[REPO_TRACE] == '1'
-except KeyError:
-  TRACE = False
 
+_ssh_proxy_path = None
+_ssh_sock_path = None
+_ssh_clients = []
+
+def ssh_sock(create=True):
+  global _ssh_sock_path
+  if _ssh_sock_path is None:
+    if not create:
+      return None
+    tmp_dir = '/tmp'
+    if not os.path.exists(tmp_dir):
+      tmp_dir = tempfile.gettempdir()
+    _ssh_sock_path = os.path.join(
+      tempfile.mkdtemp('', 'ssh-', tmp_dir),
+      'master-%r@%h:%p')
+  return _ssh_sock_path
+
+def _ssh_proxy():
+  global _ssh_proxy_path
+  if _ssh_proxy_path is None:
+    _ssh_proxy_path = os.path.join(
+      os.path.dirname(__file__),
+      'git_ssh')
+  return _ssh_proxy_path
+
+def _add_ssh_client(p):
+  _ssh_clients.append(p)
+
+def _remove_ssh_client(p):
+  try:
+    _ssh_clients.remove(p)
+  except ValueError:
+    pass
+
+def terminate_ssh_clients():
+  global _ssh_clients
+  for p in _ssh_clients:
+    try:
+      os.kill(p.pid, SIGTERM)
+      p.wait()
+    except OSError:
+      pass
+  _ssh_clients = []
+
+_git_version = None
 
 class _GitCall(object):
   def version(self):
     p = GitCommand(None, ['--version'], capture_stdout=True)
     if p.Wait() == 0:
-      return p.stdout
+      if hasattr(p.stdout, 'decode'):
+        return p.stdout.decode('utf-8')
+      else:
+        return p.stdout
     return None
 
+  def version_tuple(self):
+    global _git_version
+    if _git_version is None:
+      ver_str = git.version()
+      _git_version = Wrapper().ParseGitVersion(ver_str)
+      if _git_version is None:
+        print('fatal: "%s" unsupported' % ver_str, file=sys.stderr)
+        sys.exit(1)
+    return _git_version
+
   def __getattr__(self, name):
     name = name.replace('_','-')
     def fun(*cmdv):
@@ -47,6 +107,19 @@ class _GitCall(object):
     return fun
 git = _GitCall()
 
+def git_require(min_version, fail=False):
+  git_version = git.version_tuple()
+  if min_version <= git_version:
+    return True
+  if fail:
+    need = '.'.join(map(str, min_version))
+    print('fatal: git %s or later required' % need, file=sys.stderr)
+    sys.exit(1)
+  return False
+
+def _setenv(env, name, value):
+  env[name] = value.encode()
+
 class GitCommand(object):
   def __init__(self,
                project,
@@ -56,22 +129,39 @@ class GitCommand(object):
                capture_stdout = False,
                capture_stderr = False,
                disable_editor = False,
+               ssh_proxy = False,
                cwd = None,
                gitdir = None):
-    env = dict(os.environ)
+    env = os.environ.copy()
 
-    for e in [REPO_TRACE,
+    for key in [REPO_TRACE,
               GIT_DIR,
               'GIT_ALTERNATE_OBJECT_DIRECTORIES',
               'GIT_OBJECT_DIRECTORY',
               'GIT_WORK_TREE',
               'GIT_GRAFT_FILE',
               'GIT_INDEX_FILE']:
-      if e in env:
-        del env[e]
+      if key in env:
+        del env[key]
+
+    # If we are not capturing std* then need to print it.
+    self.tee = {'stdout': not capture_stdout, 'stderr': not capture_stderr}
 
     if disable_editor:
-      env['GIT_EDITOR'] = ':'
+      _setenv(env, 'GIT_EDITOR', ':')
+    if ssh_proxy:
+      _setenv(env, 'REPO_SSH_SOCK', ssh_sock())
+      _setenv(env, 'GIT_SSH', _ssh_proxy())
+      _setenv(env, 'GIT_SSH_VARIANT', 'ssh')
+    if 'http_proxy' in env and 'darwin' == sys.platform:
+      s = "'http.proxy=%s'" % (env['http_proxy'],)
+      p = env.get('GIT_CONFIG_PARAMETERS')
+      if p is not None:
+        s = p + ' ' + s
+      _setenv(env, 'GIT_CONFIG_PARAMETERS', s)
+    if 'GIT_ALLOW_PROTOCOL' not in env:
+      _setenv(env, 'GIT_ALLOW_PROTOCOL',
+              'file:git:http:https:ssh:persistent-http:persistent-https:sso:rpc')
 
     if project:
       if not cwd:
@@ -82,26 +172,25 @@ class GitCommand(object):
     command = [GIT]
     if bare:
       if gitdir:
-        env[GIT_DIR] = gitdir
+        _setenv(env, GIT_DIR, gitdir)
       cwd = None
-    command.extend(cmdv)
+    command.append(cmdv[0])
+    # Need to use the --progress flag for fetch/clone so output will be
+    # displayed as by default git only does progress output if stderr is a TTY.
+    if sys.stderr.isatty() and cmdv[0] in ('fetch', 'clone'):
+      if '--progress' not in cmdv and '--quiet' not in cmdv:
+        command.append('--progress')
+    command.extend(cmdv[1:])
 
     if provide_stdin:
       stdin = subprocess.PIPE
     else:
       stdin = None
 
-    if capture_stdout:
-      stdout = subprocess.PIPE
-    else:
-      stdout = None
+    stdout = subprocess.PIPE
+    stderr = subprocess.PIPE
 
-    if capture_stderr:
-      stderr = subprocess.PIPE
-    else:
-      stderr = None
-
-    if TRACE:
+    if IsTrace():
       global LAST_CWD
       global LAST_GITDIR
 
@@ -127,7 +216,7 @@ class GitCommand(object):
         dbg += ' 1>|'
       if stderr == subprocess.PIPE:
         dbg += ' 2>|'
-      print >>sys.stderr, dbg
+      Trace('%s', dbg)
 
     try:
       p = subprocess.Popen(command,
@@ -136,29 +225,45 @@ class GitCommand(object):
                            stdin = stdin,
                            stdout = stdout,
                            stderr = stderr)
-    except Exception, e:
+    except Exception as e:
       raise GitError('%s: %s' % (command[1], e))
 
+    if ssh_proxy:
+      _add_ssh_client(p)
+
     self.process = p
     self.stdin = p.stdin
 
   def Wait(self):
+    try:
+      p = self.process
+      rc = self._CaptureOutput()
+    finally:
+      _remove_ssh_client(p)
+    return rc
+
+  def _CaptureOutput(self):
     p = self.process
+    s_in = platform_utils.FileDescriptorStreams.create()
+    s_in.add(p.stdout, sys.stdout, 'stdout')
+    s_in.add(p.stderr, sys.stderr, 'stderr')
+    self.stdout = ''
+    self.stderr = ''
 
-    if p.stdin:
-      p.stdin.close()
-      self.stdin = None
-
-    if p.stdout:
-      self.stdout = p.stdout.read()
-      p.stdout.close()
-    else:
-      p.stdout = None
-
-    if p.stderr:
-      self.stderr = p.stderr.read()
-      p.stderr.close()
-    else:
-      p.stderr = None
-
-    return self.process.wait()
+    while not s_in.is_done:
+      in_ready = s_in.select()
+      for s in in_ready:
+        buf = s.read()
+        if not buf:
+          s_in.remove(s)
+          continue
+        if not hasattr(buf, 'encode'):
+          buf = buf.decode()
+        if s.std_name == 'stdout':
+          self.stdout += buf
+        else:
+          self.stderr += buf
+        if self.tee[s.std_name]:
+          s.dest.write(buf)
+          s.dest.flush()
+    return p.wait()

git_config.py

@@ -13,19 +13,70 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
+
+import contextlib
+import errno
+import json
 import os
 import re
+import ssl
+import subprocess
 import sys
-from error import GitError
-from git_command import GitCommand
+try:
+  import threading as _threading
+except ImportError:
+  import dummy_threading as _threading
+import time
 
-R_HEADS = 'refs/heads/'
-R_TAGS  = 'refs/tags/'
-ID_RE = re.compile('^[0-9a-f]{40}$')
+from pyversion import is_python3
+if is_python3():
+  import urllib.request
+  import urllib.error
+else:
+  import urllib2
+  import imp
+  urllib = imp.new_module('urllib')
+  urllib.request = urllib2
+  urllib.error = urllib2
+
+from signal import SIGTERM
+from error import GitError, UploadError
+import platform_utils
+from trace import Trace
+if is_python3():
+  from http.client import HTTPException
+else:
+  from httplib import HTTPException
+
+from git_command import GitCommand
+from git_command import ssh_sock
+from git_command import terminate_ssh_clients
+from git_refs import R_CHANGES, R_HEADS, R_TAGS
+
+ID_RE = re.compile(r'^[0-9a-f]{40}$')
+
+REVIEW_CACHE = dict()
+
+def IsChange(rev):
+  return rev.startswith(R_CHANGES)
 
 def IsId(rev):
   return ID_RE.match(rev)
 
+def IsTag(rev):
+  return rev.startswith(R_TAGS)
+
+def IsImmutable(rev):
+    return IsChange(rev) or IsId(rev) or IsTag(rev)
+
+def _key(name):
+  parts = name.split('.')
+  if len(parts) < 2:
+    return name.lower()
+  parts[ 0] = parts[ 0].lower()
+  parts[-1] = parts[-1].lower()
+  return '.'.join(parts)
 
 class GitConfig(object):
   _ForUser = None
@@ -33,26 +84,32 @@ class GitConfig(object):
   @classmethod
   def ForUser(cls):
     if cls._ForUser is None:
-      cls._ForUser = cls(file = os.path.expanduser('~/.gitconfig'))
+      cls._ForUser = cls(configfile = os.path.expanduser('~/.gitconfig'))
     return cls._ForUser
 
   @classmethod
   def ForRepository(cls, gitdir, defaults=None):
-    return cls(file = os.path.join(gitdir, 'config'),
+    return cls(configfile = os.path.join(gitdir, 'config'),
                defaults = defaults)
 
-  def __init__(self, file, defaults=None):
-    self.file = file
+  def __init__(self, configfile, defaults=None, jsonFile=None):
+    self.file = configfile
     self.defaults = defaults
     self._cache_dict = None
+    self._section_dict = None
     self._remotes = {}
     self._branches = {}
 
+    self._json = jsonFile
+    if self._json is None:
+      self._json = os.path.join(
+        os.path.dirname(self.file),
+        '.repo_' + os.path.basename(self.file) + '.json')
+
   def Has(self, name, include_defaults = True):
     """Return true if this configuration file has the key.
     """
-    name = name.lower()
-    if name in self._cache:
+    if _key(name) in self._cache:
       return True
     if include_defaults and self.defaults:
       return self.defaults.Has(name, include_defaults = True)
@@ -74,22 +131,20 @@ class GitConfig(object):
       return False
     return None
 
-  def GetString(self, name, all=False):
+  def GetString(self, name, all_keys=False):
     """Get the first value for a key, or None if it is not defined.
 
        This configuration file is used first, if the key is not
-       defined or all = True then the defaults are also searched.
+       defined or all_keys = True then the defaults are also searched.
     """
-    name = name.lower()
-
     try:
-      v = self._cache[name]
+      v = self._cache[_key(name)]
     except KeyError:
       if self.defaults:
-        return self.defaults.GetString(name, all = all)
+        return self.defaults.GetString(name, all_keys = all_keys)
       v = []
 
-    if not all:
+    if not all_keys:
       if v:
         return v[0]
       return None
@@ -97,7 +152,7 @@ class GitConfig(object):
     r = []
     r.extend(v)
     if self.defaults:
-      r.extend(self.defaults.GetString(name, all = True))
+      r.extend(self.defaults.GetString(name, all_keys = True))
     return r
 
   def SetString(self, name, value):
@@ -107,16 +162,16 @@ class GitConfig(object):
        The supplied value should be either a string,
        or a list of strings (to store multiple values).
     """
-    name = name.lower()
+    key = _key(name)
 
     try:
-      old = self._cache[name]
+      old = self._cache[key]
     except KeyError:
       old = []
 
     if value is None:
       if old:
-        del self._cache[name]
+        del self._cache[key]
         self._do('--unset-all', name)
 
     elif isinstance(value, list):
@@ -127,13 +182,13 @@ class GitConfig(object):
         self.SetString(name, value[0])
 
       elif old != value:
-        self._cache[name] = list(value)
+        self._cache[key] = list(value)
         self._do('--replace-all', name, value[0])
-        for i in xrange(1, len(value)):
+        for i in range(1, len(value)):
           self._do('--add', name, value[i])
 
     elif len(old) != 1 or old[0] != value:
-      self._cache[name] = [value]
+      self._cache[key] = [value]
       self._do('--replace-all', name, value)
 
   def GetRemote(self, name):
@@ -156,6 +211,47 @@ class GitConfig(object):
       self._branches[b.name] = b
     return b
 
+  def GetSubSections(self, section):
+    """List all subsection names matching $section.*.*
+    """
+    return self._sections.get(section, set())
+
+  def HasSection(self, section, subsection = ''):
+    """Does at least one key in section.subsection exist?
+    """
+    try:
+      return subsection in self._sections[section]
+    except KeyError:
+      return False
+
+  def UrlInsteadOf(self, url):
+    """Resolve any url.*.insteadof references.
+    """
+    for new_url in self.GetSubSections('url'):
+      for old_url in self.GetString('url.%s.insteadof' % new_url, True):
+        if old_url is not None and url.startswith(old_url):
+          return new_url + url[len(old_url):]
+    return url
+
+  @property
+  def _sections(self):
+    d = self._section_dict
+    if d is None:
+      d = {}
+      for name in self._cache.keys():
+        p = name.split('.')
+        if 2 == len(p):
+          section = p[0]
+          subsect = ''
+        else:
+          section = p[0]
+          subsect = '.'.join(p[1:-1])
+        if section not in d:
+          d[section] = set()
+        d[section].add(subsect)
+        self._section_dict = d
+    return d
+
   @property
   def _cache(self):
     if self._cache_dict is None:
@@ -163,21 +259,67 @@ class GitConfig(object):
     return self._cache_dict
 
   def _Read(self):
-    d = self._do('--null', '--list')
-    c = {}
-    while d:
-      lf = d.index('\n')
-      nul = d.index('\0', lf + 1)
+    d = self._ReadJson()
+    if d is None:
+      d = self._ReadGit()
+      self._SaveJson(d)
+    return d
 
-      key = d[0:lf]
-      val = d[lf + 1:nul]
+  def _ReadJson(self):
+    try:
+      if os.path.getmtime(self._json) \
+      <= os.path.getmtime(self.file):
+        platform_utils.remove(self._json)
+        return None
+    except OSError:
+      return None
+    try:
+      Trace(': parsing %s', self.file)
+      fd = open(self._json)
+      try:
+        return json.load(fd)
+      finally:
+        fd.close()
+    except (IOError, ValueError):
+      platform_utils.remove(self._json)
+      return None
+
+  def _SaveJson(self, cache):
+    try:
+      fd = open(self._json, 'w')
+      try:
+        json.dump(cache, fd, indent=2)
+      finally:
+        fd.close()
+    except (IOError, TypeError):
+      if os.path.exists(self._json):
+        platform_utils.remove(self._json)
+
+  def _ReadGit(self):
+    """
+    Read configuration data from git.
+
+    This internal method populates the GitConfig cache.
+
+    """
+    c = {}
+    d = self._do('--null', '--list')
+    if d is None:
+      return c
+    if not is_python3():
+      d = d.decode('utf-8')
+    for line in d.rstrip('\0').split('\0'):
+      if '\n' in line:
+        key, val = line.split('\n', 1)
+      else:
+        key = line
+        val = None
 
       if key in c:
         c[key].append(val)
       else:
         c[key] = [val]
 
-      d = d[nul + 1:]
     return c
 
   def _do(self, *args):
@@ -250,6 +392,193 @@ class RefSpec(object):
     return s
 
 
+_master_processes = []
+_master_keys = set()
+_ssh_master = True
+_master_keys_lock = None
+
+def init_ssh():
+  """Should be called once at the start of repo to init ssh master handling.
+
+  At the moment, all we do is to create our lock.
+  """
+  global _master_keys_lock
+  assert _master_keys_lock is None, "Should only call init_ssh once"
+  _master_keys_lock = _threading.Lock()
+
+def _open_ssh(host, port=None):
+  global _ssh_master
+
+  # Acquire the lock.  This is needed to prevent opening multiple masters for
+  # the same host when we're running "repo sync -jN" (for N > 1) _and_ the
+  # manifest <remote fetch="ssh://xyz"> specifies a different host from the
+  # one that was passed to repo init.
+  _master_keys_lock.acquire()
+  try:
+
+    # Check to see whether we already think that the master is running; if we
+    # think it's already running, return right away.
+    if port is not None:
+      key = '%s:%s' % (host, port)
+    else:
+      key = host
+
+    if key in _master_keys:
+      return True
+
+    if not _ssh_master \
+    or 'GIT_SSH' in os.environ \
+    or sys.platform in ('win32', 'cygwin'):
+      # failed earlier, or cygwin ssh can't do this
+      #
+      return False
+
+    # We will make two calls to ssh; this is the common part of both calls.
+    command_base = ['ssh',
+                     '-o','ControlPath %s' % ssh_sock(),
+                     host]
+    if port is not None:
+      command_base[1:1] = ['-p', str(port)]
+
+    # Since the key wasn't in _master_keys, we think that master isn't running.
+    # ...but before actually starting a master, we'll double-check.  This can
+    # be important because we can't tell that that 'git@myhost.com' is the same
+    # as 'myhost.com' where "User git" is setup in the user's ~/.ssh/config file.
+    check_command = command_base + ['-O','check']
+    try:
+      Trace(': %s', ' '.join(check_command))
+      check_process = subprocess.Popen(check_command,
+                                       stdout=subprocess.PIPE,
+                                       stderr=subprocess.PIPE)
+      check_process.communicate() # read output, but ignore it...
+      isnt_running = check_process.wait()
+
+      if not isnt_running:
+        # Our double-check found that the master _was_ infact running.  Add to
+        # the list of keys.
+        _master_keys.add(key)
+        return True
+    except Exception:
+      # Ignore excpetions.  We we will fall back to the normal command and print
+      # to the log there.
+      pass
+
+    command = command_base[:1] + \
+              ['-M', '-N'] + \
+              command_base[1:]
+    try:
+      Trace(': %s', ' '.join(command))
+      p = subprocess.Popen(command)
+    except Exception as e:
+      _ssh_master = False
+      print('\nwarn: cannot enable ssh control master for %s:%s\n%s'
+             % (host,port, str(e)), file=sys.stderr)
+      return False
+
+    time.sleep(1)
+    ssh_died = (p.poll() is not None)
+    if ssh_died:
+      return False
+
+    _master_processes.append(p)
+    _master_keys.add(key)
+    return True
+  finally:
+    _master_keys_lock.release()
+
+def close_ssh():
+  global _master_keys_lock
+
+  terminate_ssh_clients()
+
+  for p in _master_processes:
+    try:
+      os.kill(p.pid, SIGTERM)
+      p.wait()
+    except OSError:
+      pass
+  del _master_processes[:]
+  _master_keys.clear()
+
+  d = ssh_sock(create=False)
+  if d:
+    try:
+      platform_utils.rmdir(os.path.dirname(d))
+    except OSError:
+      pass
+
+  # We're done with the lock, so we can delete it.
+  _master_keys_lock = None
+
+URI_SCP = re.compile(r'^([^@:]*@?[^:/]{1,}):')
+URI_ALL = re.compile(r'^([a-z][a-z+-]*)://([^@/]*@?[^/]*)/')
+
+def GetSchemeFromUrl(url):
+  m = URI_ALL.match(url)
+  if m:
+    return m.group(1)
+  return None
+
+@contextlib.contextmanager
+def GetUrlCookieFile(url, quiet):
+  if url.startswith('persistent-'):
+    try:
+      p = subprocess.Popen(
+          ['git-remote-persistent-https', '-print_config', url],
+          stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+          stderr=subprocess.PIPE)
+      try:
+        cookieprefix = 'http.cookiefile='
+        proxyprefix = 'http.proxy='
+        cookiefile = None
+        proxy = None
+        for line in p.stdout:
+          line = line.strip()
+          if line.startswith(cookieprefix):
+            cookiefile = os.path.expanduser(line[len(cookieprefix):])
+          if line.startswith(proxyprefix):
+            proxy = line[len(proxyprefix):]
+        # Leave subprocess open, as cookie file may be transient.
+        if cookiefile or proxy:
+          yield cookiefile, proxy
+          return
+      finally:
+        p.stdin.close()
+        if p.wait():
+          err_msg = p.stderr.read()
+          if ' -print_config' in err_msg:
+            pass  # Persistent proxy doesn't support -print_config.
+          elif not quiet:
+            print(err_msg, file=sys.stderr)
+    except OSError as e:
+      if e.errno == errno.ENOENT:
+        pass  # No persistent proxy.
+      raise
+  cookiefile = GitConfig.ForUser().GetString('http.cookiefile')
+  if cookiefile:
+    cookiefile = os.path.expanduser(cookiefile)
+  yield cookiefile, None
+
+def _preconnect(url):
+  m = URI_ALL.match(url)
+  if m:
+    scheme = m.group(1)
+    host = m.group(2)
+    if ':' in host:
+      host, port = host.split(':')
+    else:
+      port = None
+    if scheme in ('ssh', 'git+ssh', 'ssh+git'):
+      return _open_ssh(host, port)
+    return False
+
+  m = URI_SCP.match(url)
+  if m:
+    host = m.group(1)
+    return _open_ssh(host)
+
+  return False
+
 class Remote(object):
   """Configuration options related to a remote.
   """
@@ -257,17 +586,105 @@ class Remote(object):
     self._config = config
     self.name = name
     self.url = self._Get('url')
+    self.pushUrl = self._Get('pushurl')
     self.review = self._Get('review')
     self.projectname = self._Get('projectname')
-    self.fetch = map(lambda x: RefSpec.FromString(x),
-                     self._Get('fetch', all=True))
+    self.fetch = list(map(RefSpec.FromString,
+                      self._Get('fetch', all_keys=True)))
+    self._review_url = None
+
+  def _InsteadOf(self):
+    globCfg = GitConfig.ForUser()
+    urlList = globCfg.GetSubSections('url')
+    longest = ""
+    longestUrl = ""
+
+    for url in urlList:
+      key = "url." + url + ".insteadOf"
+      insteadOfList = globCfg.GetString(key, all_keys=True)
+
+      for insteadOf in insteadOfList:
+        if self.url.startswith(insteadOf) \
+        and len(insteadOf) > len(longest):
+          longest = insteadOf
+          longestUrl = url
+
+    if len(longest) == 0:
+      return self.url
+
+    return self.url.replace(longest, longestUrl, 1)
+
+  def PreConnectFetch(self):
+    connectionUrl = self._InsteadOf()
+    return _preconnect(connectionUrl)
+
+  def ReviewUrl(self, userEmail, validate_certs):
+    if self._review_url is None:
+      if self.review is None:
+        return None
+
+      u = self.review
+      if u.startswith('persistent-'):
+        u = u[len('persistent-'):]
+      if u.split(':')[0] not in ('http', 'https', 'sso', 'ssh'):
+        u = 'http://%s' % u
+      if u.endswith('/Gerrit'):
+        u = u[:len(u) - len('/Gerrit')]
+      if u.endswith('/ssh_info'):
+        u = u[:len(u) - len('/ssh_info')]
+      if not u.endswith('/'):
+        u += '/'
+      http_url = u
+
+      if u in REVIEW_CACHE:
+        self._review_url = REVIEW_CACHE[u]
+      elif 'REPO_HOST_PORT_INFO' in os.environ:
+        host, port = os.environ['REPO_HOST_PORT_INFO'].split()
+        self._review_url = self._SshReviewUrl(userEmail, host, port)
+        REVIEW_CACHE[u] = self._review_url
+      elif u.startswith('sso:') or u.startswith('ssh:'):
+        self._review_url = u  # Assume it's right
+        REVIEW_CACHE[u] = self._review_url
+      elif 'REPO_IGNORE_SSH_INFO' in os.environ:
+        self._review_url = http_url
+        REVIEW_CACHE[u] = self._review_url
+      else:
+        try:
+          info_url = u + 'ssh_info'
+          if not validate_certs:
+              context = ssl._create_unverified_context()
+              info = urllib.request.urlopen(info_url, context=context).read()
+          else:
+              info = urllib.request.urlopen(info_url).read()
+          if info == 'NOT_AVAILABLE' or '<' in info:
+            # If `info` contains '<', we assume the server gave us some sort
+            # of HTML response back, like maybe a login page.
+            #
+            # Assume HTTP if SSH is not enabled or ssh_info doesn't look right.
+            self._review_url = http_url
+          else:
+            host, port = info.split()
+            self._review_url = self._SshReviewUrl(userEmail, host, port)
+        except urllib.error.HTTPError as e:
+          raise UploadError('%s: %s' % (self.review, str(e)))
+        except urllib.error.URLError as e:
+          raise UploadError('%s: %s' % (self.review, str(e)))
+        except HTTPException as e:
+          raise UploadError('%s: %s' % (self.review, e.__class__.__name__))
+
+        REVIEW_CACHE[u] = self._review_url
+    return self._review_url + self.projectname
+
+  def _SshReviewUrl(self, userEmail, host, port):
+    username = self._config.GetString('review.%s.username' % self.review)
+    if username is None:
+      username = userEmail.split('@')[0]
+    return 'ssh://%s@%s:%s/' % (username, host, port)
 
   def ToLocal(self, rev):
     """Convert a remote revision string to something we have locally.
     """
-    if IsId(rev):
-      return rev
-    if rev.startswith(R_TAGS):
+    if self.name == '.' or IsId(rev):
       return rev
 
     if not rev.startswith('refs/'):
@@ -276,6 +693,10 @@ class Remote(object):
     for spec in self.fetch:
       if spec.SourceMatches(rev):
         return spec.MapSource(rev)
+
+    if not rev.startswith(R_HEADS):
+      return rev
+
     raise GitError('remote %s does not have %s' % (self.name, rev))
 
   def WritesTo(self, ref):
@@ -299,17 +720,21 @@ class Remote(object):
     """Save this remote to the configuration.
     """
     self._Set('url', self.url)
+    if self.pushUrl is not None:
+      self._Set('pushurl', self.pushUrl + '/' + self.projectname)
+    else:
+      self._Set('pushurl', self.pushUrl)
     self._Set('review', self.review)
     self._Set('projectname', self.projectname)
-    self._Set('fetch', map(lambda x: str(x), self.fetch))
+    self._Set('fetch', list(map(str, self.fetch)))
 
   def _Set(self, key, value):
     key = 'remote.%s.%s' % (self.name, key)
     return self._config.SetString(key, value)
 
-  def _Get(self, key, all=False):
+  def _Get(self, key, all_keys=False):
     key = 'remote.%s.%s' % (self.name, key)
-    return self._config.GetString(key, all = all)
+    return self._config.GetString(key, all_keys = all_keys)
 
 
 class Branch(object):
@@ -337,16 +762,28 @@ class Branch(object):
   def Save(self):
     """Save this branch back into the configuration.
     """
-    self._Set('merge', self.merge)
-    if self.remote:
-      self._Set('remote', self.remote.name)
+    if self._config.HasSection('branch', self.name):
+      if self.remote:
+        self._Set('remote', self.remote.name)
+      else:
+        self._Set('remote', None)
+      self._Set('merge', self.merge)
+
     else:
-      self._Set('remote', None)
+      fd = open(self._config.file, 'a')
+      try:
+        fd.write('[branch "%s"]\n' % self.name)
+        if self.remote:
+          fd.write('\tremote = %s\n' % self.remote.name)
+        if self.merge:
+          fd.write('\tmerge = %s\n' % self.merge)
+      finally:
+        fd.close()
 
   def _Set(self, key, value):
     key = 'branch.%s.%s' % (self.name, key)
     return self._config.SetString(key, value)
 
-  def _Get(self, key, all=False):
+  def _Get(self, key, all_keys=False):
     key = 'branch.%s.%s' % (self.name, key)
-    return self._config.GetString(key, all = all)
+    return self._config.GetString(key, all_keys = all_keys)

git_refs.py

@@ -0,0 +1,168 @@
+#
+# Copyright (C) 2009 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+from trace import Trace
+import platform_utils
+
+HEAD      = 'HEAD'
+R_CHANGES = 'refs/changes/'
+R_HEADS   = 'refs/heads/'
+R_TAGS    = 'refs/tags/'
+R_PUB     = 'refs/published/'
+R_M       = 'refs/remotes/m/'
+
+
+class GitRefs(object):
+  def __init__(self, gitdir):
+    self._gitdir = gitdir
+    self._phyref = None
+    self._symref = None
+    self._mtime = {}
+
+  @property
+  def all(self):
+    self._EnsureLoaded()
+    return self._phyref
+
+  def get(self, name):
+    try:
+      return self.all[name]
+    except KeyError:
+      return ''
+
+  def deleted(self, name):
+    if self._phyref is not None:
+      if name in self._phyref:
+        del self._phyref[name]
+
+      if name in self._symref:
+        del self._symref[name]
+
+      if name in self._mtime:
+        del self._mtime[name]
+
+  def symref(self, name):
+    try:
+      self._EnsureLoaded()
+      return self._symref[name]
+    except KeyError:
+      return ''
+
+  def _EnsureLoaded(self):
+    if self._phyref is None or self._NeedUpdate():
+      self._LoadAll()
+
+  def _NeedUpdate(self):
+    Trace(': scan refs %s', self._gitdir)
+
+    for name, mtime in self._mtime.items():
+      try:
+        if mtime != os.path.getmtime(os.path.join(self._gitdir, name)):
+          return True
+      except OSError:
+        return True
+    return False
+
+  def _LoadAll(self):
+    Trace(': load refs %s', self._gitdir)
+
+    self._phyref = {}
+    self._symref = {}
+    self._mtime = {}
+
+    self._ReadPackedRefs()
+    self._ReadLoose('refs/')
+    self._ReadLoose1(os.path.join(self._gitdir, HEAD), HEAD)
+
+    scan = self._symref
+    attempts = 0
+    while scan and attempts < 5:
+      scan_next = {}
+      for name, dest in scan.items():
+        if dest in self._phyref:
+          self._phyref[name] = self._phyref[dest]
+        else:
+          scan_next[name] = dest
+      scan = scan_next
+      attempts += 1
+
+  def _ReadPackedRefs(self):
+    path = os.path.join(self._gitdir, 'packed-refs')
+    try:
+      fd = open(path, 'r')
+      mtime = os.path.getmtime(path)
+    except IOError:
+      return
+    except OSError:
+      return
+    try:
+      for line in fd:
+        line = str(line)
+        if line[0] == '#':
+          continue
+        if line[0] == '^':
+          continue
+
+        line = line[:-1]
+        p = line.split(' ')
+        ref_id = p[0]
+        name = p[1]
+
+        self._phyref[name] = ref_id
+    finally:
+      fd.close()
+    self._mtime['packed-refs'] = mtime
+
+  def _ReadLoose(self, prefix):
+    base = os.path.join(self._gitdir, prefix)
+    for name in platform_utils.listdir(base):
+      p = os.path.join(base, name)
+      if platform_utils.isdir(p):
+        self._mtime[prefix] = os.path.getmtime(base)
+        self._ReadLoose(prefix + name + '/')
+      elif name.endswith('.lock'):
+        pass
+      else:
+        self._ReadLoose1(p, prefix + name)
+
+  def _ReadLoose1(self, path, name):
+    try:
+      fd = open(path)
+    except IOError:
+      return
+
+    try:
+      try:
+        mtime = os.path.getmtime(path)
+        ref_id = fd.readline()
+      except (IOError, OSError):
+        return
+    finally:
+      fd.close()
+
+    try:
+      ref_id = ref_id.decode()
+    except AttributeError:
+      pass
+    if not ref_id:
+      return
+    ref_id = ref_id[:-1]
+
+    if ref_id.startswith('ref: '):
+      self._symref[name] = ref_id[5:]
+    else:
+      self._phyref[name] = ref_id
+    self._mtime[name] = mtime

git_ssh

@@ -0,0 +1,2 @@
+#!/bin/sh
+exec ssh -o "ControlMaster no" -o "ControlPath $REPO_SSH_SOCK" "$@"

gitc_utils.py

@@ -0,0 +1,154 @@
+#
+# Copyright (C) 2015 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import os
+import platform
+import re
+import sys
+import time
+
+import git_command
+import git_config
+import wrapper
+
+from error import ManifestParseError
+
+NUM_BATCH_RETRIEVE_REVISIONID = 32
+
+def get_gitc_manifest_dir():
+  return wrapper.Wrapper().get_gitc_manifest_dir()
+
+def parse_clientdir(gitc_fs_path):
+  return wrapper.Wrapper().gitc_parse_clientdir(gitc_fs_path)
+
+def _set_project_revisions(projects):
+  """Sets the revisionExpr for a list of projects.
+
+  Because of the limit of open file descriptors allowed, length of projects
+  should not be overly large. Recommend calling this function multiple times
+  with each call not exceeding NUM_BATCH_RETRIEVE_REVISIONID projects.
+
+  @param projects: List of project objects to set the revionExpr for.
+  """
+  # Retrieve the commit id for each project based off of it's current
+  # revisionExpr and it is not already a commit id.
+  project_gitcmds = [(
+      project, git_command.GitCommand(None,
+                                      ['ls-remote',
+                                       project.remote.url,
+                                       project.revisionExpr],
+                                      capture_stdout=True, cwd='/tmp'))
+      for project in projects if not git_config.IsId(project.revisionExpr)]
+  for proj, gitcmd in project_gitcmds:
+    if gitcmd.Wait():
+      print('FATAL: Failed to retrieve revisionExpr for %s' % proj)
+      sys.exit(1)
+    revisionExpr = gitcmd.stdout.split('\t')[0]
+    if not revisionExpr:
+      raise(ManifestParseError('Invalid SHA-1 revision project %s (%s)' %
+                               (proj.remote.url, proj.revisionExpr)))
+    proj.revisionExpr = revisionExpr
+
+def _manifest_groups(manifest):
+  """Returns the manifest group string that should be synced
+
+  This is the same logic used by Command.GetProjects(), which is used during
+  repo sync
+
+  @param manifest: The XmlManifest object
+  """
+  mp = manifest.manifestProject
+  groups = mp.config.GetString('manifest.groups')
+  if not groups:
+    groups = 'default,platform-' + platform.system().lower()
+  return groups
+
+def generate_gitc_manifest(gitc_manifest, manifest, paths=None):
+  """Generate a manifest for shafsd to use for this GITC client.
+
+  @param gitc_manifest: Current gitc manifest, or None if there isn't one yet.
+  @param manifest: A GitcManifest object loaded with the current repo manifest.
+  @param paths: List of project paths we want to update.
+  """
+
+  print('Generating GITC Manifest by fetching revision SHAs for each '
+        'project.')
+  if paths is None:
+    paths = manifest.paths.keys()
+
+  groups = [x for x in re.split(r'[,\s]+', _manifest_groups(manifest)) if x]
+
+  # Convert the paths to projects, and filter them to the matched groups.
+  projects = [manifest.paths[p] for p in paths]
+  projects = [p for p in projects if p.MatchesGroups(groups)]
+
+  if gitc_manifest is not None:
+    for path, proj in manifest.paths.iteritems():
+      if not proj.MatchesGroups(groups):
+        continue
+
+      if not proj.upstream and not git_config.IsId(proj.revisionExpr):
+        proj.upstream = proj.revisionExpr
+
+      if not path in gitc_manifest.paths:
+        # Any new projects need their first revision, even if we weren't asked
+        # for them.
+        projects.append(proj)
+      elif not path in paths:
+        # And copy revisions from the previous manifest if we're not updating
+        # them now.
+        gitc_proj = gitc_manifest.paths[path]
+        if gitc_proj.old_revision:
+          proj.revisionExpr = None
+          proj.old_revision = gitc_proj.old_revision
+        else:
+          proj.revisionExpr = gitc_proj.revisionExpr
+
+  index = 0
+  while index < len(projects):
+    _set_project_revisions(
+        projects[index:(index+NUM_BATCH_RETRIEVE_REVISIONID)])
+    index += NUM_BATCH_RETRIEVE_REVISIONID
+
+  if gitc_manifest is not None:
+    for path, proj in gitc_manifest.paths.iteritems():
+      if proj.old_revision and path in paths:
+        # If we updated a project that has been started, keep the old-revision
+        # updated.
+        repo_proj = manifest.paths[path]
+        repo_proj.old_revision = repo_proj.revisionExpr
+        repo_proj.revisionExpr = None
+
+  # Convert URLs from relative to absolute.
+  for _name, remote in manifest.remotes.iteritems():
+    remote.fetchUrl = remote.resolvedFetchUrl
+
+  # Save the manifest.
+  save_manifest(manifest)
+
+def save_manifest(manifest, client_dir=None):
+  """Save the manifest file in the client_dir.
+
+  @param client_dir: Client directory to save the manifest in.
+  @param manifest: Manifest object to save.
+  """
+  if not client_dir:
+    client_dir = manifest.gitc_client_dir
+  with open(os.path.join(client_dir, '.manifest'), 'w') as f:
+    manifest.Save(f, groups=_manifest_groups(manifest))
+  # TODO(sbasi/jorg): Come up with a solution to remove the sleep below.
+  # Give the GITC filesystem time to register the manifest changes.
+  time.sleep(3)

hooks/commit-msg

@@ -0,0 +1,191 @@
+#!/bin/sh
+# From Gerrit Code Review 2.14.6
+#
+# Part of Gerrit Code Review (https://www.gerritcodereview.com/)
+#
+# Copyright (C) 2009 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+unset GREP_OPTIONS
+
+CHANGE_ID_AFTER="Bug|Depends-On|Issue|Test|Feature|Fixes|Fixed"
+MSG="$1"
+
+# Check for, and add if missing, a unique Change-Id
+#
+add_ChangeId() {
+	clean_message=`sed -e '
+		/^diff --git .*/{
+			s///
+			q
+		}
+		/^Signed-off-by:/d
+		/^#/d
+	' "$MSG" | git stripspace`
+	if test -z "$clean_message"
+	then
+		return
+	fi
+
+	# Do not add Change-Id to temp commits
+	if echo "$clean_message" | head -1 | grep -q '^\(fixup\|squash\)!'
+	then
+		return
+	fi
+
+	if test "false" = "`git config --bool --get gerrit.createChangeId`"
+	then
+		return
+	fi
+
+	# Does Change-Id: already exist? if so, exit (no change).
+	if grep -i '^Change-Id:' "$MSG" >/dev/null
+	then
+		return
+	fi
+
+	id=`_gen_ChangeId`
+	T="$MSG.tmp.$$"
+	AWK=awk
+	if [ -x /usr/xpg4/bin/awk ]; then
+		# Solaris AWK is just too broken
+		AWK=/usr/xpg4/bin/awk
+	fi
+
+	# Get core.commentChar from git config or use default symbol
+	commentChar=`git config --get core.commentChar`
+	commentChar=${commentChar:-#}
+
+	# How this works:
+	# - parse the commit message as (textLine+ blankLine*)*
+	# - assume textLine+ to be a footer until proven otherwise
+	# - exception: the first block is not footer (as it is the title)
+	# - read textLine+ into a variable
+	# - then count blankLines
+	# - once the next textLine appears, print textLine+ blankLine* as these
+	#   aren't footer
+	# - in END, the last textLine+ block is available for footer parsing
+	$AWK '
+	BEGIN {
+		# while we start with the assumption that textLine+
+		# is a footer, the first block is not.
+		isFooter = 0
+		footerComment = 0
+		blankLines = 0
+	}
+
+	# Skip lines starting with commentChar without any spaces before it.
+	/^'"$commentChar"'/ { next }
+
+	# Skip the line starting with the diff command and everything after it,
+	# up to the end of the file, assuming it is only patch data.
+	# If more than one line before the diff was empty, strip all but one.
+	/^diff --git / {
+		blankLines = 0
+		while (getline) { }
+		next
+	}
+
+	# Count blank lines outside footer comments
+	/^$/ && (footerComment == 0) {
+		blankLines++
+		next
+	}
+
+	# Catch footer comment
+	/^\[[a-zA-Z0-9-]+:/ && (isFooter == 1) {
+		footerComment = 1
+	}
+
+	/]$/ && (footerComment == 1) {
+		footerComment = 2
+	}
+
+	# We have a non-blank line after blank lines. Handle this.
+	(blankLines > 0) {
+		print lines
+		for (i = 0; i < blankLines; i++) {
+			print ""
+		}
+
+		lines = ""
+		blankLines = 0
+		isFooter = 1
+		footerComment = 0
+	}
+
+	# Detect that the current block is not the footer
+	(footerComment == 0) && (!/^\[?[a-zA-Z0-9-]+:/ || /^[a-zA-Z0-9-]+:\/\//) {
+		isFooter = 0
+	}
+
+	{
+		# We need this information about the current last comment line
+		if (footerComment == 2) {
+			footerComment = 0
+		}
+		if (lines != "") {
+			lines = lines "\n";
+		}
+		lines = lines $0
+	}
+
+	# Footer handling:
+	# If the last block is considered a footer, splice in the Change-Id at the
+	# right place.
+	# Look for the right place to inject Change-Id by considering
+	# CHANGE_ID_AFTER. Keys listed in it (case insensitive) come first,
+	# then Change-Id, then everything else (eg. Signed-off-by:).
+	#
+	# Otherwise just print the last block, a new line and the Change-Id as a
+	# block of its own.
+	END {
+		unprinted = 1
+		if (isFooter == 0) {
+			print lines "\n"
+			lines = ""
+		}
+		changeIdAfter = "^(" tolower("'"$CHANGE_ID_AFTER"'") "):"
+		numlines = split(lines, footer, "\n")
+		for (line = 1; line <= numlines; line++) {
+			if (unprinted && match(tolower(footer[line]), changeIdAfter) != 1) {
+				unprinted = 0
+				print "Change-Id: I'"$id"'"
+			}
+			print footer[line]
+		}
+		if (unprinted) {
+			print "Change-Id: I'"$id"'"
+		}
+	}' "$MSG" > "$T" && mv "$T" "$MSG" || rm -f "$T"
+}
+_gen_ChangeIdInput() {
+	echo "tree `git write-tree`"
+	if parent=`git rev-parse "HEAD^0" 2>/dev/null`
+	then
+		echo "parent $parent"
+	fi
+	echo "author `git var GIT_AUTHOR_IDENT`"
+	echo "committer `git var GIT_COMMITTER_IDENT`"
+	echo
+	printf '%s' "$clean_message"
+}
+_gen_ChangeId() {
+	_gen_ChangeIdInput |
+	git hash-object -t commit --stdin
+}
+
+
+add_ChangeId

hooks/pre-auto-gc

@@ -1,9 +1,9 @@
 #!/bin/sh
 #
 # An example hook script to verify if you are on battery, in case you
-# are running Linux or OS X. Called by git-gc --auto with no arguments.
-# The hook should exit with non-zero status after issuing an appropriate
-# message if it wants to stop the auto repacking.
+# are running Windows, Linux or OS X. Called by git-gc --auto with no
+# arguments. The hook should exit with non-zero status after issuing an
+# appropriate message if it wants to stop the auto repacking.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -19,7 +19,17 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 
-if test -x /sbin/on_ac_power && /sbin/on_ac_power
+if uname -s | grep -q "_NT-"
+then
+	if test -x $SYSTEMROOT/System32/Wbem/wmic
+	then
+		STATUS=$(wmic path win32_battery get batterystatus /format:list | tr -d '\r\n')
+		[ "$STATUS" = "BatteryStatus=2" ] && exit 0 || exit 1
+	fi
+	exit 0
+fi
+
+if test -x /sbin/on_ac_power && (/sbin/on_ac_power;test $? -ne 1)
 then
 	exit 0
 elif test "$(cat /sys/class/power_supply/AC/online 2>/dev/null)" = 1
@@ -35,9 +45,14 @@ elif grep -q "AC Power \+: 1" /proc/pmu/info 2>/dev/null
 then
 	exit 0
 elif test -x /usr/bin/pmset && /usr/bin/pmset -g batt |
-	grep -q "Currently drawing from 'AC Power'"
+	grep -q "drawing from 'AC Power'"
 then
 	exit 0
+elif test -d /sys/bus/acpi/drivers/battery && test 0 = \
+  "$(find /sys/bus/acpi/drivers/battery/ -type l | wc -l)";
+then
+	# No battery exists.
+	exit 0
 fi
 
 echo "Auto packing deferred; not on AC"

main.py

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/usr/bin/env python
 #
 # Copyright (C) 2008 The Android Open Source Project
 #
@@ -14,27 +14,54 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-magic='--calling-python-from-/bin/sh--'
-"""exec" python -E "$0" "$@" """#$magic"
-if __name__ == '__main__':
-  import sys
-  if sys.argv[-1] == '#%s' % magic:
-    del sys.argv[-1]
-del magic
-
+from __future__ import print_function
+import getpass
+import imp
+import netrc
 import optparse
 import os
-import re
 import sys
+import time
 
-from command import InteractiveCommand, PagedCommand
+from pyversion import is_python3
+if is_python3():
+  import urllib.request
+else:
+  import urllib2
+  urllib = imp.new_module('urllib')
+  urllib.request = urllib2
+
+try:
+  import kerberos
+except ImportError:
+  kerberos = None
+
+from color import SetDefaultColoring
+import event_log
+from trace import SetTrace
+from git_command import git, GitCommand
+from git_config import init_ssh, close_ssh
+from command import InteractiveCommand
+from command import MirrorSafeCommand
+from command import GitcAvailableCommand, GitcClientCommand
+from subcmds.version import Version
 from editor import Editor
+from error import DownloadError
+from error import InvalidProjectGroupsError
+from error import ManifestInvalidRevisionError
+from error import ManifestParseError
+from error import NoManifestException
 from error import NoSuchProjectError
 from error import RepoChangedException
-from manifest import Manifest
-from pager import RunPager
+import gitc_utils
+from manifest_xml import GitcManifest, XmlManifest
+from pager import RunPager, TerminatePager
+from wrapper import WrapperPath, Wrapper
 
-from subcmds import all as all_commands
+from subcmds import all_commands
+
+if not is_python3():
+  input = raw_input
 
 global_options = optparse.OptionParser(
                  usage="repo [-p|--paginate|--no-pager] COMMAND [ARGS]"
@@ -45,17 +72,35 @@ global_options.add_option('-p', '--paginate',
 global_options.add_option('--no-pager',
                           dest='no_pager', action='store_true',
                           help='disable the pager')
+global_options.add_option('--color',
+                          choices=('auto', 'always', 'never'), default=None,
+                          help='control color usage: auto, always, never')
+global_options.add_option('--trace',
+                          dest='trace', action='store_true',
+                          help='trace git command execution')
+global_options.add_option('--time',
+                          dest='time', action='store_true',
+                          help='time repo command execution')
+global_options.add_option('--version',
+                          dest='show_version', action='store_true',
+                          help='display this version of repo')
+global_options.add_option('--event-log',
+                          dest='event_log', action='store',
+                          help='filename of event log to append timeline to')
 
 class _Repo(object):
   def __init__(self, repodir):
     self.repodir = repodir
     self.commands = all_commands
+    # add 'branch' as an alias for 'branches'
+    all_commands['branch'] = all_commands['branches']
 
   def _Run(self, argv):
+    result = 0
     name = None
     glob = []
 
-    for i in xrange(0, len(argv)):
+    for i in range(len(argv)):
       if not argv[i].startswith('-'):
         name = argv[i]
         if i > 0:
@@ -66,20 +111,61 @@ class _Repo(object):
       glob = argv
       name = 'help'
       argv = []
-    gopts, gargs = global_options.parse_args(glob)
+    gopts, _gargs = global_options.parse_args(glob)
+
+    if gopts.trace:
+      SetTrace()
+    if gopts.show_version:
+      if name == 'help':
+        name = 'version'
+      else:
+        print('fatal: invalid usage of --version', file=sys.stderr)
+        return 1
+
+    SetDefaultColoring(gopts.color)
 
     try:
       cmd = self.commands[name]
     except KeyError:
-      print >>sys.stderr,\
-            "repo: '%s' is not a repo command.  See 'repo help'."\
-            % name
-      sys.exit(1)
+      print("repo: '%s' is not a repo command.  See 'repo help'." % name,
+            file=sys.stderr)
+      return 1
 
     cmd.repodir = self.repodir
-    cmd.manifest = Manifest(cmd.repodir)
+    cmd.manifest = XmlManifest(cmd.repodir)
+    cmd.gitc_manifest = None
+    gitc_client_name = gitc_utils.parse_clientdir(os.getcwd())
+    if gitc_client_name:
+      cmd.gitc_manifest = GitcManifest(cmd.repodir, gitc_client_name)
+      cmd.manifest.isGitcClient = True
+
     Editor.globalConfig = cmd.manifest.globalConfig
 
+    if not isinstance(cmd, MirrorSafeCommand) and cmd.manifest.IsMirror:
+      print("fatal: '%s' requires a working directory" % name,
+            file=sys.stderr)
+      return 1
+
+    if isinstance(cmd, GitcAvailableCommand) and not gitc_utils.get_gitc_manifest_dir():
+      print("fatal: '%s' requires GITC to be available" % name,
+            file=sys.stderr)
+      return 1
+
+    if isinstance(cmd, GitcClientCommand) and not gitc_client_name:
+      print("fatal: '%s' requires a GITC client" % name,
+            file=sys.stderr)
+      return 1
+
+    try:
+      copts, cargs = cmd.OptionParser.parse_args(argv)
+      copts = cmd.ReadEnvironmentOptions(copts)
+    except NoManifestException as e:
+      print('error: in `%s`: %s' % (' '.join([name] + argv), str(e)),
+        file=sys.stderr)
+      print('error: manifest missing or unreadable -- please run init',
+            file=sys.stderr)
+      return 1
+
     if not gopts.no_pager and not isinstance(cmd, InteractiveCommand):
       config = cmd.manifest.globalConfig
       if gopts.pager:
@@ -87,70 +173,99 @@ class _Repo(object):
       else:
         use_pager = config.GetBoolean('pager.%s' % name)
         if use_pager is None:
-          use_pager = isinstance(cmd, PagedCommand)
+          use_pager = cmd.WantPager(copts)
       if use_pager:
         RunPager(config)
 
-    copts, cargs = cmd.OptionParser.parse_args(argv)
+    start = time.time()
+    cmd_event = cmd.event_log.Add(name, event_log.TASK_COMMAND, start)
+    cmd.event_log.SetParent(cmd_event)
     try:
-      cmd.Execute(copts, cargs)
-    except NoSuchProjectError, e:
+      result = cmd.Execute(copts, cargs)
+    except (DownloadError, ManifestInvalidRevisionError,
+        NoManifestException) as e:
+      print('error: in `%s`: %s' % (' '.join([name] + argv), str(e)),
+        file=sys.stderr)
+      if isinstance(e, NoManifestException):
+        print('error: manifest missing or unreadable -- please run init',
+              file=sys.stderr)
+      result = 1
+    except NoSuchProjectError as e:
       if e.name:
-        print >>sys.stderr, 'error: project %s not found' % e.name
+        print('error: project %s not found' % e.name, file=sys.stderr)
       else:
-        print >>sys.stderr, 'error: no project in current directory'
-      sys.exit(1)
+        print('error: no project in current directory', file=sys.stderr)
+      result = 1
+    except InvalidProjectGroupsError as e:
+      if e.name:
+        print('error: project group must be enabled for project %s' % e.name, file=sys.stderr)
+      else:
+        print('error: project group must be enabled for the project in the current directory', file=sys.stderr)
+      result = 1
+    except SystemExit as e:
+      if e.code:
+        result = e.code
+      raise
+    finally:
+      finish = time.time()
+      elapsed = finish - start
+      hours, remainder = divmod(elapsed, 3600)
+      minutes, seconds = divmod(remainder, 60)
+      if gopts.time:
+        if hours == 0:
+          print('real\t%dm%.3fs' % (minutes, seconds), file=sys.stderr)
+        else:
+          print('real\t%dh%dm%.3fs' % (hours, minutes, seconds),
+                file=sys.stderr)
 
-def _MyWrapperPath():
-  return os.path.join(os.path.dirname(__file__), 'repo')
+      cmd.event_log.FinishEvent(cmd_event, finish,
+                                result is None or result == 0)
+      if gopts.event_log:
+        cmd.event_log.Write(os.path.abspath(
+                            os.path.expanduser(gopts.event_log)))
+
+    return result
+
+
+def _MyRepoPath():
+  return os.path.dirname(__file__)
 
-def _CurrentWrapperVersion():
-  VERSION = None
-  pat = re.compile(r'^VERSION *=')
-  fd = open(_MyWrapperPath())
-  for line in fd:
-    if pat.match(line):
-      fd.close()
-      exec line
-      return VERSION
-  raise NameError, 'No VERSION in repo script'
 
 def _CheckWrapperVersion(ver, repo_path):
   if not repo_path:
     repo_path = '~/bin/repo'
 
   if not ver:
-     print >>sys.stderr, 'no --wrapper-version argument'
-     sys.exit(1)
+    print('no --wrapper-version argument', file=sys.stderr)
+    sys.exit(1)
 
-  exp = _CurrentWrapperVersion()
-  ver = tuple(map(lambda x: int(x), ver.split('.')))
+  exp = Wrapper().VERSION
+  ver = tuple(map(int, ver.split('.')))
   if len(ver) == 1:
     ver = (0, ver[0])
 
+  exp_str = '.'.join(map(str, exp))
   if exp[0] > ver[0] or ver < (0, 4):
-    exp_str = '.'.join(map(lambda x: str(x), exp))
-    print >>sys.stderr, """
+    print("""
 !!! A new repo command (%5s) is available.    !!!
 !!! You must upgrade before you can continue:   !!!
 
     cp %s %s
-""" % (exp_str, _MyWrapperPath(), repo_path)
+""" % (exp_str, WrapperPath(), repo_path), file=sys.stderr)
     sys.exit(1)
 
   if exp > ver:
-    exp_str = '.'.join(map(lambda x: str(x), exp))
-    print >>sys.stderr, """
+    print("""
 ... A new repo command (%5s) is available.
 ... You should upgrade soon:
 
     cp %s %s
-""" % (exp_str, _MyWrapperPath(), repo_path)
+""" % (exp_str, WrapperPath(), repo_path), file=sys.stderr)
 
-def _CheckRepoDir(dir):
-  if not dir:
-     print >>sys.stderr, 'no --repo-dir argument'
-     sys.exit(1)
+def _CheckRepoDir(repo_dir):
+  if not repo_dir:
+    print('no --repo-dir argument', file=sys.stderr)
+    sys.exit(1)
 
 def _PruneOptions(argv, opt):
   i = 0
@@ -167,7 +282,222 @@ def _PruneOptions(argv, opt):
       continue
     i += 1
 
+_user_agent = None
+
+def _UserAgent():
+  global _user_agent
+
+  if _user_agent is None:
+    py_version = sys.version_info
+
+    os_name = sys.platform
+    if os_name == 'linux2':
+      os_name = 'Linux'
+    elif os_name == 'win32':
+      os_name = 'Win32'
+    elif os_name == 'cygwin':
+      os_name = 'Cygwin'
+    elif os_name == 'darwin':
+      os_name = 'Darwin'
+
+    p = GitCommand(
+      None, ['describe', 'HEAD'],
+      cwd = _MyRepoPath(),
+      capture_stdout = True)
+    if p.Wait() == 0:
+      repo_version = p.stdout
+      if len(repo_version) > 0 and repo_version[-1] == '\n':
+        repo_version = repo_version[0:-1]
+      if len(repo_version) > 0 and repo_version[0] == 'v':
+        repo_version = repo_version[1:]
+    else:
+      repo_version = 'unknown'
+
+    _user_agent = 'git-repo/%s (%s) git/%s Python/%d.%d.%d' % (
+      repo_version,
+      os_name,
+      '.'.join(map(str, git.version_tuple())),
+      py_version[0], py_version[1], py_version[2])
+  return _user_agent
+
+class _UserAgentHandler(urllib.request.BaseHandler):
+  def http_request(self, req):
+    req.add_header('User-Agent', _UserAgent())
+    return req
+
+  def https_request(self, req):
+    req.add_header('User-Agent', _UserAgent())
+    return req
+
+def _AddPasswordFromUserInput(handler, msg, req):
+  # If repo could not find auth info from netrc, try to get it from user input
+  url = req.get_full_url()
+  user, password = handler.passwd.find_user_password(None, url)
+  if user is None:
+    print(msg)
+    try:
+      user = input('User: ')
+      password = getpass.getpass()
+    except KeyboardInterrupt:
+      return
+    handler.passwd.add_password(None, url, user, password)
+
+class _BasicAuthHandler(urllib.request.HTTPBasicAuthHandler):
+  def http_error_401(self, req, fp, code, msg, headers):
+    _AddPasswordFromUserInput(self, msg, req)
+    return urllib.request.HTTPBasicAuthHandler.http_error_401(
+      self, req, fp, code, msg, headers)
+
+  def http_error_auth_reqed(self, authreq, host, req, headers):
+    try:
+      old_add_header = req.add_header
+      def _add_header(name, val):
+        val = val.replace('\n', '')
+        old_add_header(name, val)
+      req.add_header = _add_header
+      return urllib.request.AbstractBasicAuthHandler.http_error_auth_reqed(
+        self, authreq, host, req, headers)
+    except:
+      reset = getattr(self, 'reset_retry_count', None)
+      if reset is not None:
+        reset()
+      elif getattr(self, 'retried', None):
+        self.retried = 0
+      raise
+
+class _DigestAuthHandler(urllib.request.HTTPDigestAuthHandler):
+  def http_error_401(self, req, fp, code, msg, headers):
+    _AddPasswordFromUserInput(self, msg, req)
+    return urllib.request.HTTPDigestAuthHandler.http_error_401(
+      self, req, fp, code, msg, headers)
+
+  def http_error_auth_reqed(self, auth_header, host, req, headers):
+    try:
+      old_add_header = req.add_header
+      def _add_header(name, val):
+        val = val.replace('\n', '')
+        old_add_header(name, val)
+      req.add_header = _add_header
+      return urllib.request.AbstractDigestAuthHandler.http_error_auth_reqed(
+        self, auth_header, host, req, headers)
+    except:
+      reset = getattr(self, 'reset_retry_count', None)
+      if reset is not None:
+        reset()
+      elif getattr(self, 'retried', None):
+        self.retried = 0
+      raise
+
+class _KerberosAuthHandler(urllib.request.BaseHandler):
+  def __init__(self):
+    self.retried = 0
+    self.context = None
+    self.handler_order = urllib.request.BaseHandler.handler_order - 50
+
+  def http_error_401(self, req, fp, code, msg, headers):
+    host = req.get_host()
+    retry = self.http_error_auth_reqed('www-authenticate', host, req, headers)
+    return retry
+
+  def http_error_auth_reqed(self, auth_header, host, req, headers):
+    try:
+      spn = "HTTP@%s" % host
+      authdata = self._negotiate_get_authdata(auth_header, headers)
+
+      if self.retried > 3:
+        raise urllib.request.HTTPError(req.get_full_url(), 401,
+          "Negotiate auth failed", headers, None)
+      else:
+        self.retried += 1
+
+      neghdr = self._negotiate_get_svctk(spn, authdata)
+      if neghdr is None:
+        return None
+
+      req.add_unredirected_header('Authorization', neghdr)
+      response = self.parent.open(req)
+
+      srvauth = self._negotiate_get_authdata(auth_header, response.info())
+      if self._validate_response(srvauth):
+        return response
+    except kerberos.GSSError:
+      return None
+    except:
+      self.reset_retry_count()
+      raise
+    finally:
+      self._clean_context()
+
+  def reset_retry_count(self):
+    self.retried = 0
+
+  def _negotiate_get_authdata(self, auth_header, headers):
+    authhdr = headers.get(auth_header, None)
+    if authhdr is not None:
+      for mech_tuple in authhdr.split(","):
+        mech, __, authdata = mech_tuple.strip().partition(" ")
+        if mech.lower() == "negotiate":
+          return authdata.strip()
+    return None
+
+  def _negotiate_get_svctk(self, spn, authdata):
+    if authdata is None:
+      return None
+
+    result, self.context = kerberos.authGSSClientInit(spn)
+    if result < kerberos.AUTH_GSS_COMPLETE:
+      return None
+
+    result = kerberos.authGSSClientStep(self.context, authdata)
+    if result < kerberos.AUTH_GSS_CONTINUE:
+      return None
+
+    response = kerberos.authGSSClientResponse(self.context)
+    return "Negotiate %s" % response
+
+  def _validate_response(self, authdata):
+    if authdata is None:
+      return None
+    result = kerberos.authGSSClientStep(self.context, authdata)
+    if result == kerberos.AUTH_GSS_COMPLETE:
+      return True
+    return None
+
+  def _clean_context(self):
+    if self.context is not None:
+      kerberos.authGSSClientClean(self.context)
+      self.context = None
+
+def init_http():
+  handlers = [_UserAgentHandler()]
+
+  mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
+  try:
+    n = netrc.netrc()
+    for host in n.hosts:
+      p = n.hosts[host]
+      mgr.add_password(p[1], 'http://%s/'  % host, p[0], p[2])
+      mgr.add_password(p[1], 'https://%s/' % host, p[0], p[2])
+  except netrc.NetrcParseError:
+    pass
+  except IOError:
+    pass
+  handlers.append(_BasicAuthHandler(mgr))
+  handlers.append(_DigestAuthHandler(mgr))
+  if kerberos:
+    handlers.append(_KerberosAuthHandler())
+
+  if 'http_proxy' in os.environ:
+    url = os.environ['http_proxy']
+    handlers.append(urllib.request.ProxyHandler({'http': url, 'https': url}))
+  if 'REPO_CURL_VERBOSE' in os.environ:
+    handlers.append(urllib.request.HTTPHandler(debuglevel=1))
+    handlers.append(urllib.request.HTTPSHandler(debuglevel=1))
+  urllib.request.install_opener(urllib.request.build_opener(*handlers))
+
 def _Main(argv):
+  result = 0
+
   opt = optparse.OptionParser(usage="repo wrapperinfo -- ...")
   opt.add_option("--repo-dir", dest="repodir",
                  help="path to .repo/")
@@ -181,22 +511,37 @@ def _Main(argv):
   _CheckWrapperVersion(opt.wrapper_version, opt.wrapper_path)
   _CheckRepoDir(opt.repodir)
 
+  Version.wrapper_version = opt.wrapper_version
+  Version.wrapper_path = opt.wrapper_path
+
   repo = _Repo(opt.repodir)
   try:
-    repo._Run(argv)
+    try:
+      init_ssh()
+      init_http()
+      result = repo._Run(argv) or 0
+    finally:
+      close_ssh()
   except KeyboardInterrupt:
-    sys.exit(1)
-  except RepoChangedException, rce:
+    print('aborted by user', file=sys.stderr)
+    result = 1
+  except ManifestParseError as mpe:
+    print('fatal: %s' % mpe, file=sys.stderr)
+    result = 1
+  except RepoChangedException as rce:
     # If repo changed, re-exec ourselves.
     #
     argv = list(sys.argv)
     argv.extend(rce.extra_args)
     try:
       os.execv(__file__, argv)
-    except OSError, e:
-      print >>sys.stderr, 'fatal: cannot restart repo after upgrade'
-      print >>sys.stderr, 'fatal: %s' % e
-      sys.exit(128)
+    except OSError as e:
+      print('fatal: cannot restart repo after upgrade', file=sys.stderr)
+      print('fatal: %s' % e, file=sys.stderr)
+      result = 128
+
+  TerminatePager()
+  sys.exit(result)
 
 if __name__ == '__main__':
   _Main(sys.argv[1:])

manifest.py

@@ -1,340 +0,0 @@
-#
-# Copyright (C) 2008 The Android Open Source Project
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import os
-import sys
-import xml.dom.minidom
-
-from git_config import GitConfig, IsId
-from project import Project, MetaProject, R_HEADS
-from remote import Remote
-from error import ManifestParseError
-
-MANIFEST_FILE_NAME = 'manifest.xml'
-LOCAL_MANIFEST_NAME = 'local_manifest.xml'
-
-class _Default(object):
-  """Project defaults within the manifest."""
-
-  revision = None
-  remote = None
-
-
-class Manifest(object):
-  """manages the repo configuration file"""
-
-  def __init__(self, repodir):
-    self.repodir = os.path.abspath(repodir)
-    self.topdir = os.path.dirname(self.repodir)
-    self.manifestFile = os.path.join(self.repodir, MANIFEST_FILE_NAME)
-    self.globalConfig = GitConfig.ForUser()
-
-    self.repoProject = MetaProject(self, 'repo',
-      gitdir   = os.path.join(repodir, 'repo/.git'),
-      worktree = os.path.join(repodir, 'repo'))
-
-    self.manifestProject = MetaProject(self, 'manifests',
-      gitdir   = os.path.join(repodir, 'manifests.git'),
-      worktree = os.path.join(repodir, 'manifests'))
-
-    self._Unload()
-
-  def Link(self, name):
-    """Update the repo metadata to use a different manifest.
-    """
-    path = os.path.join(self.manifestProject.worktree, name)
-    if not os.path.isfile(path):
-      raise ManifestParseError('manifest %s not found' % name)
-
-    old = self.manifestFile
-    try:
-      self.manifestFile = path
-      self._Unload()
-      self._Load()
-    finally:
-      self.manifestFile = old
-
-    try:
-      if os.path.exists(self.manifestFile):
-        os.remove(self.manifestFile)
-      os.symlink('manifests/%s' % name, self.manifestFile)
-    except OSError, e:
-      raise ManifestParseError('cannot link manifest %s' % name)
-
-  @property
-  def projects(self):
-    self._Load()
-    return self._projects
-
-  @property
-  def remotes(self):
-    self._Load()
-    return self._remotes
-
-  @property
-  def default(self):
-    self._Load()
-    return self._default
-
-  @property
-  def IsMirror(self):
-    return self.manifestProject.config.GetBoolean('repo.mirror')
-
-  def _Unload(self):
-    self._loaded = False
-    self._projects = {}
-    self._remotes = {}
-    self._default = None
-    self.branch = None
-
-  def _Load(self):
-    if not self._loaded:
-      m = self.manifestProject
-      b = m.GetBranch(m.CurrentBranch).merge
-      if b.startswith(R_HEADS):
-        b = b[len(R_HEADS):]
-      self.branch = b
-
-      self._ParseManifest(True)
-
-      local = os.path.join(self.repodir, LOCAL_MANIFEST_NAME)
-      if os.path.exists(local):
-        try:
-          real = self.manifestFile
-          self.manifestFile = local
-          self._ParseManifest(False)
-        finally:
-          self.manifestFile = real
-
-      if self.IsMirror:
-        self._AddMetaProjectMirror(self.repoProject)
-        self._AddMetaProjectMirror(self.manifestProject)
-
-      self._loaded = True
-
-  def _ParseManifest(self, is_root_file):
-    root = xml.dom.minidom.parse(self.manifestFile)
-    if not root or not root.childNodes:
-      raise ManifestParseError, \
-            "no root node in %s" % \
-            self.manifestFile
-
-    config = root.childNodes[0]
-    if config.nodeName != 'manifest':
-      raise ManifestParseError, \
-            "no <manifest> in %s" % \
-            self.manifestFile
-
-    for node in config.childNodes:
-      if node.nodeName == 'remote':
-        remote = self._ParseRemote(node)
-        if self._remotes.get(remote.name):
-          raise ManifestParseError, \
-                'duplicate remote %s in %s' % \
-                (remote.name, self.manifestFile)
-        self._remotes[remote.name] = remote
-
-    for node in config.childNodes:
-      if node.nodeName == 'default':
-        if self._default is not None:
-          raise ManifestParseError, \
-                'duplicate default in %s' % \
-                (self.manifestFile)
-        self._default = self._ParseDefault(node)
-    if self._default is None:
-      self._default = _Default()
-
-    for node in config.childNodes:
-      if node.nodeName == 'project':
-        project = self._ParseProject(node)
-        if self._projects.get(project.name):
-          raise ManifestParseError, \
-                'duplicate project %s in %s' % \
-                (project.name, self.manifestFile)
-        self._projects[project.name] = project
-
-    for node in config.childNodes:
-      if node.nodeName == 'add-remote':
-        pn = self._reqatt(node, 'to-project')
-        project = self._projects.get(pn)
-        if not project:
-          raise ManifestParseError, \
-                'project %s not defined in %s' % \
-                (pn, self.manifestFile)
-        self._ParseProjectExtraRemote(project, node)
-
-  def _AddMetaProjectMirror(self, m):
-    name = None
-    m_url = m.GetRemote(m.remote.name).url
-    if m_url.endswith('/.git'):
-      raise ManifestParseError, 'refusing to mirror %s' % m_url
-
-    if self._default and self._default.remote:
-      url = self._default.remote.fetchUrl
-      if not url.endswith('/'):
-        url += '/'
-      if m_url.startswith(url):
-        remote = self._default.remote
-        name = m_url[len(url):]
-
-    if name is None:
-      s = m_url.rindex('/') + 1
-      remote = Remote('origin', fetch = m_url[:s])
-      name = m_url[s:]
-
-    if name.endswith('.git'):
-      name = name[:-4]
-
-    if name not in self._projects:
-      m.PreSync()
-      gitdir = os.path.join(self.topdir, '%s.git' % name)
-      project = Project(manifest = self,
-                        name = name,
-                        remote = remote,
-                        gitdir = gitdir,
-                        worktree = None,
-                        relpath = None,
-                        revision = m.revision)
-      self._projects[project.name] = project
-
-  def _ParseRemote(self, node):
-    """
-    reads a <remote> element from the manifest file
-    """
-    name = self._reqatt(node, 'name')
-    fetch = self._reqatt(node, 'fetch')
-    review = node.getAttribute('review')
-    if review == '':
-      review = None
-
-    projectName = node.getAttribute('project-name')
-    if projectName == '':
-      projectName = None
-
-    r = Remote(name=name,
-               fetch=fetch,
-               review=review,
-               projectName=projectName)
-
-    for n in node.childNodes:
-      if n.nodeName == 'require':
-        r.requiredCommits.append(self._reqatt(n, 'commit'))
-
-    return r
-
-  def _ParseDefault(self, node):
-    """
-    reads a <default> element from the manifest file
-    """
-    d = _Default()
-    d.remote = self._get_remote(node)
-    d.revision = node.getAttribute('revision')
-    if d.revision == '':
-      d.revision = None
-    return d
-
-  def _ParseProject(self, node):
-    """
-    reads a <project> element from the manifest file
-    """ 
-    name = self._reqatt(node, 'name')
-
-    remote = self._get_remote(node)
-    if remote is None:
-      remote = self._default.remote
-    if remote is None:
-      raise ManifestParseError, \
-            "no remote for project %s within %s" % \
-            (name, self.manifestFile)
-
-    revision = node.getAttribute('revision')
-    if not revision:
-      revision = self._default.revision
-    if not revision:
-      raise ManifestParseError, \
-            "no revision for project %s within %s" % \
-            (name, self.manifestFile)
-
-    path = node.getAttribute('path')
-    if not path:
-      path = name
-    if path.startswith('/'):
-      raise ManifestParseError, \
-            "project %s path cannot be absolute in %s" % \
-            (name, self.manifestFile)
-
-    if self.IsMirror:
-      relpath = None
-      worktree = None
-      gitdir = os.path.join(self.topdir, '%s.git' % name)
-    else:
-      worktree = os.path.join(self.topdir, path)
-      gitdir = os.path.join(self.repodir, 'projects/%s.git' % path)
-
-    project = Project(manifest = self,
-                      name = name,
-                      remote = remote,
-                      gitdir = gitdir,
-                      worktree = worktree,
-                      relpath = path,
-                      revision = revision)
-
-    for n in node.childNodes:
-      if n.nodeName == 'remote':
-        self._ParseProjectExtraRemote(project, n)
-      elif n.nodeName == 'copyfile':
-        self._ParseCopyFile(project, n)
-
-    return project
-
-  def _ParseProjectExtraRemote(self, project, n):
-    r = self._ParseRemote(n)
-    if project.extraRemotes.get(r.name) \
-       or project.remote.name == r.name:
-      raise ManifestParseError, \
-            'duplicate remote %s in project %s in %s' % \
-            (r.name, project.name, self.manifestFile)
-    project.extraRemotes[r.name] = r
-
-  def _ParseCopyFile(self, project, node):
-    src = self._reqatt(node, 'src')
-    dest = self._reqatt(node, 'dest')
-    if not self.IsMirror:
-      # src is project relative;
-      # dest is relative to the top of the tree
-      project.AddCopyFile(src, os.path.join(self.topdir, dest))
-
-  def _get_remote(self, node):
-    name = node.getAttribute('remote')
-    if not name:
-      return None
-
-    v = self._remotes.get(name)
-    if not v:
-      raise ManifestParseError, \
-            "remote %s not defined in %s" % \
-            (name, self.manifestFile)
-    return v
-
-  def _reqatt(self, node, attname):
-    """
-    reads a required attribute from the node.
-    """
-    v = node.getAttribute(attname)
-    if not v:
-      raise ManifestParseError, \
-            "no %s in <%s> within %s" % \
-            (attname, node.nodeName, self.manifestFile)
-    return v

pager.py

@@ -13,21 +13,56 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
 import os
 import select
+import subprocess
 import sys
 
+import platform_utils
+
 active = False
+pager_process = None
+old_stdout = None
+old_stderr = None
 
 def RunPager(globalConfig):
-  global active
-
-  if not os.isatty(0):
+  if not os.isatty(0) or not os.isatty(1):
     return
   pager = _SelectPager(globalConfig)
   if pager == '' or pager == 'cat':
     return
 
+  if platform_utils.isWindows():
+    _PipePager(pager);
+  else:
+    _ForkPager(pager)
+
+def TerminatePager():
+  global pager_process, old_stdout, old_stderr
+  if pager_process:
+    sys.stdout.flush()
+    sys.stderr.flush()
+    pager_process.stdin.close()
+    pager_process.wait();
+    pager_process = None
+    # Restore initial stdout/err in case there is more output in this process
+    # after shutting down the pager process
+    sys.stdout = old_stdout
+    sys.stderr = old_stderr
+
+def _PipePager(pager):
+  global pager_process, old_stdout, old_stderr
+  assert pager_process is None, "Only one active pager process at a time"
+  # Create pager process, piping stdout/err into its stdin
+  pager_process = subprocess.Popen([pager], stdin=subprocess.PIPE, stdout=sys.stdout, stderr=sys.stderr)
+  old_stdout = sys.stdout
+  old_stderr = sys.stderr
+  sys.stdout = pager_process.stdin
+  sys.stderr = pager_process.stdin
+
+def _ForkPager(pager):
+  global active
   # This process turns into the pager; a child it forks will
   # do the real processing and output back to the pager. This
   # is necessary to keep the pager in control of the tty.
@@ -49,8 +84,8 @@ def RunPager(globalConfig):
 
     _BecomePager(pager)
   except Exception:
-    print >>sys.stderr, "fatal: cannot start pager '%s'" % pager
-    os.exit(255)
+    print("fatal: cannot start pager '%s'" % pager, file=sys.stderr)
+    sys.exit(255)
 
 def _SelectPager(globalConfig):
   try:
@@ -74,11 +109,11 @@ def _BecomePager(pager):
   # ready works around a long-standing bug in popularly
   # available versions of 'less', a better 'more'.
   #
-  a, b, c = select.select([0], [], [0])
+  _a, _b, _c = select.select([0], [], [0])
 
   os.environ['LESS'] = 'FRSX'
 
   try:
     os.execvp(pager, [pager])
-  except OSError, e:
+  except OSError:
     os.execv('/bin/sh', ['sh', '-c', pager])

platform_utils.py

@@ -0,0 +1,414 @@
+#
+# Copyright (C) 2016 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import errno
+import os
+import platform
+import select
+import shutil
+import stat
+
+from pyversion import is_python3
+if is_python3():
+  from queue import Queue
+else:
+  from Queue import Queue
+
+from threading import Thread
+
+
+def isWindows():
+  """ Returns True when running with the native port of Python for Windows,
+  False when running on any other platform (including the Cygwin port of
+  Python).
+  """
+  # Note: The cygwin port of Python returns "CYGWIN_NT_xxx"
+  return platform.system() == "Windows"
+
+
+class FileDescriptorStreams(object):
+  """ Platform agnostic abstraction enabling non-blocking I/O over a
+  collection of file descriptors. This abstraction is required because
+  fctnl(os.O_NONBLOCK) is not supported on Windows.
+  """
+  @classmethod
+  def create(cls):
+    """ Factory method: instantiates the concrete class according to the
+    current platform.
+    """
+    if isWindows():
+      return _FileDescriptorStreamsThreads()
+    else:
+      return _FileDescriptorStreamsNonBlocking()
+
+  def __init__(self):
+    self.streams = []
+
+  def add(self, fd, dest, std_name):
+    """ Wraps an existing file descriptor as a stream.
+    """
+    self.streams.append(self._create_stream(fd, dest, std_name))
+
+  def remove(self, stream):
+    """ Removes a stream, when done with it.
+    """
+    self.streams.remove(stream)
+
+  @property
+  def is_done(self):
+    """ Returns True when all streams have been processed.
+    """
+    return len(self.streams) == 0
+
+  def select(self):
+    """ Returns the set of streams that have data available to read.
+    The returned streams each expose a read() and a close() method.
+    When done with a stream, call the remove(stream) method.
+    """
+    raise NotImplementedError
+
+  def _create_stream(fd, dest, std_name):
+    """ Creates a new stream wrapping an existing file descriptor.
+    """
+    raise NotImplementedError
+
+
+class _FileDescriptorStreamsNonBlocking(FileDescriptorStreams):
+  """ Implementation of FileDescriptorStreams for platforms that support
+  non blocking I/O.
+  """
+  class Stream(object):
+    """ Encapsulates a file descriptor """
+    def __init__(self, fd, dest, std_name):
+      self.fd = fd
+      self.dest = dest
+      self.std_name = std_name
+      self.set_non_blocking()
+
+    def set_non_blocking(self):
+      import fcntl
+      flags = fcntl.fcntl(self.fd, fcntl.F_GETFL)
+      fcntl.fcntl(self.fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
+
+    def fileno(self):
+      return self.fd.fileno()
+
+    def read(self):
+      return self.fd.read(4096)
+
+    def close(self):
+      self.fd.close()
+
+  def _create_stream(self, fd, dest, std_name):
+    return self.Stream(fd, dest, std_name)
+
+  def select(self):
+    ready_streams, _, _ = select.select(self.streams, [], [])
+    return ready_streams
+
+
+class _FileDescriptorStreamsThreads(FileDescriptorStreams):
+  """ Implementation of FileDescriptorStreams for platforms that don't support
+  non blocking I/O. This implementation requires creating threads issuing
+  blocking read operations on file descriptors.
+  """
+  def __init__(self):
+    super(_FileDescriptorStreamsThreads, self).__init__()
+    # The queue is shared accross all threads so we can simulate the
+    # behavior of the select() function
+    self.queue = Queue(10)  # Limit incoming data from streams
+
+  def _create_stream(self, fd, dest, std_name):
+    return self.Stream(fd, dest, std_name, self.queue)
+
+  def select(self):
+    # Return only one stream at a time, as it is the most straighforward
+    # thing to do and it is compatible with the select() function.
+    item = self.queue.get()
+    stream = item.stream
+    stream.data = item.data
+    return [stream]
+
+  class QueueItem(object):
+    """ Item put in the shared queue """
+    def __init__(self, stream, data):
+      self.stream = stream
+      self.data = data
+
+  class Stream(object):
+    """ Encapsulates a file descriptor """
+    def __init__(self, fd, dest, std_name, queue):
+      self.fd = fd
+      self.dest = dest
+      self.std_name = std_name
+      self.queue = queue
+      self.data = None
+      self.thread = Thread(target=self.read_to_queue)
+      self.thread.daemon = True
+      self.thread.start()
+
+    def close(self):
+      self.fd.close()
+
+    def read(self):
+      data = self.data
+      self.data = None
+      return data
+
+    def read_to_queue(self):
+      """ The thread function: reads everything from the file descriptor into
+      the shared queue and terminates when reaching EOF.
+      """
+      for line in iter(self.fd.readline, b''):
+        self.queue.put(_FileDescriptorStreamsThreads.QueueItem(self, line))
+      self.fd.close()
+      self.queue.put(_FileDescriptorStreamsThreads.QueueItem(self, None))
+
+
+def symlink(source, link_name):
+  """Creates a symbolic link pointing to source named link_name.
+  Note: On Windows, source must exist on disk, as the implementation needs
+  to know whether to create a "File" or a "Directory" symbolic link.
+  """
+  if isWindows():
+    import platform_utils_win32
+    source = _validate_winpath(source)
+    link_name = _validate_winpath(link_name)
+    target = os.path.join(os.path.dirname(link_name), source)
+    if isdir(target):
+      platform_utils_win32.create_dirsymlink(_makelongpath(source), link_name)
+    else:
+      platform_utils_win32.create_filesymlink(_makelongpath(source), link_name)
+  else:
+    return os.symlink(source, link_name)
+
+
+def _validate_winpath(path):
+  path = os.path.normpath(path)
+  if _winpath_is_valid(path):
+    return path
+  raise ValueError("Path \"%s\" must be a relative path or an absolute "
+                   "path starting with a drive letter".format(path))
+
+
+def _winpath_is_valid(path):
+  """Windows only: returns True if path is relative (e.g. ".\\foo") or is
+  absolute including a drive letter (e.g. "c:\\foo"). Returns False if path
+  is ambiguous (e.g. "x:foo" or "\\foo").
+  """
+  assert isWindows()
+  path = os.path.normpath(path)
+  drive, tail = os.path.splitdrive(path)
+  if tail:
+    if not drive:
+      return tail[0] != os.sep  # "\\foo" is invalid
+    else:
+      return tail[0] == os.sep  # "x:foo" is invalid
+  else:
+    return not drive  # "x:" is invalid
+
+
+def _makelongpath(path):
+  """Return the input path normalized to support the Windows long path syntax
+  ("\\\\?\\" prefix) if needed, i.e. if the input path is longer than the
+  MAX_PATH limit.
+  """
+  if isWindows():
+    # Note: MAX_PATH is 260, but, for directories, the maximum value is actually 246.
+    if len(path) < 246:
+      return path
+    if path.startswith(u"\\\\?\\"):
+      return path
+    if not os.path.isabs(path):
+      return path
+    # Append prefix and ensure unicode so that the special longpath syntax
+    # is supported by underlying Win32 API calls
+    return u"\\\\?\\" + os.path.normpath(path)
+  else:
+    return path
+
+
+def rmtree(path):
+  """shutil.rmtree(path) wrapper with support for long paths on Windows.
+
+  Availability: Unix, Windows."""
+  if isWindows():
+    shutil.rmtree(_makelongpath(path), onerror=handle_rmtree_error)
+  else:
+    shutil.rmtree(path)
+
+
+def handle_rmtree_error(function, path, excinfo):
+  # Allow deleting read-only files
+  os.chmod(path, stat.S_IWRITE)
+  function(path)
+
+
+def rename(src, dst):
+  """os.rename(src, dst) wrapper with support for long paths on Windows.
+
+  Availability: Unix, Windows."""
+  if isWindows():
+    # On Windows, rename fails if destination exists, see
+    # https://docs.python.org/2/library/os.html#os.rename
+    try:
+      os.rename(_makelongpath(src), _makelongpath(dst))
+    except OSError as e:
+      if e.errno == errno.EEXIST:
+        os.remove(_makelongpath(dst))
+        os.rename(_makelongpath(src), _makelongpath(dst))
+      else:
+        raise
+  else:
+    os.rename(src, dst)
+
+
+def remove(path):
+  """Remove (delete) the file path. This is a replacement for os.remove that
+  allows deleting read-only files on Windows, with support for long paths and
+  for deleting directory symbolic links.
+
+  Availability: Unix, Windows."""
+  if isWindows():
+    longpath = _makelongpath(path)
+    try:
+      os.remove(longpath)
+    except OSError as e:
+      if e.errno == errno.EACCES:
+        os.chmod(longpath, stat.S_IWRITE)
+        # Directory symbolic links must be deleted with 'rmdir'.
+        if islink(longpath) and isdir(longpath):
+          os.rmdir(longpath)
+        else:
+          os.remove(longpath)
+      else:
+        raise
+  else:
+    os.remove(path)
+
+
+def walk(top, topdown=True, onerror=None, followlinks=False):
+  """os.walk(path) wrapper with support for long paths on Windows.
+
+  Availability: Windows, Unix.
+  """
+  if isWindows():
+    return _walk_windows_impl(top, topdown, onerror, followlinks)
+  else:
+    return os.walk(top, topdown, onerror, followlinks)
+
+
+def _walk_windows_impl(top, topdown, onerror, followlinks):
+  try:
+    names = listdir(top)
+  except Exception as err:
+    if onerror is not None:
+      onerror(err)
+    return
+
+  dirs, nondirs = [], []
+  for name in names:
+    if isdir(os.path.join(top, name)):
+      dirs.append(name)
+    else:
+      nondirs.append(name)
+
+  if topdown:
+    yield top, dirs, nondirs
+  for name in dirs:
+    new_path = os.path.join(top, name)
+    if followlinks or not islink(new_path):
+      for x in _walk_windows_impl(new_path, topdown, onerror, followlinks):
+        yield x
+  if not topdown:
+    yield top, dirs, nondirs
+
+
+def listdir(path):
+  """os.listdir(path) wrapper with support for long paths on Windows.
+
+  Availability: Windows, Unix.
+  """
+  return os.listdir(_makelongpath(path))
+
+
+def rmdir(path):
+  """os.rmdir(path) wrapper with support for long paths on Windows.
+
+  Availability: Windows, Unix.
+  """
+  os.rmdir(_makelongpath(path))
+
+
+def isdir(path):
+  """os.path.isdir(path) wrapper with support for long paths on Windows.
+
+  Availability: Windows, Unix.
+  """
+  return os.path.isdir(_makelongpath(path))
+
+
+def islink(path):
+  """os.path.islink(path) wrapper with support for long paths on Windows.
+
+  Availability: Windows, Unix.
+  """
+  if isWindows():
+    import platform_utils_win32
+    return platform_utils_win32.islink(_makelongpath(path))
+  else:
+    return os.path.islink(path)
+
+
+def readlink(path):
+  """Return a string representing the path to which the symbolic link
+  points. The result may be either an absolute or relative pathname;
+  if it is relative, it may be converted to an absolute pathname using
+  os.path.join(os.path.dirname(path), result).
+
+  Availability: Windows, Unix.
+  """
+  if isWindows():
+    import platform_utils_win32
+    return platform_utils_win32.readlink(_makelongpath(path))
+  else:
+    return os.readlink(path)
+
+
+def realpath(path):
+  """Return the canonical path of the specified filename, eliminating
+  any symbolic links encountered in the path.
+
+  Availability: Windows, Unix.
+  """
+  if isWindows():
+    current_path = os.path.abspath(path)
+    path_tail = []
+    for c in range(0, 100):  # Avoid cycles
+      if islink(current_path):
+        target = readlink(current_path)
+        current_path = os.path.join(os.path.dirname(current_path), target)
+      else:
+        basename = os.path.basename(current_path)
+        if basename == '':
+          path_tail.append(current_path)
+          break
+        path_tail.append(basename)
+        current_path = os.path.dirname(current_path)
+    path_tail.reverse()
+    result = os.path.normpath(os.path.join(*path_tail))
+    return result
+  else:
+    return os.path.realpath(path)

platform_utils_win32.py

@@ -0,0 +1,225 @@
+#
+# Copyright (C) 2016 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import errno
+
+from ctypes import WinDLL, get_last_error, FormatError, WinError, addressof
+from ctypes import c_buffer
+from ctypes.wintypes import BOOL, LPCWSTR, DWORD, HANDLE, POINTER, c_ubyte
+from ctypes.wintypes import WCHAR, USHORT, LPVOID, Structure, Union, ULONG
+from ctypes.wintypes import byref
+
+kernel32 = WinDLL('kernel32', use_last_error=True)
+
+LPDWORD = POINTER(DWORD)
+UCHAR = c_ubyte
+
+# Win32 error codes
+ERROR_SUCCESS = 0
+ERROR_NOT_SUPPORTED = 50
+ERROR_PRIVILEGE_NOT_HELD = 1314
+
+# Win32 API entry points
+CreateSymbolicLinkW = kernel32.CreateSymbolicLinkW
+CreateSymbolicLinkW.restype = BOOL
+CreateSymbolicLinkW.argtypes = (LPCWSTR,  # lpSymlinkFileName In
+                                LPCWSTR,  # lpTargetFileName In
+                                DWORD)    # dwFlags In
+
+# Symbolic link creation flags
+SYMBOLIC_LINK_FLAG_FILE = 0x00
+SYMBOLIC_LINK_FLAG_DIRECTORY = 0x01
+# symlink support for CreateSymbolicLink() starting with Windows 10 (1703, v10.0.14972)
+SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE = 0x02
+
+GetFileAttributesW = kernel32.GetFileAttributesW
+GetFileAttributesW.restype = DWORD
+GetFileAttributesW.argtypes = (LPCWSTR,)  # lpFileName In
+
+INVALID_FILE_ATTRIBUTES = 0xFFFFFFFF
+FILE_ATTRIBUTE_REPARSE_POINT = 0x00400
+
+CreateFileW = kernel32.CreateFileW
+CreateFileW.restype = HANDLE
+CreateFileW.argtypes = (LPCWSTR,  # lpFileName In
+                        DWORD,    # dwDesiredAccess In
+                        DWORD,    # dwShareMode In
+                        LPVOID,   # lpSecurityAttributes In_opt
+                        DWORD,    # dwCreationDisposition In
+                        DWORD,    # dwFlagsAndAttributes In
+                        HANDLE)   # hTemplateFile In_opt
+
+CloseHandle = kernel32.CloseHandle
+CloseHandle.restype = BOOL
+CloseHandle.argtypes = (HANDLE,)  # hObject In
+
+INVALID_HANDLE_VALUE = HANDLE(-1).value
+OPEN_EXISTING = 3
+FILE_FLAG_BACKUP_SEMANTICS = 0x02000000
+FILE_FLAG_OPEN_REPARSE_POINT = 0x00200000
+
+DeviceIoControl = kernel32.DeviceIoControl
+DeviceIoControl.restype = BOOL
+DeviceIoControl.argtypes = (HANDLE,   # hDevice In
+                            DWORD,    # dwIoControlCode In
+                            LPVOID,   # lpInBuffer In_opt
+                            DWORD,    # nInBufferSize In
+                            LPVOID,   # lpOutBuffer Out_opt
+                            DWORD,    # nOutBufferSize In
+                            LPDWORD,  # lpBytesReturned Out_opt
+                            LPVOID)   # lpOverlapped Inout_opt
+
+# Device I/O control flags and options
+FSCTL_GET_REPARSE_POINT = 0x000900A8
+IO_REPARSE_TAG_MOUNT_POINT = 0xA0000003
+IO_REPARSE_TAG_SYMLINK = 0xA000000C
+MAXIMUM_REPARSE_DATA_BUFFER_SIZE = 0x4000
+
+
+class GENERIC_REPARSE_BUFFER(Structure):
+  _fields_ = (('DataBuffer', UCHAR * 1),)
+
+
+class SYMBOLIC_LINK_REPARSE_BUFFER(Structure):
+  _fields_ = (('SubstituteNameOffset', USHORT),
+              ('SubstituteNameLength', USHORT),
+              ('PrintNameOffset', USHORT),
+              ('PrintNameLength', USHORT),
+              ('Flags', ULONG),
+              ('PathBuffer', WCHAR * 1))
+
+  @property
+  def PrintName(self):
+    arrayt = WCHAR * (self.PrintNameLength // 2)
+    offset = type(self).PathBuffer.offset + self.PrintNameOffset
+    return arrayt.from_address(addressof(self) + offset).value
+
+
+class MOUNT_POINT_REPARSE_BUFFER(Structure):
+  _fields_ = (('SubstituteNameOffset', USHORT),
+              ('SubstituteNameLength', USHORT),
+              ('PrintNameOffset', USHORT),
+              ('PrintNameLength', USHORT),
+              ('PathBuffer', WCHAR * 1))
+
+  @property
+  def PrintName(self):
+    arrayt = WCHAR * (self.PrintNameLength // 2)
+    offset = type(self).PathBuffer.offset + self.PrintNameOffset
+    return arrayt.from_address(addressof(self) + offset).value
+
+
+class REPARSE_DATA_BUFFER(Structure):
+  class REPARSE_BUFFER(Union):
+    _fields_ = (('SymbolicLinkReparseBuffer', SYMBOLIC_LINK_REPARSE_BUFFER),
+                ('MountPointReparseBuffer', MOUNT_POINT_REPARSE_BUFFER),
+                ('GenericReparseBuffer', GENERIC_REPARSE_BUFFER))
+  _fields_ = (('ReparseTag', ULONG),
+              ('ReparseDataLength', USHORT),
+              ('Reserved', USHORT),
+              ('ReparseBuffer', REPARSE_BUFFER))
+  _anonymous_ = ('ReparseBuffer',)
+
+
+def create_filesymlink(source, link_name):
+  """Creates a Windows file symbolic link source pointing to link_name."""
+  _create_symlink(source, link_name, SYMBOLIC_LINK_FLAG_FILE)
+
+
+def create_dirsymlink(source, link_name):
+  """Creates a Windows directory symbolic link source pointing to link_name.
+  """
+  _create_symlink(source, link_name, SYMBOLIC_LINK_FLAG_DIRECTORY)
+
+
+def _create_symlink(source, link_name, dwFlags):
+  # Note: Win32 documentation for CreateSymbolicLink is incorrect.
+  # On success, the function returns "1".
+  # On error, the function returns some random value (e.g. 1280).
+  # The best bet seems to use "GetLastError" and check for error/success.
+  CreateSymbolicLinkW(link_name, source, dwFlags | SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE)
+  code = get_last_error()
+  if code != ERROR_SUCCESS:
+    # See https://github.com/golang/go/pull/24307/files#diff-b87bc12e4da2497308f9ef746086e4f0
+    # "the unprivileged create flag is unsupported below Windows 10 (1703, v10.0.14972).
+    # retry without it."
+    CreateSymbolicLinkW(link_name, source, dwFlags)
+    code = get_last_error()
+    if code != ERROR_SUCCESS:
+      error_desc = FormatError(code).strip()
+      if code == ERROR_PRIVILEGE_NOT_HELD:
+        raise OSError(errno.EPERM, error_desc, link_name)
+      _raise_winerror(
+          code,
+          'Error creating symbolic link \"%s\"'.format(link_name))
+
+
+def islink(path):
+  result = GetFileAttributesW(path)
+  if result == INVALID_FILE_ATTRIBUTES:
+    return False
+  return bool(result & FILE_ATTRIBUTE_REPARSE_POINT)
+
+
+def readlink(path):
+  reparse_point_handle = CreateFileW(path,
+                                     0,
+                                     0,
+                                     None,
+                                     OPEN_EXISTING,
+                                     FILE_FLAG_OPEN_REPARSE_POINT |
+                                     FILE_FLAG_BACKUP_SEMANTICS,
+                                     None)
+  if reparse_point_handle == INVALID_HANDLE_VALUE:
+    _raise_winerror(
+        get_last_error(),
+        'Error opening symblic link \"%s\"'.format(path))
+  target_buffer = c_buffer(MAXIMUM_REPARSE_DATA_BUFFER_SIZE)
+  n_bytes_returned = DWORD()
+  io_result = DeviceIoControl(reparse_point_handle,
+                              FSCTL_GET_REPARSE_POINT,
+                              None,
+                              0,
+                              target_buffer,
+                              len(target_buffer),
+                              byref(n_bytes_returned),
+                              None)
+  CloseHandle(reparse_point_handle)
+  if not io_result:
+    _raise_winerror(
+        get_last_error(),
+        'Error reading symblic link \"%s\"'.format(path))
+  rdb = REPARSE_DATA_BUFFER.from_buffer(target_buffer)
+  if rdb.ReparseTag == IO_REPARSE_TAG_SYMLINK:
+    return _preserve_encoding(path, rdb.SymbolicLinkReparseBuffer.PrintName)
+  elif rdb.ReparseTag == IO_REPARSE_TAG_MOUNT_POINT:
+    return _preserve_encoding(path, rdb.MountPointReparseBuffer.PrintName)
+  # Unsupported reparse point type
+  _raise_winerror(
+      ERROR_NOT_SUPPORTED,
+      'Error reading symblic link \"%s\"'.format(path))
+
+
+def _preserve_encoding(source, target):
+  """Ensures target is the same string type (i.e. unicode or str) as source."""
+  if isinstance(source, unicode):
+    return unicode(target)
+  return str(target)
+
+
+def _raise_winerror(code, error_desc):
+  win_error_desc = FormatError(code).strip()
+  error_desc = "%s: %s".format(error_desc, win_error_desc)
+  raise WinError(code, error_desc)

progress.py

@@ -0,0 +1,82 @@
+#
+# Copyright (C) 2009 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+from time import time
+from trace import IsTrace
+
+_NOT_TTY = not os.isatty(2)
+
+class Progress(object):
+  def __init__(self, title, total=0, units='', print_newline=False,
+               always_print_percentage=False):
+    self._title = title
+    self._total = total
+    self._done = 0
+    self._lastp = -1
+    self._start = time()
+    self._show = False
+    self._units = units
+    self._print_newline = print_newline
+    self._always_print_percentage = always_print_percentage
+
+  def update(self, inc=1):
+    self._done += inc
+
+    if _NOT_TTY or IsTrace():
+      return
+
+    if not self._show:
+      if 0.5 <= time() - self._start:
+        self._show = True
+      else:
+        return
+
+    if self._total <= 0:
+      sys.stderr.write('\r%s: %d, ' % (
+        self._title,
+        self._done))
+      sys.stderr.flush()
+    else:
+      p = (100 * self._done) / self._total
+
+      if self._lastp != p or self._always_print_percentage:
+        self._lastp = p
+        sys.stderr.write('\r%s: %3d%% (%d%s/%d%s)%s' % (
+          self._title,
+          p,
+          self._done, self._units,
+          self._total, self._units,
+          "\n" if self._print_newline else ""))
+        sys.stderr.flush()
+
+  def end(self):
+    if _NOT_TTY or IsTrace() or not self._show:
+      return
+
+    if self._total <= 0:
+      sys.stderr.write('\r%s: %d, done.  \n' % (
+        self._title,
+        self._done))
+      sys.stderr.flush()
+    else:
+      p = (100 * self._done) / self._total
+      sys.stderr.write('\r%s: %3d%% (%d%s/%d%s), done.  \n' % (
+        self._title,
+        p,
+        self._done, self._units,
+        self._total, self._units))
+      sys.stderr.flush()

pyversion.py

@@ -1,11 +1,11 @@
 #
-# Copyright 2008 Google Inc.
+# Copyright (C) 2013 The Android Open Source Project
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
-# http://www.apache.org/licenses/LICENSE-2.0
+#      http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
@@ -13,17 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-GERRIT_SRC=../gerrit
-GERRIT_MODULES=codereview froofle
+import sys
 
-all:
-
-clean:
-	find . -name \*.pyc -type f | xargs rm -f
-
-update-pyclient:
-	$(MAKE) -C $(GERRIT_SRC) release-pyclient
-	rm -rf $(GERRIT_MODULES)
-	(cd $(GERRIT_SRC)/release/pyclient && \
-	 find . -type f \
-	 | cpio -pd $(abspath .))
+def is_python3():
+  return sys.version_info[0] == 3

repo

@@ -1,9 +1,12 @@
-#!/bin/sh
+#!/usr/bin/env python
 
-## repo default configuration
-##
-REPO_URL='git://android.kernel.org/tools/repo.git'
-REPO_REV='stable'
+# repo default configuration
+#
+import os
+REPO_URL = os.environ.get('REPO_URL', None)
+if not REPO_URL:
+  REPO_URL = 'https://gerrit.googlesource.com/git-repo'
+REPO_REV = 'stable'
 
 # Copyright (C) 2008 Google Inc.
 #
@@ -19,19 +22,14 @@ REPO_REV='stable'
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-magic='--calling-python-from-/bin/sh--'
-"""exec" python -E "$0" "$@" """#$magic"
-if __name__ == '__main__':
-  import sys
-  if sys.argv[-1] == '#%s' % magic:
-    del sys.argv[-1]
-del magic
-
 # increment this whenever we make important changes to this script
-VERSION = (1, 7)
+VERSION = (1, 25)
 
 # increment this if the MAINTAINER_KEYS block is modified
-KEYRING_VERSION = (1,0)
+KEYRING_VERSION = (1, 2)
+
+# Each individual key entry is created by using:
+# gpg --armor --export keyid
 MAINTAINER_KEYS = """
 
      Repo Maintainer <repo@android.kernel.org>
@@ -74,24 +72,94 @@ HTHs37+/QLMomGEGKZMWi0dShU2J5mNRQu3Hhxl3hHDVbt5CeJBb26aQcQrFz69W
 zE3GNvmJosh6leayjtI9P2A6iEkEGBECAAkFAkj3uiACGwwACgkQFlMNXpIPXGWp
 TACbBS+Up3RpfYVfd63c1cDdlru13pQAn3NQy/SN858MkxN+zym86UBgOad2
 =CMiZ
+-----END PGP PUBLIC KEY BLOCK-----
+
+     Conley Owens <cco3@android.com>
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+Version: GnuPG v1.4.11 (GNU/Linux)
+
+mQENBFHRvc8BCADFg45Xx/y6QDC+T7Y/gGc7vx0ww7qfOwIKlAZ9xG3qKunMxo+S
+hPCnzEl3cq+6I1Ww/ndop/HB3N3toPXRCoN8Vs4/Hc7by+SnaLFnacrm+tV5/OgT
+V37Lzt8lhay1Kl+YfpFwHYYpIEBLFV9knyfRXS/428W2qhdzYfvB15/AasRmwmor
+py4NIzSs8UD/SPr1ihqNCdZM76+MQyN5HMYXW/ALZXUFG0pwluHFA7hrfPG74i8C
+zMiP7qvMWIl/r/jtzHioH1dRKgbod+LZsrDJ8mBaqsZaDmNJMhss9g76XvfMyLra
+9DI9/iFuBpGzeqBv0hwOGQspLRrEoyTeR6n1ABEBAAG0H0NvbmxleSBPd2VucyA8
+Y2NvM0BhbmRyb2lkLmNvbT6JATgEEwECACIFAlHRvc8CGwMGCwkIBwMCBhUIAgkK
+CwQWAgMBAh4BAheAAAoJEGe35EhpKzgsP6AIAJKJmNtn4l7hkYHKHFSo3egb6RjQ
+zEIP3MFTcu8HFX1kF1ZFbrp7xqurLaE53kEkKuAAvjJDAgI8mcZHP1JyplubqjQA
+xvv84gK+OGP3Xk+QK1ZjUQSbjOpjEiSZpRhWcHci3dgOUH4blJfByHw25hlgHowd
+a/2PrNKZVcJ92YienaxxGjcXEUcd0uYEG2+rwllQigFcnMFDhr9B71MfalRHjFKE
+fmdoypqLrri61YBc59P88Rw2/WUpTQjgNubSqa3A2+CKdaRyaRw+2fdF4TdR0h8W
+zbg+lbaPtJHsV+3mJC7fq26MiJDRJa5ZztpMn8su20gbLgi2ShBOaHAYDDi5AQ0E
+UdG9zwEIAMoOBq+QLNozAhxOOl5GL3StTStGRgPRXINfmViTsihrqGCWBBUfXlUE
+OytC0mYcrDUQev/8ToVoyqw+iGSwDkcSXkrEUCKFtHV/GECWtk1keyHgR10YKI1R
+mquSXoubWGqPeG1PAI74XWaRx8UrL8uCXUtmD8Q5J7mDjKR5NpxaXrwlA0bKsf2E
+Gp9tu1kKauuToZhWHMRMqYSOGikQJwWSFYKT1KdNcOXLQF6+bfoJ6sjVYdwfmNQL
+Ixn8QVhoTDedcqClSWB17VDEFDFa7MmqXZz2qtM3X1R/MUMHqPtegQzBGNhRdnI2
+V45+1Nnx/uuCxDbeI4RbHzujnxDiq70AEQEAAYkBHwQYAQIACQUCUdG9zwIbDAAK
+CRBnt+RIaSs4LNVeB/0Y2pZ8I7gAAcEM0Xw8drr4omg2fUoK1J33ozlA/RxeA/lJ
+I3KnyCDTpXuIeBKPGkdL8uMATC9Z8DnBBajRlftNDVZS3Hz4G09G9QpMojvJkFJV
+By+01Flw/X+eeN8NpqSuLV4W+AjEO8at/VvgKr1AFvBRdZ7GkpI1o6DgPe7ZqX+1
+dzQZt3e13W0rVBb/bUgx9iSLoeWP3aq/k+/GRGOR+S6F6BBSl0SQ2EF2+dIywb1x
+JuinEP+AwLAUZ1Bsx9ISC0Agpk2VeHXPL3FGhroEmoMvBzO0kTFGyoeT7PR/BfKv
++H/g3HsL2LOB9uoIm8/5p2TTU5ttYCXMHhQZ81AY
+=AUp4
 -----END PGP PUBLIC KEY BLOCK-----
 """
 
-GIT = 'git'                     # our git command
-MIN_GIT_VERSION = (1, 5, 4)     # minimum supported git version
-repodir = '.repo'               # name of repo's private directory
-S_repo = 'repo'                 # special repo reposiory
-S_manifests = 'manifests'       # special manifest repository
-REPO_MAIN = S_repo + '/main.py' # main script
+GIT = 'git'                      # our git command
+MIN_GIT_VERSION = (1, 7, 2)      # minimum supported git version
+repodir = '.repo'                # name of repo's private directory
+S_repo = 'repo'                  # special repo repository
+S_manifests = 'manifests'        # special manifest repository
+REPO_MAIN = S_repo + '/main.py'  # main script
+MIN_PYTHON_VERSION = (2, 7)      # minimum supported python version
+GITC_CONFIG_FILE = '/gitc/.config'
+GITC_FS_ROOT_DIR = '/gitc/manifest-rw/'
 
 
+import errno
 import optparse
-import os
+import platform
 import re
-import readline
+import shutil
+import stat
 import subprocess
 import sys
 
+if sys.version_info[0] == 3:
+  import urllib.request
+  import urllib.error
+else:
+  import imp
+  import urllib2
+  urllib = imp.new_module('urllib')
+  urllib.request = urllib2
+  urllib.error = urllib2
+
+
+def _print(*objects, **kwargs):
+  sep = kwargs.get('sep', ' ')
+  end = kwargs.get('end', '\n')
+  out = kwargs.get('file', sys.stdout)
+  out.write(sep.join(objects) + end)
+
+  # On Windows stderr is buffered, so flush to maintain the order of error messages.
+  if out == sys.stderr and platform.system() == "Windows":
+    out.flush()
+
+
+# Python version check
+ver = sys.version_info
+if (ver[0], ver[1]) < MIN_PYTHON_VERSION:
+  _print('error: Python version {} unsupported.\n'
+         'Please use Python {}.{} instead.'.format(
+             sys.version.split(' ')[0],
+             MIN_PYTHON_VERSION[0],
+             MIN_PYTHON_VERSION[1],
+         ), file=sys.stderr)
+  sys.exit(1)
+
 home_dot_repo = os.path.expanduser('~/.repoconfig')
 gpg_dir = os.path.join(home_dot_repo, 'gnupg')
 
@@ -115,12 +183,49 @@ group.add_option('-b', '--manifest-branch',
 group.add_option('-m', '--manifest-name',
                  dest='manifest_name',
                  help='initial manifest file', metavar='NAME.xml')
+group.add_option('--current-branch',
+                 dest='current_branch_only', action='store_true',
+                 help='fetch only current manifest branch from server')
 group.add_option('--mirror',
                  dest='mirror', action='store_true',
-                 help='mirror the forrest')
+                 help='create a replica of the remote repositories '
+                      'rather than a client working directory')
+group.add_option('--reference',
+                 dest='reference',
+                 help='location of mirror directory', metavar='DIR')
+group.add_option('--dissociate',
+                 dest='dissociate', action='store_true',
+                 help='dissociate from reference mirrors after clone')
+group.add_option('--depth', type='int', default=None,
+                 dest='depth',
+                 help='create a shallow clone with given depth; see git clone')
+group.add_option('--archive',
+                 dest='archive', action='store_true',
+                 help='checkout an archive instead of a git repository for '
+                      'each project. See git archive.')
+group.add_option('--submodules',
+                 dest='submodules', action='store_true',
+                 help='sync any submodules associated with the manifest repo')
+group.add_option('-g', '--groups',
+                 dest='groups', default='default',
+                 help='restrict manifest projects to ones with specified '
+                      'group(s) [default|all|G1,G2,G3|G4,-G5,-G6]',
+                 metavar='GROUP')
+group.add_option('-p', '--platform',
+                 dest='platform', default="auto",
+                 help='restrict manifest projects to ones with a specified '
+                      'platform group [auto|all|none|linux|darwin|...]',
+                 metavar='PLATFORM')
+group.add_option('--no-clone-bundle',
+                 dest='no_clone_bundle', action='store_true',
+                 help='disable use of /clone.bundle on HTTP/HTTPS')
+group.add_option('--no-tags',
+                 dest='no_tags', action='store_true',
+                 help="don't fetch tags in the manifest")
+
 
 # Tool
-group = init_optparse.add_option_group('Version options')
+group = init_optparse.add_option_group('repo Version options')
 group.add_option('--repo-url',
                  dest='repo_url',
                  help='repo repository location', metavar='URL')
@@ -131,17 +236,77 @@ group.add_option('--no-repo-verify',
                  dest='no_repo_verify', action='store_true',
                  help='do not verify repo source code')
 
+# Other
+group = init_optparse.add_option_group('Other options')
+group.add_option('--config-name',
+                 dest='config_name', action="store_true", default=False,
+                 help='Always prompt for name/e-mail')
+
+
+def _GitcInitOptions(init_optparse_arg):
+  init_optparse_arg.set_usage("repo gitc-init -u url -c client [options]")
+  g = init_optparse_arg.add_option_group('GITC options')
+  g.add_option('-f', '--manifest-file',
+               dest='manifest_file',
+               help='Optional manifest file to use for this GITC client.')
+  g.add_option('-c', '--gitc-client',
+               dest='gitc_client',
+               help='The name of the gitc_client instance to create or modify.')
+
+_gitc_manifest_dir = None
+
+
+def get_gitc_manifest_dir():
+  global _gitc_manifest_dir
+  if _gitc_manifest_dir is None:
+    _gitc_manifest_dir = ''
+    try:
+      with open(GITC_CONFIG_FILE, 'r') as gitc_config:
+        for line in gitc_config:
+          match = re.match('gitc_dir=(?P<gitc_manifest_dir>.*)', line)
+          if match:
+            _gitc_manifest_dir = match.group('gitc_manifest_dir')
+    except IOError:
+      pass
+  return _gitc_manifest_dir
+
+
+def gitc_parse_clientdir(gitc_fs_path):
+  """Parse a path in the GITC FS and return its client name.
+
+  @param gitc_fs_path: A subdirectory path within the GITC_FS_ROOT_DIR.
+
+  @returns: The GITC client name
+  """
+  if gitc_fs_path == GITC_FS_ROOT_DIR:
+    return None
+  if not gitc_fs_path.startswith(GITC_FS_ROOT_DIR):
+    manifest_dir = get_gitc_manifest_dir()
+    if manifest_dir == '':
+      return None
+    if manifest_dir[-1] != '/':
+      manifest_dir += '/'
+    if gitc_fs_path == manifest_dir:
+      return None
+    if not gitc_fs_path.startswith(manifest_dir):
+      return None
+    return gitc_fs_path.split(manifest_dir)[1].split('/')[0]
+  return gitc_fs_path.split(GITC_FS_ROOT_DIR)[1].split('/')[0]
+
 
 class CloneFailure(Exception):
+
   """Indicate the remote clone of repo itself failed.
   """
 
 
-def _Init(args):
+def _Init(args, gitc_init=False):
   """Installs repo by cloning it over the network.
   """
+  if gitc_init:
+    _GitcInitOptions(init_optparse)
   opt, args = init_optparse.parse_args(args)
-  if args or not opt.manifest_url:
+  if args:
     init_optparse.print_usage()
     sys.exit(1)
 
@@ -158,34 +323,49 @@ def _Init(args):
   if branch.startswith('refs/heads/'):
     branch = branch[len('refs/heads/'):]
   if branch.startswith('refs/'):
-    print >>sys.stderr, "fatal: invalid branch name '%s'" % branch
+    _print("fatal: invalid branch name '%s'" % branch, file=sys.stderr)
     raise CloneFailure()
 
-  if not os.path.isdir(repodir):
-    try:
-      os.mkdir(repodir)
-    except OSError, e:
-      print >>sys.stderr, \
-            'fatal: cannot make %s directory: %s' % (
-            repodir, e.strerror)
-      # Don't faise CloneFailure; that would delete the
+  try:
+    if gitc_init:
+      gitc_manifest_dir = get_gitc_manifest_dir()
+      if not gitc_manifest_dir:
+        _print('fatal: GITC filesystem is not available. Exiting...',
+               file=sys.stderr)
+        sys.exit(1)
+      gitc_client = opt.gitc_client
+      if not gitc_client:
+        gitc_client = gitc_parse_clientdir(os.getcwd())
+      if not gitc_client:
+        _print('fatal: GITC client (-c) is required.', file=sys.stderr)
+        sys.exit(1)
+      client_dir = os.path.join(gitc_manifest_dir, gitc_client)
+      if not os.path.exists(client_dir):
+        os.makedirs(client_dir)
+      os.chdir(client_dir)
+      if os.path.exists(repodir):
+        # This GITC Client has already initialized repo so continue.
+        return
+
+    os.mkdir(repodir)
+  except OSError as e:
+    if e.errno != errno.EEXIST:
+      _print('fatal: cannot make %s directory: %s'
+             % (repodir, e.strerror), file=sys.stderr)
+      # Don't raise CloneFailure; that would delete the
       # name. Instead exit immediately.
       #
       sys.exit(1)
 
   _CheckGitVersion()
   try:
-    if _NeedSetupGnuPG():
-      can_verify = _SetupGnuPG(opt.quiet)
+    if NeedSetupGnuPG():
+      can_verify = SetupGnuPG(opt.quiet)
     else:
       can_verify = True
 
-    if not opt.quiet:
-      print >>sys.stderr, 'Getting repo ...'
-      print >>sys.stderr, '   from %s' % url
-
     dst = os.path.abspath(os.path.join(repodir, S_repo))
-    _Clone(url, dst, opt.quiet)
+    _Clone(url, dst, opt.quiet, not opt.no_clone_bundle)
 
     if can_verify and not opt.no_repo_verify:
       rev = _Verify(dst, branch, opt.quiet)
@@ -193,33 +373,61 @@ def _Init(args):
       rev = 'refs/remotes/origin/%s^0' % branch
 
     _Checkout(dst, branch, rev, opt.quiet)
+
+    if not os.path.isfile(os.path.join(dst, 'repo')):
+      _print("warning: '%s' does not look like a git-repo repository, is "
+             "REPO_URL set correctly?" % url, file=sys.stderr)
+
   except CloneFailure:
     if opt.quiet:
-      print >>sys.stderr, \
-        'fatal: repo init failed; run without --quiet to see why'
+      _print('fatal: repo init failed; run without --quiet to see why',
+             file=sys.stderr)
     raise
 
 
+def ParseGitVersion(ver_str):
+  if not ver_str.startswith('git version '):
+    return None
+
+  num_ver_str = ver_str[len('git version '):].strip().split('-')[0]
+  to_tuple = []
+  for num_str in num_ver_str.split('.')[:3]:
+    if num_str.isdigit():
+      to_tuple.append(int(num_str))
+    else:
+      to_tuple.append(0)
+  return tuple(to_tuple)
+
+
 def _CheckGitVersion():
   cmd = [GIT, '--version']
-  proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
+  try:
+    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
+  except OSError as e:
+    _print(file=sys.stderr)
+    _print("fatal: '%s' is not available" % GIT, file=sys.stderr)
+    _print('fatal: %s' % e, file=sys.stderr)
+    _print(file=sys.stderr)
+    _print('Please make sure %s is installed and in your path.' % GIT,
+           file=sys.stderr)
+    raise CloneFailure()
+
   ver_str = proc.stdout.read().strip()
   proc.stdout.close()
   proc.wait()
 
-  if not ver_str.startswith('git version '):
-    print >>sys.stderr, 'error: "%s" unsupported' % ver_str
+  ver_act = ParseGitVersion(ver_str)
+  if ver_act is None:
+    _print('error: "%s" unsupported' % ver_str, file=sys.stderr)
     raise CloneFailure()
 
-  ver_str = ver_str[len('git version '):].strip()
-  ver_act = tuple(map(lambda x: int(x), ver_str.split('.')[0:3]))
   if ver_act < MIN_GIT_VERSION:
-    need = '.'.join(map(lambda x: str(x), MIN_GIT_VERSION))
-    print >>sys.stderr, 'fatal: git %s or later required' % need
+    need = '.'.join(map(str, MIN_GIT_VERSION))
+    _print('fatal: git %s or later required' % need, file=sys.stderr)
     raise CloneFailure()
 
 
-def _NeedSetupGnuPG():
+def NeedSetupGnuPG():
   if not os.path.isdir(home_dot_repo):
     return True
 
@@ -231,56 +439,57 @@ def _NeedSetupGnuPG():
   if not kv:
     return True
 
-  kv = tuple(map(lambda x: int(x), kv.split('.')))
+  kv = tuple(map(int, kv.split('.')))
   if kv < KEYRING_VERSION:
     return True
   return False
 
 
-def _SetupGnuPG(quiet):
-  if not os.path.isdir(home_dot_repo):
-    try:
-      os.mkdir(home_dot_repo)
-    except OSError, e:
-      print >>sys.stderr, \
-            'fatal: cannot make %s directory: %s' % (
-            home_dot_repo, e.strerror)
+def SetupGnuPG(quiet):
+  try:
+    os.mkdir(home_dot_repo)
+  except OSError as e:
+    if e.errno != errno.EEXIST:
+      _print('fatal: cannot make %s directory: %s'
+             % (home_dot_repo, e.strerror), file=sys.stderr)
       sys.exit(1)
 
-  if not os.path.isdir(gpg_dir):
-    try:
-      os.mkdir(gpg_dir, 0700)
-    except OSError, e:
-      print >>sys.stderr, \
-            'fatal: cannot make %s directory: %s' % (
-            gpg_dir, e.strerror)
+  try:
+    os.mkdir(gpg_dir, stat.S_IRWXU)
+  except OSError as e:
+    if e.errno != errno.EEXIST:
+      _print('fatal: cannot make %s directory: %s' % (gpg_dir, e.strerror),
+             file=sys.stderr)
       sys.exit(1)
 
-  env = dict(os.environ)
-  env['GNUPGHOME'] = gpg_dir
+  env = os.environ.copy()
+  try:
+    env['GNUPGHOME'] = gpg_dir
+  except UnicodeEncodeError:
+    env['GNUPGHOME'] = gpg_dir.encode()
 
   cmd = ['gpg', '--import']
   try:
     proc = subprocess.Popen(cmd,
-                            env = env,
-                            stdin = subprocess.PIPE)
-  except OSError, e:
+                            env=env,
+                            stdin=subprocess.PIPE)
+  except OSError as e:
     if not quiet:
-      print >>sys.stderr, 'warning: gpg (GnuPG) is not available.'
-      print >>sys.stderr, 'warning: Installing it is strongly encouraged.'
-      print >>sys.stderr
+      _print('warning: gpg (GnuPG) is not available.', file=sys.stderr)
+      _print('warning: Installing it is strongly encouraged.', file=sys.stderr)
+      _print(file=sys.stderr)
     return False
 
   proc.stdin.write(MAINTAINER_KEYS)
   proc.stdin.close()
 
   if proc.wait() != 0:
-    print >>sys.stderr, 'fatal: registering repo maintainer keys failed'
+    _print('fatal: registering repo maintainer keys failed', file=sys.stderr)
     sys.exit(1)
-  print
+  _print()
 
   fd = open(os.path.join(home_dot_repo, 'keyring-version'), 'w')
-  fd.write('.'.join(map(lambda x: str(x), KEYRING_VERSION)) + '\n')
+  fd.write('.'.join(map(str, KEYRING_VERSION)) + '\n')
   fd.close()
   return True
 
@@ -289,21 +498,50 @@ def _SetConfig(local, name, value):
   """Set a git configuration option to the specified value.
   """
   cmd = [GIT, 'config', name, value]
-  if subprocess.Popen(cmd, cwd = local).wait() != 0:
+  if subprocess.Popen(cmd, cwd=local).wait() != 0:
     raise CloneFailure()
 
 
-def _Fetch(local, quiet, *args):
+def _InitHttp():
+  handlers = []
+
+  mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
+  try:
+    import netrc
+    n = netrc.netrc()
+    for host in n.hosts:
+      p = n.hosts[host]
+      mgr.add_password(p[1], 'http://%s/' % host, p[0], p[2])
+      mgr.add_password(p[1], 'https://%s/' % host, p[0], p[2])
+  except:
+    pass
+  handlers.append(urllib.request.HTTPBasicAuthHandler(mgr))
+  handlers.append(urllib.request.HTTPDigestAuthHandler(mgr))
+
+  if 'http_proxy' in os.environ:
+    url = os.environ['http_proxy']
+    handlers.append(urllib.request.ProxyHandler({'http': url, 'https': url}))
+  if 'REPO_CURL_VERBOSE' in os.environ:
+    handlers.append(urllib.request.HTTPHandler(debuglevel=1))
+    handlers.append(urllib.request.HTTPSHandler(debuglevel=1))
+  urllib.request.install_opener(urllib.request.build_opener(*handlers))
+
+
+def _Fetch(url, local, src, quiet):
+  if not quiet:
+    _print('Get %s' % url, file=sys.stderr)
+
   cmd = [GIT, 'fetch']
   if quiet:
     cmd.append('--quiet')
     err = subprocess.PIPE
   else:
     err = None
-  cmd.extend(args)
-  cmd.append('origin')
+  cmd.append(src)
+  cmd.append('+refs/heads/*:refs/remotes/origin/*')
+  cmd.append('+refs/tags/*:refs/tags/*')
 
-  proc = subprocess.Popen(cmd, cwd = local, stderr = err)
+  proc = subprocess.Popen(cmd, cwd=local, stderr=err)
   if err:
     proc.stderr.read()
     proc.stderr.close()
@@ -311,37 +549,98 @@ def _Fetch(local, quiet, *args):
     raise CloneFailure()
 
 
-def _Clone(url, local, quiet):
+def _DownloadBundle(url, local, quiet):
+  if not url.endswith('/'):
+    url += '/'
+  url += 'clone.bundle'
+
+  proc = subprocess.Popen(
+      [GIT, 'config', '--get-regexp', 'url.*.insteadof'],
+      cwd=local,
+      stdout=subprocess.PIPE)
+  for line in proc.stdout:
+    m = re.compile(r'^url\.(.*)\.insteadof (.*)$').match(line)
+    if m:
+      new_url = m.group(1)
+      old_url = m.group(2)
+      if url.startswith(old_url):
+        url = new_url + url[len(old_url):]
+        break
+  proc.stdout.close()
+  proc.wait()
+
+  if not url.startswith('http:') and not url.startswith('https:'):
+    return False
+
+  dest = open(os.path.join(local, '.git', 'clone.bundle'), 'w+b')
+  try:
+    try:
+      r = urllib.request.urlopen(url)
+    except urllib.error.HTTPError as e:
+      if e.code in [401, 403, 404, 501]:
+        return False
+      _print('fatal: Cannot get %s' % url, file=sys.stderr)
+      _print('fatal: HTTP error %s' % e.code, file=sys.stderr)
+      raise CloneFailure()
+    except urllib.error.URLError as e:
+      _print('fatal: Cannot get %s' % url, file=sys.stderr)
+      _print('fatal: error %s' % e.reason, file=sys.stderr)
+      raise CloneFailure()
+    try:
+      if not quiet:
+        _print('Get %s' % url, file=sys.stderr)
+      while True:
+        buf = r.read(8192)
+        if buf == '':
+          return True
+        dest.write(buf)
+    finally:
+      r.close()
+  finally:
+    dest.close()
+
+
+def _ImportBundle(local):
+  path = os.path.join(local, '.git', 'clone.bundle')
+  try:
+    _Fetch(local, local, path, True)
+  finally:
+    os.remove(path)
+
+
+def _Clone(url, local, quiet, clone_bundle):
   """Clones a git repository to a new subdirectory of repodir
   """
   try:
     os.mkdir(local)
-  except OSError, e:
-    print >>sys.stderr, \
-          'fatal: cannot make %s directory: %s' \
-          % (local, e.strerror)
+  except OSError as e:
+    _print('fatal: cannot make %s directory: %s' % (local, e.strerror),
+           file=sys.stderr)
     raise CloneFailure()
 
   cmd = [GIT, 'init', '--quiet']
   try:
-    proc = subprocess.Popen(cmd, cwd = local)
-  except OSError, e:
-    print >>sys.stderr
-    print >>sys.stderr, "fatal: '%s' is not available" % GIT
-    print >>sys.stderr, 'fatal: %s' % e
-    print >>sys.stderr
-    print >>sys.stderr, 'Please make sure %s is installed'\
-                        ' and in your path.' % GIT
+    proc = subprocess.Popen(cmd, cwd=local)
+  except OSError as e:
+    _print(file=sys.stderr)
+    _print("fatal: '%s' is not available" % GIT, file=sys.stderr)
+    _print('fatal: %s' % e, file=sys.stderr)
+    _print(file=sys.stderr)
+    _print('Please make sure %s is installed and in your path.' % GIT,
+           file=sys.stderr)
     raise CloneFailure()
   if proc.wait() != 0:
-    print >>sys.stderr, 'fatal: could not create %s' % local
+    _print('fatal: could not create %s' % local, file=sys.stderr)
     raise CloneFailure()
 
+  _InitHttp()
   _SetConfig(local, 'remote.origin.url', url)
-  _SetConfig(local, 'remote.origin.fetch',
-                    '+refs/heads/*:refs/remotes/origin/*')
-  _Fetch(local, quiet)
-  _Fetch(local, quiet, '--tags')
+  _SetConfig(local,
+             'remote.origin.fetch',
+             '+refs/heads/*:refs/remotes/origin/*')
+  if clone_bundle and _DownloadBundle(url, local, quiet):
+    _ImportBundle(local)
+  _Fetch(url, local, 'origin', quiet)
 
 
 def _Verify(cwd, branch, quiet):
@@ -351,7 +650,7 @@ def _Verify(cwd, branch, quiet):
   proc = subprocess.Popen(cmd,
                           stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE,
-                          cwd = cwd)
+                          cwd=cwd)
   cur = proc.stdout.read().strip()
   proc.stdout.close()
 
@@ -359,31 +658,31 @@ def _Verify(cwd, branch, quiet):
   proc.stderr.close()
 
   if proc.wait() != 0 or not cur:
-    print >>sys.stderr
-    print >>sys.stderr,\
-      "fatal: branch '%s' has not been signed" \
-      % branch
+    _print(file=sys.stderr)
+    _print("fatal: branch '%s' has not been signed" % branch, file=sys.stderr)
     raise CloneFailure()
 
   m = re.compile(r'^(.*)-[0-9]{1,}-g[0-9a-f]{1,}$').match(cur)
   if m:
     cur = m.group(1)
     if not quiet:
-      print >>sys.stderr
-      print >>sys.stderr, \
-        "info: Ignoring branch '%s'; using tagged release '%s'" \
-        % (branch, cur)
-      print >>sys.stderr
+      _print(file=sys.stderr)
+      _print("info: Ignoring branch '%s'; using tagged release '%s'"
+             % (branch, cur), file=sys.stderr)
+      _print(file=sys.stderr)
 
-  env = dict(os.environ)
-  env['GNUPGHOME'] = gpg_dir
+  env = os.environ.copy()
+  try:
+    env['GNUPGHOME'] = gpg_dir
+  except UnicodeEncodeError:
+    env['GNUPGHOME'] = gpg_dir.encode()
 
   cmd = [GIT, 'tag', '-v', cur]
   proc = subprocess.Popen(cmd,
-                          stdout = subprocess.PIPE,
-                          stderr = subprocess.PIPE,
-                          cwd = cwd,
-                          env = env)
+                          stdout=subprocess.PIPE,
+                          stderr=subprocess.PIPE,
+                          cwd=cwd,
+                          env=env)
   out = proc.stdout.read()
   proc.stdout.close()
 
@@ -391,10 +690,10 @@ def _Verify(cwd, branch, quiet):
   proc.stderr.close()
 
   if proc.wait() != 0:
-    print >>sys.stderr
-    print >>sys.stderr, out
-    print >>sys.stderr, err
-    print >>sys.stderr
+    _print(file=sys.stderr)
+    _print(out, file=sys.stderr)
+    _print(err, file=sys.stderr)
+    _print(file=sys.stderr)
     raise CloneFailure()
   return '%s^0' % cur
 
@@ -403,39 +702,43 @@ def _Checkout(cwd, branch, rev, quiet):
   """Checkout an upstream branch into the repository and track it.
   """
   cmd = [GIT, 'update-ref', 'refs/heads/default', rev]
-  if subprocess.Popen(cmd, cwd = cwd).wait() != 0:
+  if subprocess.Popen(cmd, cwd=cwd).wait() != 0:
     raise CloneFailure()
 
   _SetConfig(cwd, 'branch.default.remote', 'origin')
   _SetConfig(cwd, 'branch.default.merge', 'refs/heads/%s' % branch)
 
   cmd = [GIT, 'symbolic-ref', 'HEAD', 'refs/heads/default']
-  if subprocess.Popen(cmd, cwd = cwd).wait() != 0:
+  if subprocess.Popen(cmd, cwd=cwd).wait() != 0:
     raise CloneFailure()
 
   cmd = [GIT, 'read-tree', '--reset', '-u']
   if not quiet:
     cmd.append('-v')
   cmd.append('HEAD')
-  if subprocess.Popen(cmd, cwd = cwd).wait() != 0:
+  if subprocess.Popen(cmd, cwd=cwd).wait() != 0:
     raise CloneFailure()
 
 
 def _FindRepo():
   """Look for a repo installation, starting at the current directory.
   """
-  dir = os.getcwd()
+  curdir = os.getcwd()
   repo = None
 
-  while dir != '/' and not repo:
-    repo = os.path.join(dir, repodir, REPO_MAIN)
+  olddir = None
+  while curdir != '/' \
+          and curdir != olddir \
+          and not repo:
+    repo = os.path.join(curdir, repodir, REPO_MAIN)
     if not os.path.isfile(repo):
       repo = None
-      dir = os.path.dirname(dir)
-  return (repo, os.path.join(dir, repodir))
+      olddir = curdir
+      curdir = os.path.dirname(curdir)
+  return (repo, os.path.join(curdir, repodir))
 
 
-class _Options:
+class _Options(object):
   help = False
 
 
@@ -444,7 +747,7 @@ def _ParseArguments(args):
   opt = _Options()
   arg = []
 
-  for i in xrange(0, len(args)):
+  for i in range(len(args)):
     a = args[i]
     if a == '-h' or a == '--help':
       opt.help = True
@@ -457,18 +760,23 @@ def _ParseArguments(args):
 
 
 def _Usage():
-  print >>sys.stderr,\
-"""usage: repo COMMAND [ARGS]
+  gitc_usage = ""
+  if get_gitc_manifest_dir():
+    gitc_usage = "  gitc-init Initialize a GITC Client.\n"
+
+  _print(
+      """usage: repo COMMAND [ARGS]
 
 repo is not yet installed.  Use "repo init" to install it here.
 
 The most commonly used repo commands are:
 
   init      Install repo in the current working directory
-  help      Display detailed help on a command
+""" + gitc_usage +
+      """  help      Display detailed help on a command
 
 For access to the full online help, install repo ("repo init").
-"""
+""", file=sys.stderr)
   sys.exit(1)
 
 
@@ -476,26 +784,29 @@ def _Help(args):
   if args:
     if args[0] == 'init':
       init_optparse.print_help()
+      sys.exit(0)
+    elif args[0] == 'gitc-init':
+      _GitcInitOptions(init_optparse)
+      init_optparse.print_help()
+      sys.exit(0)
     else:
-      print >>sys.stderr,\
-      "error: '%s' is not a bootstrap command.\n"\
-      '        For access to online help, install repo ("repo init").'\
-      % args[0]
+      _print("error: '%s' is not a bootstrap command.\n"
+             '        For access to online help, install repo ("repo init").'
+             % args[0], file=sys.stderr)
   else:
     _Usage()
   sys.exit(1)
 
 
 def _NotInstalled():
-  print >>sys.stderr,\
-'error: repo is not installed.  Use "repo init" to install it here.'
+  _print('error: repo is not installed.  Use "repo init" to install it here.',
+         file=sys.stderr)
   sys.exit(1)
 
 
 def _NoCommands(cmd):
-  print >>sys.stderr,\
-"""error: command '%s' requires repo to be installed first.
-       Use "repo init" to install it here.""" % cmd
+  _print("""error: command '%s' requires repo to be installed first.
+         Use "repo init" to install it here.""" % cmd, file=sys.stderr)
   sys.exit(1)
 
 
@@ -505,7 +816,7 @@ def _RunSelf(wrapper_path):
   my_git = os.path.join(my_dir, '.git')
 
   if os.path.isfile(my_main) and os.path.isdir(my_git):
-    for name in ['manifest.py',
+    for name in ['git_config.py',
                  'project.py',
                  'subcmds']:
       if not os.path.exists(os.path.join(my_dir, name)):
@@ -523,8 +834,8 @@ def _SetDefaultsTo(gitdir):
                            '--git-dir=%s' % gitdir,
                            'symbolic-ref',
                            'HEAD'],
-                          stdout = subprocess.PIPE,
-                          stderr = subprocess.PIPE)
+                          stdout=subprocess.PIPE,
+                          stderr=subprocess.PIPE)
   REPO_REV = proc.stdout.read().strip()
   proc.stdout.close()
 
@@ -532,60 +843,75 @@ def _SetDefaultsTo(gitdir):
   proc.stderr.close()
 
   if proc.wait() != 0:
-    print >>sys.stderr, 'fatal: %s has no current branch' % gitdir
+    _print('fatal: %s has no current branch' % gitdir, file=sys.stderr)
     sys.exit(1)
 
 
 def main(orig_args):
-  main, dir = _FindRepo()
   cmd, opt, args = _ParseArguments(orig_args)
 
+  repo_main, rel_repo_dir = None, None
+  # Don't use the local repo copy, make sure to switch to the gitc client first.
+  if cmd != 'gitc-init':
+    repo_main, rel_repo_dir = _FindRepo()
+
   wrapper_path = os.path.abspath(__file__)
   my_main, my_git = _RunSelf(wrapper_path)
 
-  if not main:
+  cwd = os.getcwd()
+  if get_gitc_manifest_dir() and cwd.startswith(get_gitc_manifest_dir()):
+    _print('error: repo cannot be used in the GITC local manifest directory.'
+           '\nIf you want to work on this GITC client please rerun this '
+           'command from the corresponding client under /gitc/',
+           file=sys.stderr)
+    sys.exit(1)
+  if not repo_main:
     if opt.help:
       _Usage()
     if cmd == 'help':
       _Help(args)
     if not cmd:
       _NotInstalled()
-    if cmd == 'init':
+    if cmd == 'init' or cmd == 'gitc-init':
       if my_git:
         _SetDefaultsTo(my_git)
       try:
-        _Init(args)
+        _Init(args, gitc_init=(cmd == 'gitc-init'))
       except CloneFailure:
-        for root, dirs, files in os.walk(repodir, topdown=False):
-          for name in files:
-            os.remove(os.path.join(root, name))
-          for name in dirs:
-            os.rmdir(os.path.join(root, name))
-        os.rmdir(repodir)
+        path = os.path.join(repodir, S_repo)
+        _print("fatal: cloning the git-repo repository failed, will remove "
+               "'%s' " % path, file=sys.stderr)
+        shutil.rmtree(path, ignore_errors=True)
         sys.exit(1)
-      main, dir = _FindRepo()
+      repo_main, rel_repo_dir = _FindRepo()
     else:
       _NoCommands(cmd)
 
   if my_main:
-    main = my_main
+    repo_main = my_main
 
-  ver_str = '.'.join(map(lambda x: str(x), VERSION))
-  me = [main,
-        '--repo-dir=%s' % dir,
+  ver_str = '.'.join(map(str, VERSION))
+  me = [sys.executable, repo_main,
+        '--repo-dir=%s' % rel_repo_dir,
         '--wrapper-version=%s' % ver_str,
         '--wrapper-path=%s' % wrapper_path,
         '--']
   me.extend(orig_args)
   me.extend(extra_args)
   try:
-    os.execv(main, me)
-  except OSError, e:
-    print >>sys.stderr, "fatal: unable to start %s" % main
-    print >>sys.stderr, "fatal: %s" % e
+    if platform.system() == "Windows":
+      sys.exit(subprocess.call(me))
+    else:
+      os.execv(sys.executable, me)
+  except OSError as e:
+    _print("fatal: unable to start %s" % repo_main, file=sys.stderr)
+    _print("fatal: %s" % e, file=sys.stderr)
     sys.exit(148)
 
 
 if __name__ == '__main__':
+  if ver[0] == 3:
+    _print('warning: Python 3 support is currently experimental. YMMV.\n'
+           'Please use Python 2.7 instead.',
+           file=sys.stderr)
   main(sys.argv[1:])
-

subcmds/__init__.py

@@ -15,7 +15,7 @@
 
 import os
 
-all = {}
+all_commands = {}
 
 my_dir = os.path.dirname(__file__)
 for py in os.listdir(my_dir):
@@ -38,12 +38,12 @@ for py in os.listdir(my_dir):
     try:
       cmd = getattr(mod, clsn)()
     except AttributeError:
-      raise SyntaxError, '%s/%s does not define class %s' % (
-                         __name__, py, clsn)
+      raise SyntaxError('%s/%s does not define class %s' % (
+                         __name__, py, clsn))
 
     name = name.replace('_', '-')
     cmd.NAME = name
-    all[name] = cmd
+    all_commands[name] = cmd
 
-if 'help' in all:
-  all['help'].commands = all
+if 'help' in all_commands:
+  all_commands['help'].commands = all_commands

subcmds/abandon.py

@@ -13,30 +13,86 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
 import sys
 from command import Command
+from collections import defaultdict
 from git_command import git
+from progress import Progress
 
 class Abandon(Command):
   common = True
   helpSummary = "Permanently abandon a development branch"
   helpUsage = """
-%prog <branchname> [<project>...]
+%prog [--all | <branchname>] [<project>...]
 
 This subcommand permanently abandons a development branch by
 deleting it (and all its history) from your local repository.
 
 It is equivalent to "git branch -D <branchname>".
 """
+  def _Options(self, p):
+    p.add_option('--all',
+                 dest='all', action='store_true',
+                 help='delete all branches in all projects')
 
   def Execute(self, opt, args):
-    if not args:
+    if not opt.all and not args:
       self.Usage()
 
-    nb = args[0]
-    if not git.check_ref_format('heads/%s' % nb):
-      print >>sys.stderr, "error: '%s' is not a valid name" % nb
-      sys.exit(1)
+    if not opt.all:
+      nb = args[0]
+      if not git.check_ref_format('heads/%s' % nb):
+        print("error: '%s' is not a valid name" % nb, file=sys.stderr)
+        sys.exit(1)
+    else:
+      args.insert(0,None)
+      nb = "'All local branches'"
 
-    for project in self.GetProjects(args[1:]):
-      project.AbandonBranch(nb)
+    err = defaultdict(list)
+    success = defaultdict(list)
+    all_projects = self.GetProjects(args[1:])
+
+    pm = Progress('Abandon %s' % nb, len(all_projects))
+    for project in all_projects:
+      pm.update()
+
+      if opt.all:
+        branches = project.GetBranches().keys()
+      else:
+        branches = [nb]
+
+      for name in branches:
+        status = project.AbandonBranch(name)
+        if status is not None:
+          if status:
+            success[name].append(project)
+          else:
+            err[name].append(project)
+    pm.end()
+
+    width = 25
+    for name in branches:
+      if width < len(name):
+        width = len(name)
+
+    if err:
+      for br in err.keys():
+        err_msg = "error: cannot abandon %s" %br
+        print(err_msg, file=sys.stderr)
+        for proj in err[br]:
+          print(' '*len(err_msg) + " | %s" % proj.relpath, file=sys.stderr)
+      sys.exit(1)
+    elif not success:
+      print('error: no project has local branch(es) : %s' % nb,
+            file=sys.stderr)
+      sys.exit(1)
+    else:
+      print('Abandoned branches:', file=sys.stderr)
+      for br in success.keys():
+        if len(all_projects) > 1 and len(all_projects) == len(success[br]):
+          result = "all project"
+        else:
+          result = "%s" % (
+            ('\n'+' '*width + '| ').join(p.relpath for p in success[br]))
+        print("%s%s| %s\n" % (br,' '*(width-len(br)), result),file=sys.stderr)

subcmds/branches.py

@@ -0,0 +1,179 @@
+#
+# Copyright (C) 2009 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import sys
+from color import Coloring
+from command import Command
+
+class BranchColoring(Coloring):
+  def __init__(self, config):
+    Coloring.__init__(self, config, 'branch')
+    self.current = self.printer('current', fg='green')
+    self.local   = self.printer('local')
+    self.notinproject = self.printer('notinproject', fg='red')
+
+class BranchInfo(object):
+  def __init__(self, name):
+    self.name = name
+    self.current = 0
+    self.published = 0
+    self.published_equal = 0
+    self.projects = []
+
+  def add(self, b):
+    if b.current:
+      self.current += 1
+    if b.published:
+      self.published += 1
+    if b.revision == b.published:
+      self.published_equal += 1
+    self.projects.append(b)
+
+  @property
+  def IsCurrent(self):
+    return self.current > 0
+
+  @property
+  def IsSplitCurrent(self):
+    return self.current != 0 and self.current != len(self.projects)
+
+  @property
+  def IsPublished(self):
+    return self.published > 0
+
+  @property
+  def IsPublishedEqual(self):
+    return self.published_equal == len(self.projects)
+
+
+class Branches(Command):
+  common = True
+  helpSummary = "View current topic branches"
+  helpUsage = """
+%prog [<project>...]
+
+Summarizes the currently available topic branches.
+
+# Branch Display
+
+The branch display output by this command is organized into four
+columns of information; for example:
+
+ *P nocolor                   | in repo
+    repo2                     |
+
+The first column contains a * if the branch is the currently
+checked out branch in any of the specified projects, or a blank
+if no project has the branch checked out.
+
+The second column contains either blank, p or P, depending upon
+the upload status of the branch.
+
+ (blank): branch not yet published by repo upload
+       P: all commits were published by repo upload
+       p: only some commits were published by repo upload
+
+The third column contains the branch name.
+
+The fourth column (after the | separator) lists the projects that
+the branch appears in, or does not appear in.  If no project list
+is shown, then the branch appears in all projects.
+
+"""
+
+  def Execute(self, opt, args):
+    projects = self.GetProjects(args)
+    out = BranchColoring(self.manifest.manifestProject.config)
+    all_branches = {}
+    project_cnt = len(projects)
+
+    for project in projects:
+      for name, b in project.GetBranches().items():
+        b.project = project
+        if name not in all_branches:
+          all_branches[name] = BranchInfo(name)
+        all_branches[name].add(b)
+
+    names = list(sorted(all_branches))
+
+    if not names:
+      print('   (no branches)', file=sys.stderr)
+      return
+
+    width = 25
+    for name in names:
+      if width < len(name):
+        width = len(name)
+
+    for name in names:
+      i = all_branches[name]
+      in_cnt = len(i.projects)
+
+      if i.IsCurrent:
+        current = '*'
+        hdr = out.current
+      else:
+        current = ' '
+        hdr = out.local
+
+      if i.IsPublishedEqual:
+        published = 'P'
+      elif i.IsPublished:
+        published = 'p'
+      else:
+        published = ' '
+
+      hdr('%c%c %-*s' % (current, published, width, name))
+      out.write(' |')
+
+      if in_cnt < project_cnt:
+        fmt = out.write
+        paths = []
+        non_cur_paths = []
+        if i.IsSplitCurrent or (in_cnt < project_cnt - in_cnt):
+          in_type = 'in'
+          for b in i.projects:
+            if not i.IsSplitCurrent or b.current:
+              paths.append(b.project.relpath)
+            else:
+              non_cur_paths.append(b.project.relpath)
+        else:
+          fmt = out.notinproject
+          in_type = 'not in'
+          have = set()
+          for b in i.projects:
+            have.add(b.project)
+          for p in projects:
+            if not p in have:
+              paths.append(p.relpath)
+
+        s = ' %s %s' % (in_type, ', '.join(paths))
+        if not i.IsSplitCurrent and (width + 7 + len(s) < 80):
+          fmt = out.current if i.IsCurrent else fmt
+          fmt(s)
+        else:
+          fmt(' %s:' % in_type)
+          fmt = out.current if i.IsCurrent else out.write
+          for p in paths:
+            out.nl()
+            fmt(width*' ' + '          %s' % p)
+          fmt = out.write
+          for p in non_cur_paths:
+            out.nl()
+            fmt(width*' ' + '          %s' % p)
+      else:
+        out.write(' in all projects')
+      out.nl()

subcmds/checkout.py

@@ -0,0 +1,64 @@
+#
+# Copyright (C) 2009 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import sys
+from command import Command
+from progress import Progress
+
+class Checkout(Command):
+  common = True
+  helpSummary = "Checkout a branch for development"
+  helpUsage = """
+%prog <branchname> [<project>...]
+"""
+  helpDescription = """
+The '%prog' command checks out an existing branch that was previously
+created by 'repo start'.
+
+The command is equivalent to:
+
+  repo forall [<project>...] -c git checkout <branchname>
+"""
+
+  def Execute(self, opt, args):
+    if not args:
+      self.Usage()
+
+    nb = args[0]
+    err = []
+    success = []
+    all_projects = self.GetProjects(args[1:])
+
+    pm = Progress('Checkout %s' % nb, len(all_projects))
+    for project in all_projects:
+      pm.update()
+
+      status = project.CheckoutBranch(nb)
+      if status is not None:
+        if status:
+          success.append(project)
+        else:
+          err.append(project)
+    pm.end()
+
+    if err:
+      for p in err:
+        print("error: %s/: cannot checkout %s" % (p.relpath, nb),
+              file=sys.stderr)
+      sys.exit(1)
+    elif not success:
+      print('error: no project has branch %s' % nb, file=sys.stderr)
+      sys.exit(1)

subcmds/cherry_pick.py

@@ -0,0 +1,115 @@
+#
+# Copyright (C) 2010 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import re
+import sys
+from command import Command
+from git_command import GitCommand
+
+CHANGE_ID_RE = re.compile(r'^\s*Change-Id: I([0-9a-f]{40})\s*$')
+
+class CherryPick(Command):
+  common = True
+  helpSummary = "Cherry-pick a change."
+  helpUsage = """
+%prog <sha1>
+"""
+  helpDescription = """
+'%prog' cherry-picks a change from one branch to another.
+The change id will be updated, and a reference to the old
+change id will be added.
+"""
+
+  def _Options(self, p):
+    pass
+
+  def Execute(self, opt, args):
+    if len(args) != 1:
+      self.Usage()
+
+    reference = args[0]
+
+    p = GitCommand(None,
+                   ['rev-parse', '--verify', reference],
+                   capture_stdout = True,
+                   capture_stderr = True)
+    if p.Wait() != 0:
+      print(p.stderr, file=sys.stderr)
+      sys.exit(1)
+    sha1 = p.stdout.strip()
+
+    p = GitCommand(None, ['cat-file', 'commit', sha1], capture_stdout=True)
+    if p.Wait() != 0:
+      print("error: Failed to retrieve old commit message", file=sys.stderr)
+      sys.exit(1)
+    old_msg = self._StripHeader(p.stdout)
+
+    p = GitCommand(None,
+                   ['cherry-pick', sha1],
+                   capture_stdout = True,
+                   capture_stderr = True)
+    status = p.Wait()
+
+    print(p.stdout, file=sys.stdout)
+    print(p.stderr, file=sys.stderr)
+
+    if status == 0:
+      # The cherry-pick was applied correctly. We just need to edit the
+      # commit message.
+      new_msg = self._Reformat(old_msg, sha1)
+
+      p = GitCommand(None, ['commit', '--amend', '-F', '-'],
+                     provide_stdin = True,
+                     capture_stdout = True,
+                     capture_stderr = True)
+      p.stdin.write(new_msg)
+      p.stdin.close()
+      if p.Wait() != 0:
+        print("error: Failed to update commit message", file=sys.stderr)
+        sys.exit(1)
+
+    else:
+      print('NOTE: When committing (please see above) and editing the commit '
+            'message, please remove the old Change-Id-line and add:')
+      print(self._GetReference(sha1), file=sys.stderr)
+      print(file=sys.stderr)
+
+  def _IsChangeId(self, line):
+    return CHANGE_ID_RE.match(line)
+
+  def _GetReference(self, sha1):
+    return "(cherry picked from commit %s)" % sha1
+
+  def _StripHeader(self, commit_msg):
+    lines = commit_msg.splitlines()
+    return "\n".join(lines[lines.index("")+1:])
+
+  def _Reformat(self, old_msg, sha1):
+    new_msg = []
+
+    for line in old_msg.splitlines():
+      if not self._IsChangeId(line):
+        new_msg.append(line)
+
+    # Add a blank line between the message and the change id/reference
+    try:
+      if new_msg[-1].strip() != "":
+        new_msg.append("")
+    except IndexError:
+      pass
+
+    new_msg.append(self._GetReference(sha1))
+    return "\n".join(new_msg)

subcmds/diff.py

@@ -20,8 +20,21 @@ class Diff(PagedCommand):
   helpSummary = "Show changes between commit and working tree"
   helpUsage = """
 %prog [<project>...]
+
+The -u option causes '%prog' to generate diff output with file paths
+relative to the repository root, so the output can be applied
+to the Unix 'patch' command.
 """
 
+  def _Options(self, p):
+    def cmd(option, opt_str, value, parser):
+      setattr(parser.values, option.dest, list(parser.rargs))
+      while parser.rargs:
+        del parser.rargs[0]
+    p.add_option('-u', '--absolute',
+                 dest='absolute', action='store_true',
+                 help='Paths are relative to the repository root')
+
   def Execute(self, opt, args):
     for project in self.GetProjects(args):
-      project.PrintWorkTreeDiff()
+      project.PrintWorkTreeDiff(opt.absolute)

subcmds/diffmanifests.py

@@ -0,0 +1,204 @@
+#
+# Copyright (C) 2014 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from color import Coloring
+from command import PagedCommand
+from manifest_xml import XmlManifest
+
+class _Coloring(Coloring):
+  def __init__(self, config):
+    Coloring.__init__(self, config, "status")
+
+class Diffmanifests(PagedCommand):
+  """ A command to see logs in projects represented by manifests
+
+  This is used to see deeper differences between manifests. Where a simple
+  diff would only show a diff of sha1s for example, this command will display
+  the logs of the project between both sha1s, allowing user to see diff at a
+  deeper level.
+  """
+
+  common = True
+  helpSummary = "Manifest diff utility"
+  helpUsage = """%prog manifest1.xml [manifest2.xml] [options]"""
+
+  helpDescription = """
+The %prog command shows differences between project revisions of manifest1 and
+manifest2. if manifest2 is not specified, current manifest.xml will be used
+instead. Both absolute and relative paths may be used for manifests. Relative
+paths start from project's ".repo/manifests" folder.
+
+The --raw option Displays the diff in a way that facilitates parsing, the
+project pattern will be <status> <path> <revision from> [<revision to>] and the
+commit pattern will be <status> <onelined log> with status values respectively :
+
+  A = Added project
+  R = Removed project
+  C = Changed project
+  U = Project with unreachable revision(s) (revision(s) not found)
+
+for project, and
+
+   A = Added commit
+   R = Removed commit
+
+for a commit.
+
+Only changed projects may contain commits, and commit status always starts with
+a space, and are part of last printed project.
+Unreachable revisions may occur if project is not up to date or if repo has not
+been initialized with all the groups, in which case some projects won't be
+synced and their revisions won't be found.
+
+"""
+
+  def _Options(self, p):
+    p.add_option('--raw',
+                 dest='raw', action='store_true',
+                 help='Display raw diff.')
+    p.add_option('--no-color',
+                 dest='color', action='store_false', default=True,
+                 help='does not display the diff in color.')
+    p.add_option('--pretty-format',
+                 dest='pretty_format', action='store',
+                 metavar='<FORMAT>',
+                 help='print the log using a custom git pretty format string')
+
+  def _printRawDiff(self, diff):
+    for project in diff['added']:
+      self.printText("A %s %s" % (project.relpath, project.revisionExpr))
+      self.out.nl()
+
+    for project in diff['removed']:
+      self.printText("R %s %s" % (project.relpath, project.revisionExpr))
+      self.out.nl()
+
+    for project, otherProject in diff['changed']:
+      self.printText("C %s %s %s" % (project.relpath, project.revisionExpr,
+                                     otherProject.revisionExpr))
+      self.out.nl()
+      self._printLogs(project, otherProject, raw=True, color=False)
+
+    for project, otherProject in diff['unreachable']:
+      self.printText("U %s %s %s" % (project.relpath, project.revisionExpr,
+                                     otherProject.revisionExpr))
+      self.out.nl()
+
+  def _printDiff(self, diff, color=True, pretty_format=None):
+    if diff['added']:
+      self.out.nl()
+      self.printText('added projects : \n')
+      self.out.nl()
+      for project in diff['added']:
+        self.printProject('\t%s' % (project.relpath))
+        self.printText(' at revision ')
+        self.printRevision(project.revisionExpr)
+        self.out.nl()
+
+    if diff['removed']:
+      self.out.nl()
+      self.printText('removed projects : \n')
+      self.out.nl()
+      for project in diff['removed']:
+        self.printProject('\t%s' % (project.relpath))
+        self.printText(' at revision ')
+        self.printRevision(project.revisionExpr)
+        self.out.nl()
+
+    if diff['changed']:
+      self.out.nl()
+      self.printText('changed projects : \n')
+      self.out.nl()
+      for project, otherProject in diff['changed']:
+        self.printProject('\t%s' % (project.relpath))
+        self.printText(' changed from ')
+        self.printRevision(project.revisionExpr)
+        self.printText(' to ')
+        self.printRevision(otherProject.revisionExpr)
+        self.out.nl()
+        self._printLogs(project, otherProject, raw=False, color=color,
+                        pretty_format=pretty_format)
+        self.out.nl()
+
+    if diff['unreachable']:
+      self.out.nl()
+      self.printText('projects with unreachable revisions : \n')
+      self.out.nl()
+      for project, otherProject in diff['unreachable']:
+        self.printProject('\t%s ' % (project.relpath))
+        self.printRevision(project.revisionExpr)
+        self.printText(' or ')
+        self.printRevision(otherProject.revisionExpr)
+        self.printText(' not found')
+        self.out.nl()
+
+  def _printLogs(self, project, otherProject, raw=False, color=True,
+                 pretty_format=None):
+
+    logs = project.getAddedAndRemovedLogs(otherProject,
+                                          oneline=(pretty_format is None),
+                                          color=color,
+                                          pretty_format=pretty_format)
+    if logs['removed']:
+      removedLogs = logs['removed'].split('\n')
+      for log in removedLogs:
+        if log.strip():
+          if raw:
+            self.printText(' R ' + log)
+            self.out.nl()
+          else:
+            self.printRemoved('\t\t[-] ')
+            self.printText(log)
+            self.out.nl()
+
+    if logs['added']:
+      addedLogs = logs['added'].split('\n')
+      for log in addedLogs:
+        if log.strip():
+          if raw:
+            self.printText(' A ' + log)
+            self.out.nl()
+          else:
+            self.printAdded('\t\t[+] ')
+            self.printText(log)
+            self.out.nl()
+
+  def Execute(self, opt, args):
+    if not args or len(args) > 2:
+      self.Usage()
+
+    self.out = _Coloring(self.manifest.globalConfig)
+    self.printText = self.out.nofmt_printer('text')
+    if opt.color:
+      self.printProject = self.out.nofmt_printer('project', attr = 'bold')
+      self.printAdded = self.out.nofmt_printer('green', fg = 'green', attr = 'bold')
+      self.printRemoved = self.out.nofmt_printer('red', fg = 'red', attr = 'bold')
+      self.printRevision = self.out.nofmt_printer('revision', fg = 'yellow')
+    else:
+      self.printProject = self.printAdded = self.printRemoved = self.printRevision = self.printText
+
+    manifest1 = XmlManifest(self.manifest.repodir)
+    manifest1.Override(args[0])
+    if len(args) == 1:
+      manifest2 = self.manifest
+    else:
+      manifest2 = XmlManifest(self.manifest.repodir)
+      manifest2.Override(args[1])
+
+    diff = manifest1.projectsDiff(manifest2)
+    if opt.raw:
+      self._printRawDiff(diff)
+    else:
+      self._printDiff(diff, color=opt.color, pretty_format=opt.pretty_format)

subcmds/download.py

@@ -13,11 +13,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import os
+from __future__ import print_function
 import re
 import sys
 
 from command import Command
+from error import GitError
 
 CHANGE_RE = re.compile(r'^([1-9][0-9]*)(?:[/\.-]([1-9][0-9]*))?$')
 
@@ -25,17 +26,29 @@ class Download(Command):
   common = True
   helpSummary = "Download and checkout a change"
   helpUsage = """
-%prog {project change[/patchset]}...
+%prog {[project] change[/patchset]}...
 """
   helpDescription = """
 The '%prog' command downloads a change from the review system and
 makes it available in your project's local working directory.
+If no project is specified try to use current directory as a project.
 """
 
   def _Options(self, p):
-    pass
+    p.add_option('-c', '--cherry-pick',
+                 dest='cherrypick', action='store_true',
+                 help="cherry-pick instead of checkout")
+    p.add_option('-r', '--revert',
+                 dest='revert', action='store_true',
+                 help="revert instead of checkout")
+    p.add_option('-f', '--ff-only',
+                 dest='ffonly', action='store_true',
+                 help="force fast-forward merge")
 
   def _ParseChangeIds(self, args):
+    if not args:
+      self.Usage()
+
     to_get = []
     project = None
 
@@ -43,12 +56,21 @@ makes it available in your project's local working directory.
       m = CHANGE_RE.match(a)
       if m:
         if not project:
-          self.Usage()
+          project = self.GetProjects(".")[0]
         chg_id = int(m.group(1))
         if m.group(2):
           ps_id = int(m.group(2))
         else:
           ps_id = 1
+          refs = 'refs/changes/%2.2d/%d/' % (chg_id % 100, chg_id)
+          output = project._LsRemote(refs + '*')
+          if output:
+            regex = refs + r'(\d+)'
+            rcomp = re.compile(regex, re.I)
+            for line in output.splitlines():
+              match = rcomp.search(line)
+              if match:
+                ps_id = max(int(match.group(1)), ps_id)
         to_get.append((project, chg_id, ps_id))
       else:
         project = self.GetProjects([a])[0]
@@ -58,21 +80,34 @@ makes it available in your project's local working directory.
     for project, change_id, ps_id in self._ParseChangeIds(args):
       dl = project.DownloadPatchSet(change_id, ps_id)
       if not dl:
-        print >>sys.stderr, \
-          '[%s] change %d/%d not found' \
-          % (project.name, change_id, ps_id)
+        print('[%s] change %d/%d not found'
+              % (project.name, change_id, ps_id),
+              file=sys.stderr)
         sys.exit(1)
 
-      if not dl.commits:
-        print >>sys.stderr, \
-          '[%s] change %d/%d has already been merged' \
-          % (project.name, change_id, ps_id)
+      if not opt.revert and not dl.commits:
+        print('[%s] change %d/%d has already been merged'
+              % (project.name, change_id, ps_id),
+              file=sys.stderr)
         continue
 
       if len(dl.commits) > 1:
-        print >>sys.stderr, \
-          '[%s] %d/%d depends on %d unmerged changes:' \
-          % (project.name, change_id, ps_id, len(dl.commits))
+        print('[%s] %d/%d depends on %d unmerged changes:' \
+              % (project.name, change_id, ps_id, len(dl.commits)),
+              file=sys.stderr)
         for c in dl.commits:
-          print >>sys.stderr, '  %s' % (c)
-      project._Checkout(dl.commit)
+          print('  %s' % (c), file=sys.stderr)
+      if opt.cherrypick:
+        try:
+          project._CherryPick(dl.commit)
+        except GitError:
+          print('[%s] Could not complete the cherry-pick of %s' \
+                % (project.name, dl.commit), file=sys.stderr)
+          sys.exit(1)
+
+      elif opt.revert:
+        project._Revert(dl.commit)
+      elif opt.ffonly:
+        project._FastForward(dl.commit, ffonly=True)
+      else:
+        project._Checkout(dl.commit)

subcmds/forall.py

@@ -13,32 +13,107 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
+import errno
+import multiprocessing
 import re
 import os
+import signal
 import sys
 import subprocess
-from command import Command
 
-class Forall(Command):
+from color import Coloring
+from command import Command, MirrorSafeCommand
+import platform_utils
+
+_CAN_COLOR = [
+  'branch',
+  'diff',
+  'grep',
+  'log',
+]
+
+
+class ForallColoring(Coloring):
+  def __init__(self, config):
+    Coloring.__init__(self, config, 'forall')
+    self.project = self.printer('project', attr='bold')
+
+
+class Forall(Command, MirrorSafeCommand):
   common = False
   helpSummary = "Run a shell command in each project"
   helpUsage = """
 %prog [<project>...] -c <command> [<arg>...]
+%prog -r str1 [str2] ... -c <command> [<arg>...]"
 """
   helpDescription = """
 Executes the same shell command in each project.
 
-Environment
------------
-pwd is the project's working directory.
+The -r option allows running the command only on projects matching
+regex or wildcard expression.
+
+# Output Formatting
+
+The -p option causes '%prog' to bind pipes to the command's stdin,
+stdout and stderr streams, and pipe all output into a continuous
+stream that is displayed in a single pager session.  Project headings
+are inserted before the output of each command is displayed.  If the
+command produces no output in a project, no heading is displayed.
+
+The formatting convention used by -p is very suitable for some
+types of searching, e.g. `repo forall -p -c git log -SFoo` will
+print all commits that add or remove references to Foo.
+
+The -v option causes '%prog' to display stderr messages if a
+command produces output only on stderr.  Normally the -p option
+causes command output to be suppressed until the command produces
+at least one byte of output on stdout.
+
+# Environment
+
+pwd is the project's working directory.  If the current client is
+a mirror client, then pwd is the Git repository.
 
 REPO_PROJECT is set to the unique name of the project.
 
+REPO_PATH is the path relative the the root of the client.
+
+REPO_REMOTE is the name of the remote system from the manifest.
+
+REPO_LREV is the name of the revision from the manifest, translated
+to a local tracking branch.  If you need to pass the manifest
+revision to a locally executed git command, use REPO_LREV.
+
+REPO_RREV is the name of the revision from the manifest, exactly
+as written in the manifest.
+
+REPO_COUNT is the total number of projects being iterated.
+
+REPO_I is the current (1-based) iteration count. Can be used in
+conjunction with REPO_COUNT to add a simple progress indicator to your
+command.
+
+REPO__* are any extra environment variables, specified by the
+"annotation" element under any project element.  This can be useful
+for differentiating trees based on user-specific criteria, or simply
+annotating tree details.
+
 shell positional arguments ($1, $2, .., $#) are set to any arguments
 following <command>.
 
-stdin, stdout, stderr are inherited from the terminal and are
-not redirected.
+Example: to list projects:
+
+  %prog% forall -c 'echo $REPO_PROJECT'
+
+Notice that $REPO_PROJECT is quoted to ensure it is expanded in
+the context of running <command> instead of in the calling shell.
+
+Unless -p is used, stdin, stdout, stderr are inherited from the
+terminal and are not redirected.
+
+If -e is used, when a command exits unsuccessfully, '%prog' will abort
+without iterating through the remaining projects.
 """
 
   def _Options(self, p):
@@ -46,11 +121,60 @@ not redirected.
       setattr(parser.values, option.dest, list(parser.rargs))
       while parser.rargs:
         del parser.rargs[0]
+    p.add_option('-r', '--regex',
+                 dest='regex', action='store_true',
+                 help="Execute the command only on projects matching regex or wildcard expression")
+    p.add_option('-i', '--inverse-regex',
+                 dest='inverse_regex', action='store_true',
+                 help="Execute the command only on projects not matching regex or wildcard expression")
+    p.add_option('-g', '--groups',
+                 dest='groups',
+                 help="Execute the command only on projects matching the specified groups")
     p.add_option('-c', '--command',
                  help='Command (and arguments) to execute',
                  dest='command',
                  action='callback',
                  callback=cmd)
+    p.add_option('-e', '--abort-on-errors',
+                 dest='abort_on_errors', action='store_true',
+                 help='Abort if a command exits unsuccessfully')
+
+    g = p.add_option_group('Output')
+    g.add_option('-p',
+                 dest='project_header', action='store_true',
+                 help='Show project headers before output')
+    g.add_option('-v', '--verbose',
+                 dest='verbose', action='store_true',
+                 help='Show command error messages')
+    g.add_option('-j', '--jobs',
+                 dest='jobs', action='store', type='int', default=1,
+                 help='number of commands to execute simultaneously')
+
+  def WantPager(self, opt):
+    return opt.project_header and opt.jobs == 1
+
+  def _SerializeProject(self, project):
+    """ Serialize a project._GitGetByExec instance.
+
+    project._GitGetByExec is not pickle-able. Instead of trying to pass it
+    around between processes, make a dict ourselves containing only the
+    attributes that we need.
+
+    """
+    if not self.manifest.IsMirror:
+      lrev = project.GetRevisionId()
+    else:
+      lrev = None
+    return {
+      'name': project.name,
+      'relpath': project.relpath,
+      'remote_name': project.remote.name,
+      'lrev': lrev,
+      'rrev': project.revisionExpr,
+      'annotations': dict((a.name, a.value) for a in project.annotations),
+      'gitdir': project.gitdir,
+      'worktree': project.worktree,
+    }
 
   def Execute(self, opt, args):
     if not opt.command:
@@ -66,17 +190,203 @@ not redirected.
       cmd.append(cmd[0])
     cmd.extend(opt.command[1:])
 
-    rc = 0
-    for project in self.GetProjects(args):
-      env = dict(os.environ.iteritems())
-      env['REPO_PROJECT'] = project.name
+    if  opt.project_header \
+    and not shell \
+    and cmd[0] == 'git':
+      # If this is a direct git command that can enable colorized
+      # output and the user prefers coloring, add --color into the
+      # command line because we are going to wrap the command into
+      # a pipe and git won't know coloring should activate.
+      #
+      for cn in cmd[1:]:
+        if not cn.startswith('-'):
+          break
+      else:
+        cn = None
+      if cn and cn in _CAN_COLOR:
+        class ColorCmd(Coloring):
+          def __init__(self, config, cmd):
+            Coloring.__init__(self, config, cmd)
+        if ColorCmd(self.manifest.manifestProject.config, cn).is_on:
+          cmd.insert(cmd.index(cn) + 1, '--color')
 
-      p = subprocess.Popen(cmd,
-                           cwd = project.worktree,
-                           shell = shell,
-                           env = env)
-      r = p.wait()
-      if r != 0 and r != rc:
-        rc = r
+    mirror = self.manifest.IsMirror
+    rc = 0
+
+    smart_sync_manifest_name = "smart_sync_override.xml"
+    smart_sync_manifest_path = os.path.join(
+      self.manifest.manifestProject.worktree, smart_sync_manifest_name)
+
+    if os.path.isfile(smart_sync_manifest_path):
+      self.manifest.Override(smart_sync_manifest_path)
+
+    if opt.regex:
+      projects = self.FindProjects(args)
+    elif opt.inverse_regex:
+      projects = self.FindProjects(args, inverse=True)
+    else:
+      projects = self.GetProjects(args, groups=opt.groups)
+
+    os.environ['REPO_COUNT'] = str(len(projects))
+
+    pool = multiprocessing.Pool(opt.jobs, InitWorker)
+    try:
+      config = self.manifest.manifestProject.config
+      results_it = pool.imap(
+         DoWorkWrapper,
+         self.ProjectArgs(projects, mirror, opt, cmd, shell, config))
+      pool.close()
+      for r in results_it:
+        rc = rc or r
+        if r != 0 and opt.abort_on_errors:
+          raise Exception('Aborting due to previous error')
+    except (KeyboardInterrupt, WorkerKeyboardInterrupt):
+      # Catch KeyboardInterrupt raised inside and outside of workers
+      print('Interrupted - terminating the pool')
+      pool.terminate()
+      rc = rc or errno.EINTR
+    except Exception as e:
+      # Catch any other exceptions raised
+      print('Got an error, terminating the pool: %s: %s' %
+              (type(e).__name__, e),
+            file=sys.stderr)
+      pool.terminate()
+      rc = rc or getattr(e, 'errno', 1)
+    finally:
+      pool.join()
     if rc != 0:
       sys.exit(rc)
+
+  def ProjectArgs(self, projects, mirror, opt, cmd, shell, config):
+    for cnt, p in enumerate(projects):
+      try:
+        project = self._SerializeProject(p)
+      except Exception as e:
+        print('Project list error on project %s: %s: %s' %
+                (p.name, type(e).__name__, e),
+              file=sys.stderr)
+        return
+      except KeyboardInterrupt:
+        print('Project list interrupted',
+              file=sys.stderr)
+        return
+      yield [mirror, opt, cmd, shell, cnt, config, project]
+
+class WorkerKeyboardInterrupt(Exception):
+  """ Keyboard interrupt exception for worker processes. """
+  pass
+
+
+def InitWorker():
+  signal.signal(signal.SIGINT, signal.SIG_IGN)
+
+def DoWorkWrapper(args):
+  """ A wrapper around the DoWork() method.
+
+  Catch the KeyboardInterrupt exceptions here and re-raise them as a different,
+  ``Exception``-based exception to stop it flooding the console with stacktraces
+  and making the parent hang indefinitely.
+
+  """
+  project = args.pop()
+  try:
+    return DoWork(project, *args)
+  except KeyboardInterrupt:
+    print('%s: Worker interrupted' % project['name'])
+    raise WorkerKeyboardInterrupt()
+
+
+def DoWork(project, mirror, opt, cmd, shell, cnt, config):
+  env = os.environ.copy()
+  def setenv(name, val):
+    if val is None:
+      val = ''
+    if hasattr(val, 'encode'):
+      val = val.encode()
+    env[name] = val
+
+  setenv('REPO_PROJECT', project['name'])
+  setenv('REPO_PATH', project['relpath'])
+  setenv('REPO_REMOTE', project['remote_name'])
+  setenv('REPO_LREV', project['lrev'])
+  setenv('REPO_RREV', project['rrev'])
+  setenv('REPO_I', str(cnt + 1))
+  for name in project['annotations']:
+    setenv("REPO__%s" % (name), project['annotations'][name])
+
+  if mirror:
+    setenv('GIT_DIR', project['gitdir'])
+    cwd = project['gitdir']
+  else:
+    cwd = project['worktree']
+
+  if not os.path.exists(cwd):
+    if (opt.project_header and opt.verbose) \
+    or not opt.project_header:
+      print('skipping %s/' % project['relpath'], file=sys.stderr)
+    return
+
+  if opt.project_header:
+    stdin = subprocess.PIPE
+    stdout = subprocess.PIPE
+    stderr = subprocess.PIPE
+  else:
+    stdin = None
+    stdout = None
+    stderr = None
+
+  p = subprocess.Popen(cmd,
+                       cwd=cwd,
+                       shell=shell,
+                       env=env,
+                       stdin=stdin,
+                       stdout=stdout,
+                       stderr=stderr)
+
+  if opt.project_header:
+    out = ForallColoring(config)
+    out.redirect(sys.stdout)
+    empty = True
+    errbuf = ''
+
+    p.stdin.close()
+    s_in = platform_utils.FileDescriptorStreams.create()
+    s_in.add(p.stdout, sys.stdout, 'stdout')
+    s_in.add(p.stderr, sys.stderr, 'stderr')
+
+    while not s_in.is_done:
+      in_ready = s_in.select()
+      for s in in_ready:
+        buf = s.read()
+        if not buf:
+          s.close()
+          s_in.remove(s)
+          continue
+
+        if not opt.verbose:
+          if s.std_name == 'stderr':
+            errbuf += buf
+            continue
+
+        if empty and out:
+          if not cnt == 0:
+            out.nl()
+
+          if mirror:
+            project_header_path = project['name']
+          else:
+            project_header_path = project['relpath']
+          out.project('project %s/', project_header_path)
+          out.nl()
+          out.flush()
+          if errbuf:
+            sys.stderr.write(errbuf)
+            sys.stderr.flush()
+            errbuf = ''
+          empty = False
+
+        s.dest.write(buf)
+        s.dest.flush()
+
+  r = p.wait()
+  return r

subcmds/gitc_delete.py

@@ -0,0 +1,51 @@
+#
+# Copyright (C) 2015 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import sys
+
+from command import Command, GitcClientCommand
+import platform_utils
+
+from pyversion import is_python3
+if not is_python3():
+  input = raw_input
+
+class GitcDelete(Command, GitcClientCommand):
+  common = True
+  visible_everywhere = False
+  helpSummary = "Delete a GITC Client."
+  helpUsage = """
+%prog
+"""
+  helpDescription = """
+This subcommand deletes the current GITC client, deleting the GITC manifest
+and all locally downloaded sources.
+"""
+
+  def _Options(self, p):
+    p.add_option('-f', '--force',
+                 dest='force', action='store_true',
+                 help='Force the deletion (no prompt).')
+
+  def Execute(self, opt, args):
+    if not opt.force:
+      prompt = ('This will delete GITC client: %s\nAre you sure? (yes/no) ' %
+                self.gitc_manifest.gitc_client_name)
+      response = input(prompt).lower()
+      if not response == 'yes':
+        print('Response was not "yes"\n Exiting...')
+        sys.exit(1)
+    platform_utils.rmtree(self.gitc_manifest.gitc_client_dir)

subcmds/gitc_init.py

@@ -0,0 +1,82 @@
+#
+# Copyright (C) 2015 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import os
+import sys
+
+import gitc_utils
+from command import GitcAvailableCommand
+from manifest_xml import GitcManifest
+from subcmds import init
+import wrapper
+
+
+class GitcInit(init.Init, GitcAvailableCommand):
+  common = True
+  helpSummary = "Initialize a GITC Client."
+  helpUsage = """
+%prog [options] [client name]
+"""
+  helpDescription = """
+The '%prog' command is ran to initialize a new GITC client for use
+with the GITC file system.
+
+This command will setup the client directory, initialize repo, just
+like repo init does, and then downloads the manifest collection
+and installs it in the .repo/directory of the GITC client.
+
+Once this is done, a GITC manifest is generated by pulling the HEAD
+SHA for each project and generates the properly formatted XML file
+and installs it as .manifest in the GITC client directory.
+
+The -c argument is required to specify the GITC client name.
+
+The optional -f argument can be used to specify the manifest file to
+use for this GITC client.
+"""
+
+  def _Options(self, p):
+    super(GitcInit, self)._Options(p)
+    g = p.add_option_group('GITC options')
+    g.add_option('-f', '--manifest-file',
+                 dest='manifest_file',
+                 help='Optional manifest file to use for this GITC client.')
+    g.add_option('-c', '--gitc-client',
+                 dest='gitc_client',
+                 help='The name of the gitc_client instance to create or modify.')
+
+  def Execute(self, opt, args):
+    gitc_client = gitc_utils.parse_clientdir(os.getcwd())
+    if not gitc_client or (opt.gitc_client and gitc_client != opt.gitc_client):
+      print('fatal: Please update your repo command. See go/gitc for instructions.', file=sys.stderr)
+      sys.exit(1)
+    self.client_dir = os.path.join(gitc_utils.get_gitc_manifest_dir(),
+                                   gitc_client)
+    super(GitcInit, self).Execute(opt, args)
+
+    manifest_file = self.manifest.manifestFile
+    if opt.manifest_file:
+      if not os.path.exists(opt.manifest_file):
+        print('fatal: Specified manifest file %s does not exist.' %
+              opt.manifest_file)
+        sys.exit(1)
+      manifest_file = opt.manifest_file
+
+    manifest = GitcManifest(self.repodir, gitc_client)
+    manifest.Override(manifest_file)
+    gitc_utils.generate_gitc_manifest(None, manifest)
+    print('Please run `cd %s` to view your GITC client.' %
+          os.path.join(wrapper.Wrapper().GITC_FS_ROOT_DIR, gitc_client))

subcmds/grep.py

@@ -0,0 +1,240 @@
+#
+# Copyright (C) 2009 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import sys
+from color import Coloring
+from command import PagedCommand
+from git_command import git_require, GitCommand
+
+class GrepColoring(Coloring):
+  def __init__(self, config):
+    Coloring.__init__(self, config, 'grep')
+    self.project = self.printer('project', attr='bold')
+
+class Grep(PagedCommand):
+  common = True
+  helpSummary = "Print lines matching a pattern"
+  helpUsage = """
+%prog {pattern | -e pattern} [<project>...]
+"""
+  helpDescription = """
+Search for the specified patterns in all project files.
+
+# Boolean Options
+
+The following options can appear as often as necessary to express
+the pattern to locate:
+
+ -e PATTERN
+ --and, --or, --not, -(, -)
+
+Further, the -r/--revision option may be specified multiple times
+in order to scan multiple trees.  If the same file matches in more
+than one tree, only the first result is reported, prefixed by the
+revision name it was found under.
+
+# Examples
+
+Look for a line that has '#define' and either 'MAX_PATH or 'PATH_MAX':
+
+  repo grep -e '#define' --and -\\( -e MAX_PATH -e PATH_MAX \\)
+
+Look for a line that has 'NODE' or 'Unexpected' in files that
+contain a line that matches both expressions:
+
+  repo grep --all-match -e NODE -e Unexpected
+
+"""
+
+  def _Options(self, p):
+    def carry(option,
+              opt_str,
+              value,
+              parser):
+      pt = getattr(parser.values, 'cmd_argv', None)
+      if pt is None:
+        pt = []
+        setattr(parser.values, 'cmd_argv', pt)
+
+      if opt_str == '-(':
+        pt.append('(')
+      elif opt_str == '-)':
+        pt.append(')')
+      else:
+        pt.append(opt_str)
+
+      if value is not None:
+        pt.append(value)
+
+    g = p.add_option_group('Sources')
+    g.add_option('--cached',
+                 action='callback', callback=carry,
+                 help='Search the index, instead of the work tree')
+    g.add_option('-r', '--revision',
+                 dest='revision', action='append', metavar='TREEish',
+                 help='Search TREEish, instead of the work tree')
+
+    g = p.add_option_group('Pattern')
+    g.add_option('-e',
+                 action='callback', callback=carry,
+                 metavar='PATTERN', type='str',
+                 help='Pattern to search for')
+    g.add_option('-i', '--ignore-case',
+                 action='callback', callback=carry,
+                 help='Ignore case differences')
+    g.add_option('-a', '--text',
+                 action='callback', callback=carry,
+                 help="Process binary files as if they were text")
+    g.add_option('-I',
+                 action='callback', callback=carry,
+                 help="Don't match the pattern in binary files")
+    g.add_option('-w', '--word-regexp',
+                 action='callback', callback=carry,
+                 help='Match the pattern only at word boundaries')
+    g.add_option('-v', '--invert-match',
+                 action='callback', callback=carry,
+                 help='Select non-matching lines')
+    g.add_option('-G', '--basic-regexp',
+                 action='callback', callback=carry,
+                 help='Use POSIX basic regexp for patterns (default)')
+    g.add_option('-E', '--extended-regexp',
+                 action='callback', callback=carry,
+                 help='Use POSIX extended regexp for patterns')
+    g.add_option('-F', '--fixed-strings',
+                 action='callback', callback=carry,
+                 help='Use fixed strings (not regexp) for pattern')
+
+    g = p.add_option_group('Pattern Grouping')
+    g.add_option('--all-match',
+                 action='callback', callback=carry,
+                 help='Limit match to lines that have all patterns')
+    g.add_option('--and', '--or', '--not',
+                 action='callback', callback=carry,
+                 help='Boolean operators to combine patterns')
+    g.add_option('-(', '-)',
+                 action='callback', callback=carry,
+                 help='Boolean operator grouping')
+
+    g = p.add_option_group('Output')
+    g.add_option('-n',
+                 action='callback', callback=carry,
+                 help='Prefix the line number to matching lines')
+    g.add_option('-C',
+                 action='callback', callback=carry,
+                 metavar='CONTEXT', type='str',
+                 help='Show CONTEXT lines around match')
+    g.add_option('-B',
+                 action='callback', callback=carry,
+                 metavar='CONTEXT', type='str',
+                 help='Show CONTEXT lines before match')
+    g.add_option('-A',
+                 action='callback', callback=carry,
+                 metavar='CONTEXT', type='str',
+                 help='Show CONTEXT lines after match')
+    g.add_option('-l', '--name-only', '--files-with-matches',
+                 action='callback', callback=carry,
+                 help='Show only file names containing matching lines')
+    g.add_option('-L', '--files-without-match',
+                 action='callback', callback=carry,
+                 help='Show only file names not containing matching lines')
+
+
+  def Execute(self, opt, args):
+    out = GrepColoring(self.manifest.manifestProject.config)
+
+    cmd_argv = ['grep']
+    if out.is_on and git_require((1, 6, 3)):
+      cmd_argv.append('--color')
+    cmd_argv.extend(getattr(opt, 'cmd_argv', []))
+
+    if '-e' not in cmd_argv:
+      if not args:
+        self.Usage()
+      cmd_argv.append('-e')
+      cmd_argv.append(args[0])
+      args = args[1:]
+
+    projects = self.GetProjects(args)
+
+    full_name = False
+    if len(projects) > 1:
+      cmd_argv.append('--full-name')
+      full_name = True
+
+    have_rev = False
+    if opt.revision:
+      if '--cached' in cmd_argv:
+        print('fatal: cannot combine --cached and --revision', file=sys.stderr)
+        sys.exit(1)
+      have_rev = True
+      cmd_argv.extend(opt.revision)
+    cmd_argv.append('--')
+
+    bad_rev = False
+    have_match = False
+
+    for project in projects:
+      p = GitCommand(project,
+                     cmd_argv,
+                     bare = False,
+                     capture_stdout = True,
+                     capture_stderr = True)
+      if p.Wait() != 0:
+        # no results
+        #
+        if p.stderr:
+          if have_rev and 'fatal: ambiguous argument' in p.stderr:
+            bad_rev = True
+          else:
+            out.project('--- project %s ---' % project.relpath)
+            out.nl()
+            out.write("%s", p.stderr)
+            out.nl()
+        continue
+      have_match = True
+
+      # We cut the last element, to avoid a blank line.
+      #
+      r = p.stdout.split('\n')
+      r = r[0:-1]
+
+      if have_rev and full_name:
+        for line in r:
+          rev, line = line.split(':', 1)
+          out.write("%s", rev)
+          out.write(':')
+          out.project(project.relpath)
+          out.write('/')
+          out.write("%s", line)
+          out.nl()
+      elif full_name:
+        for line in r:
+          out.project(project.relpath)
+          out.write('/')
+          out.write("%s", line)
+          out.nl()
+      else:
+        for line in r:
+          print(line)
+
+    if have_match:
+      sys.exit(0)
+    elif have_rev and bad_rev:
+      for r in opt.revision:
+        print("error: can't search revision %s" % r, file=sys.stderr)
+      sys.exit(1)
+    else:
+      sys.exit(1)

subcmds/help.py

@@ -13,13 +13,16 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
+import re
 import sys
 from formatter import AbstractFormatter, DumbWriter
 
 from color import Coloring
-from command import PagedCommand
+from command import PagedCommand, MirrorSafeCommand, GitcAvailableCommand, GitcClientCommand
+import gitc_utils
 
-class Help(PagedCommand):
+class Help(PagedCommand, MirrorSafeCommand):
   common = False
   helpSummary = "Display detailed help on a command"
   helpUsage = """
@@ -30,12 +33,9 @@ Displays detailed usage information about a command.
 """
 
   def _PrintAllCommands(self):
-    print 'usage: repo COMMAND [ARGS]'
-    print """
-The complete list of recognized repo commands are:
-"""
-    commandNames = self.commands.keys()
-    commandNames.sort()
+    print('usage: repo COMMAND [ARGS]')
+    print('The complete list of recognized repo commands are:')
+    commandNames = list(sorted(self.commands))
 
     maxlen = 0
     for name in commandNames:
@@ -48,20 +48,28 @@ The complete list of recognized repo commands are:
         summary = command.helpSummary.strip()
       except AttributeError:
         summary = ''
-      print fmt % (name, summary)
-    print """
-See 'repo help <command>' for more information on a specific command.
-"""
+      print(fmt % (name, summary))
+    print("See 'repo help <command>' for more information on a "
+          'specific command.')
 
   def _PrintCommonCommands(self):
-    print 'usage: repo COMMAND [ARGS]'
-    print """
-The most commonly used repo commands are:
-"""
-    commandNames = [name 
-                    for name in self.commands.keys()
-                    if self.commands[name].common]
-    commandNames.sort()
+    print('usage: repo COMMAND [ARGS]')
+    print('The most commonly used repo commands are:')
+
+    def gitc_supported(cmd):
+      if not isinstance(cmd, GitcAvailableCommand) and not isinstance(cmd, GitcClientCommand):
+        return True
+      if self.manifest.isGitcClient:
+        return True
+      if isinstance(cmd, GitcClientCommand):
+        return False
+      if gitc_utils.get_gitc_manifest_dir():
+        return True
+      return False
+
+    commandNames = list(sorted([name
+                    for name, command in self.commands.items()
+                    if command.common and gitc_supported(command)]))
 
     maxlen = 0
     for name in commandNames:
@@ -74,10 +82,10 @@ The most commonly used repo commands are:
         summary = command.helpSummary.strip()
       except AttributeError:
         summary = ''
-      print fmt % (name, summary)
-    print """
-See 'repo help <command>' for more information on a specific command.
-"""
+      print(fmt % (name, summary))
+    print(
+"See 'repo help <command>' for more information on a specific command.\n"
+"See 'repo help --all' for a complete list of recognized commands.")
 
   def _PrintCommandHelp(self, cmd):
     class _Out(Coloring):
@@ -92,32 +100,41 @@ See 'repo help <command>' for more information on a specific command.
           body = getattr(cmd, bodyAttr)
         except AttributeError:
           return
+        if body == '' or body is None:
+          return
 
         self.nl()
 
         self.heading('%s', heading)
         self.nl()
-
-        self.heading('%s', ''.ljust(len(heading), '-'))
         self.nl()
 
         me = 'repo %s' % cmd.NAME
         body = body.strip()
         body = body.replace('%prog', me)
 
+        asciidoc_hdr = re.compile(r'^\n?#+ (.+)$')
         for para in body.split("\n\n"):
           if para.startswith(' '):
             self.write('%s', para)
             self.nl()
             self.nl()
-          else:
-            self.wrap.add_flowing_data(para)
-            self.wrap.end_paragraph(1)
+            continue
+
+          m = asciidoc_hdr.match(para)
+          if m:
+            self.heading(m.group(1))
+            self.nl()
+            self.nl()
+            continue
+
+          self.wrap.add_flowing_data(para)
+          self.wrap.end_paragraph(1)
         self.wrap.end_paragraph(0)
 
     out = _Out(self.manifest.globalConfig)
-    cmd.OptionParser.print_help()
     out._PrintSection('Summary', 'helpSummary')
+    cmd.OptionParser.print_help()
     out._PrintSection('Description', 'helpDescription')
 
   def _Options(self, p):
@@ -138,9 +155,10 @@ See 'repo help <command>' for more information on a specific command.
       try:
         cmd = self.commands[name]
       except KeyError:
-        print >>sys.stderr, "repo: '%s' is not a repo command." % name
+        print("repo: '%s' is not a repo command." % name, file=sys.stderr)
         sys.exit(1)
 
+      cmd.manifest = self.manifest
       self._PrintCommandHelp(cmd)
 
     else:

subcmds/info.py

@@ -0,0 +1,202 @@
+#
+# Copyright (C) 2012 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from command import PagedCommand
+from color import Coloring
+from error import NoSuchProjectError
+from git_refs import R_M
+
+class _Coloring(Coloring):
+  def __init__(self, config):
+    Coloring.__init__(self, config, "status")
+
+class Info(PagedCommand):
+  common = True
+  helpSummary = "Get info on the manifest branch, current branch or unmerged branches"
+  helpUsage = "%prog [-dl] [-o [-b]] [<project>...]"
+
+  def _Options(self, p):
+    p.add_option('-d', '--diff',
+                 dest='all', action='store_true',
+                 help="show full info and commit diff including remote branches")
+    p.add_option('-o', '--overview',
+                 dest='overview', action='store_true',
+                 help='show overview of all local commits')
+    p.add_option('-b', '--current-branch',
+                 dest="current_branch", action="store_true",
+                 help="consider only checked out branches")
+    p.add_option('-l', '--local-only',
+                 dest="local", action="store_true",
+                 help="Disable all remote operations")
+
+
+  def Execute(self, opt, args):
+    self.out = _Coloring(self.manifest.globalConfig)
+    self.heading = self.out.printer('heading', attr = 'bold')
+    self.headtext = self.out.printer('headtext', fg = 'yellow')
+    self.redtext = self.out.printer('redtext', fg = 'red')
+    self.sha = self.out.printer("sha", fg = 'yellow')
+    self.text = self.out.nofmt_printer('text')
+    self.dimtext = self.out.printer('dimtext', attr = 'dim')
+
+    self.opt = opt
+
+    manifestConfig = self.manifest.manifestProject.config
+    mergeBranch = manifestConfig.GetBranch("default").merge
+    manifestGroups = (manifestConfig.GetString('manifest.groups')
+                      or 'all,-notdefault')
+
+    self.heading("Manifest branch: ")
+    if self.manifest.default.revisionExpr:
+        self.headtext(self.manifest.default.revisionExpr)
+    self.out.nl()
+    self.heading("Manifest merge branch: ")
+    self.headtext(mergeBranch)
+    self.out.nl()
+    self.heading("Manifest groups: ")
+    self.headtext(manifestGroups)
+    self.out.nl()
+
+    self.printSeparator()
+
+    if not opt.overview:
+      self.printDiffInfo(args)
+    else:
+      self.printCommitOverview(args)
+
+  def printSeparator(self):
+    self.text("----------------------------")
+    self.out.nl()
+
+  def printDiffInfo(self, args):
+    try:
+      projs = self.GetProjects(args)
+    except NoSuchProjectError:
+      return
+
+    for p in projs:
+      self.heading("Project: ")
+      self.headtext(p.name)
+      self.out.nl()
+
+      self.heading("Mount path: ")
+      self.headtext(p.worktree)
+      self.out.nl()
+
+      self.heading("Current revision: ")
+      self.headtext(p.revisionExpr)
+      self.out.nl()
+
+      localBranches = p.GetBranches().keys()
+      self.heading("Local Branches: ")
+      self.redtext(str(len(localBranches)))
+      if len(localBranches) > 0:
+        self.text(" [")
+        self.text(", ".join(localBranches))
+        self.text("]")
+      self.out.nl()
+
+      if self.opt.all:
+        self.findRemoteLocalDiff(p)
+
+      self.printSeparator()
+
+  def findRemoteLocalDiff(self, project):
+    #Fetch all the latest commits
+    if not self.opt.local:
+      project.Sync_NetworkHalf(quiet=True, current_branch_only=True)
+
+    logTarget = R_M + self.manifest.manifestProject.config.GetBranch("default").merge
+
+    bareTmp = project.bare_git._bare
+    project.bare_git._bare = False
+    localCommits = project.bare_git.rev_list(
+        '--abbrev=8',
+        '--abbrev-commit',
+        '--pretty=oneline',
+        logTarget + "..",
+        '--')
+
+    originCommits = project.bare_git.rev_list(
+        '--abbrev=8',
+        '--abbrev-commit',
+        '--pretty=oneline',
+        ".." + logTarget,
+        '--')
+    project.bare_git._bare = bareTmp
+
+    self.heading("Local Commits: ")
+    self.redtext(str(len(localCommits)))
+    self.dimtext(" (on current branch)")
+    self.out.nl()
+
+    for c in localCommits:
+      split = c.split()
+      self.sha(split[0] + " ")
+      self.text(" ".join(split[1:]))
+      self.out.nl()
+
+    self.printSeparator()
+
+    self.heading("Remote Commits: ")
+    self.redtext(str(len(originCommits)))
+    self.out.nl()
+
+    for c in originCommits:
+      split = c.split()
+      self.sha(split[0] + " ")
+      self.text(" ".join(split[1:]))
+      self.out.nl()
+
+  def printCommitOverview(self, args):
+    all_branches = []
+    for project in self.GetProjects(args):
+      br = [project.GetUploadableBranch(x)
+            for x in project.GetBranches()]
+      br = [x for x in br if x]
+      if self.opt.current_branch:
+        br = [x for x in br if x.name == project.CurrentBranch]
+      all_branches.extend(br)
+
+    if not all_branches:
+      return
+
+    self.out.nl()
+    self.heading('Projects Overview')
+    project = None
+
+    for branch in all_branches:
+      if project != branch.project:
+        project = branch.project
+        self.out.nl()
+        self.headtext(project.relpath)
+        self.out.nl()
+
+      commits = branch.commits
+      date = branch.date
+      self.text('%s %-33s (%2d commit%s, %s)' % (
+        branch.name == project.CurrentBranch and '*' or ' ',
+        branch.name,
+        len(commits),
+        len(commits) != 1 and 's' or '',
+        date))
+      self.out.nl()
+
+      for commit in commits:
+        split = commit.split()
+        self.text('{0:38}{1} '.format('','-'))
+        self.sha(split[0] + " ")
+        self.text(" ".join(split[1:]))
+        self.out.nl()

subcmds/init.py

@@ -13,16 +13,30 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
 import os
+import platform
+import re
 import sys
 
-from color import Coloring
-from command import InteractiveCommand
-from error import ManifestParseError
-from remote import Remote
-from git_command import git, MIN_GIT_VERSION
+from pyversion import is_python3
+if is_python3():
+  import urllib.parse
+else:
+  import imp
+  import urlparse
+  urllib = imp.new_module('urllib')
+  urllib.parse = urlparse
 
-class Init(InteractiveCommand):
+from color import Coloring
+from command import InteractiveCommand, MirrorSafeCommand
+from error import ManifestParseError
+from project import SyncBuffer
+from git_config import GitConfig
+from git_command import git_require, MIN_GIT_VERSION
+import platform_utils
+
+class Init(InteractiveCommand, MirrorSafeCommand):
   common = True
   helpSummary = "Initialize repo in the current directory"
   helpUsage = """
@@ -34,9 +48,36 @@ The latest repo source code and manifest collection is downloaded
 from the server and is installed in the .repo/ directory in the
 current working directory.
 
-The optional <manifest> argument can be used to specify an alternate
-manifest to be used. If no manifest is specified, the manifest
-default.xml will be used.
+The optional -b argument can be used to select the manifest branch
+to checkout and use.  If no branch is specified, master is assumed.
+
+The optional -m argument can be used to specify an alternate manifest
+to be used. If no manifest is specified, the manifest default.xml
+will be used.
+
+The --reference option can be used to point to a directory that
+has the content of a --mirror sync. This will make the working
+directory use as much data as possible from the local reference
+directory when fetching from the server. This will make the sync
+go a lot faster by reducing data traffic on the network.
+
+The --dissociate option can be used to borrow the objects from
+the directory specified with the --reference option only to reduce
+network transfer, and stop borrowing from them after a first clone
+is made by making necessary local copies of borrowed objects.
+
+The --no-clone-bundle option disables any attempt to use
+$URL/clone.bundle to bootstrap a new Git repository from a
+resumeable bundle file on a content delivery network. This
+may be necessary if there are problems with the local Python
+HTTP client or proxy configuration, but the Git binary works.
+
+# Switching Manifest Branches
+
+To switch to another manifest branch, `repo init -b otherbranch`
+may be used in an existing client.  However, as this only updates the
+manifest, a subsequent `repo sync` (or `repo sync -d`) is necessary
+to update the working directory files.
 """
 
   def _Options(self, p):
@@ -54,16 +95,51 @@ default.xml will be used.
     g.add_option('-b', '--manifest-branch',
                  dest='manifest_branch',
                  help='manifest branch or revision', metavar='REVISION')
+    g.add_option('--current-branch',
+                 dest='current_branch_only', action='store_true',
+                 help='fetch only current manifest branch from server')
     g.add_option('-m', '--manifest-name',
                  dest='manifest_name', default='default.xml',
                  help='initial manifest file', metavar='NAME.xml')
     g.add_option('--mirror',
                  dest='mirror', action='store_true',
-                 help='mirror the forrest')
-
+                 help='create a replica of the remote repositories '
+                      'rather than a client working directory')
+    g.add_option('--reference',
+                 dest='reference',
+                 help='location of mirror directory', metavar='DIR')
+    g.add_option('--dissociate',
+                 dest='dissociate', action='store_true',
+                 help='dissociate from reference mirrors after clone')
+    g.add_option('--depth', type='int', default=None,
+                 dest='depth',
+                 help='create a shallow clone with given depth; see git clone')
+    g.add_option('--archive',
+                 dest='archive', action='store_true',
+                 help='checkout an archive instead of a git repository for '
+                      'each project. See git archive.')
+    g.add_option('--submodules',
+                 dest='submodules', action='store_true',
+                 help='sync any submodules associated with the manifest repo')
+    g.add_option('-g', '--groups',
+                 dest='groups', default='default',
+                 help='restrict manifest projects to ones with specified '
+                      'group(s) [default|all|G1,G2,G3|G4,-G5,-G6]',
+                 metavar='GROUP')
+    g.add_option('-p', '--platform',
+                 dest='platform', default='auto',
+                 help='restrict manifest projects to ones with a specified '
+                      'platform group [auto|all|none|linux|darwin|...]',
+                 metavar='PLATFORM')
+    g.add_option('--no-clone-bundle',
+                 dest='no_clone_bundle', action='store_true',
+                 help='disable use of /clone.bundle on HTTP/HTTPS')
+    g.add_option('--no-tags',
+                 dest='no_tags', action='store_true',
+                 help="don't fetch tags in the manifest")
 
     # Tool
-    g = p.add_option_group('Version options')
+    g = p.add_option_group('repo Version options')
     g.add_option('--repo-url',
                  dest='repo_url',
                  help='repo repository location', metavar='URL')
@@ -74,39 +150,50 @@ default.xml will be used.
                  dest='no_repo_verify', action='store_true',
                  help='do not verify repo source code')
 
-  def _CheckGitVersion(self):
-    ver_str = git.version()
-    if not ver_str.startswith('git version '):
-      print >>sys.stderr, 'error: "%s" unsupported' % ver_str
-      sys.exit(1)
+    # Other
+    g = p.add_option_group('Other options')
+    g.add_option('--config-name',
+                 dest='config_name', action="store_true", default=False,
+                 help='Always prompt for name/e-mail')
 
-    ver_str = ver_str[len('git version '):].strip()
-    ver_act = tuple(map(lambda x: int(x), ver_str.split('.')[0:3]))
-    if ver_act < MIN_GIT_VERSION:
-      need = '.'.join(map(lambda x: str(x), MIN_GIT_VERSION))
-      print >>sys.stderr, 'fatal: git %s or later required' % need
-      sys.exit(1)
+  def _RegisteredEnvironmentOptions(self):
+    return {'REPO_MANIFEST_URL': 'manifest_url',
+            'REPO_MIRROR_LOCATION': 'reference'}
 
   def _SyncManifest(self, opt):
     m = self.manifest.manifestProject
+    is_new = not m.Exists
 
-    if not m.Exists:
+    if is_new:
       if not opt.manifest_url:
-        print >>sys.stderr, 'fatal: manifest url (-u) is required.'
+        print('fatal: manifest url (-u) is required.', file=sys.stderr)
         sys.exit(1)
 
       if not opt.quiet:
-        print >>sys.stderr, 'Getting manifest ...'
-        print >>sys.stderr, '   from %s' % opt.manifest_url
-      m._InitGitDir()
+        print('Get %s' % GitConfig.ForUser().UrlInsteadOf(opt.manifest_url),
+              file=sys.stderr)
+
+      # The manifest project object doesn't keep track of the path on the
+      # server where this git is located, so let's save that here.
+      mirrored_manifest_git = None
+      if opt.reference:
+        manifest_git_path = urllib.parse.urlparse(opt.manifest_url).path[1:]
+        mirrored_manifest_git = os.path.join(opt.reference, manifest_git_path)
+        if not mirrored_manifest_git.endswith(".git"):
+          mirrored_manifest_git += ".git"
+        if not os.path.exists(mirrored_manifest_git):
+          mirrored_manifest_git = os.path.join(opt.reference,
+                                               '.repo/manifests.git')
+
+      m._InitGitDir(mirror_git=mirrored_manifest_git)
 
       if opt.manifest_branch:
-        m.revision = opt.manifest_branch
+        m.revisionExpr = opt.manifest_branch
       else:
-        m.revision = 'refs/heads/master'
+        m.revisionExpr = 'refs/heads/master'
     else:
       if opt.manifest_branch:
-        m.revision = opt.manifest_branch
+        m.revisionExpr = opt.manifest_branch
       else:
         m.PreSync()
 
@@ -116,39 +203,137 @@ default.xml will be used.
       r.ResetFetch()
       r.Save()
 
-    if opt.mirror:
-      m.config.SetString('repo.mirror', 'true')
+    groups = re.split(r'[,\s]+', opt.groups)
+    all_platforms = ['linux', 'darwin', 'windows']
+    platformize = lambda x: 'platform-' + x
+    if opt.platform == 'auto':
+      if (not opt.mirror and
+          not m.config.GetString('repo.mirror') == 'true'):
+        groups.append(platformize(platform.system().lower()))
+    elif opt.platform == 'all':
+      groups.extend(map(platformize, all_platforms))
+    elif opt.platform in all_platforms:
+      groups.append(platformize(opt.platform))
+    elif opt.platform != 'none':
+      print('fatal: invalid platform flag', file=sys.stderr)
+      sys.exit(1)
 
-    m.Sync_NetworkHalf()
-    m.Sync_LocalHalf()
-    m.StartBranch('default')
+    groups = [x for x in groups if x]
+    groupstr = ','.join(groups)
+    if opt.platform == 'auto' and groupstr == 'default,platform-' + platform.system().lower():
+      groupstr = None
+    m.config.SetString('manifest.groups', groupstr)
+
+    if opt.reference:
+      m.config.SetString('repo.reference', opt.reference)
+
+    if opt.dissociate:
+      m.config.SetString('repo.dissociate', 'true')
+
+    if opt.archive:
+      if is_new:
+        m.config.SetString('repo.archive', 'true')
+      else:
+        print('fatal: --archive is only supported when initializing a new '
+              'workspace.', file=sys.stderr)
+        print('Either delete the .repo folder in this workspace, or initialize '
+              'in another location.', file=sys.stderr)
+        sys.exit(1)
+
+    if opt.mirror:
+      if is_new:
+        m.config.SetString('repo.mirror', 'true')
+      else:
+        print('fatal: --mirror is only supported when initializing a new '
+              'workspace.', file=sys.stderr)
+        print('Either delete the .repo folder in this workspace, or initialize '
+              'in another location.', file=sys.stderr)
+        sys.exit(1)
+
+    if opt.submodules:
+      m.config.SetString('repo.submodules', 'true')
+
+    if not m.Sync_NetworkHalf(is_new=is_new, quiet=opt.quiet,
+        clone_bundle=not opt.no_clone_bundle,
+        current_branch_only=opt.current_branch_only,
+        no_tags=opt.no_tags, submodules=opt.submodules):
+      r = m.GetRemote(m.remote.name)
+      print('fatal: cannot obtain manifest %s' % r.url, file=sys.stderr)
+
+      # Better delete the manifest git dir if we created it; otherwise next
+      # time (when user fixes problems) we won't go through the "is_new" logic.
+      if is_new:
+        platform_utils.rmtree(m.gitdir)
+      sys.exit(1)
+
+    if opt.manifest_branch:
+      m.MetaBranchSwitch(submodules=opt.submodules)
+
+    syncbuf = SyncBuffer(m.config)
+    m.Sync_LocalHalf(syncbuf, submodules=opt.submodules)
+    syncbuf.Finish()
+
+    if is_new or m.CurrentBranch is None:
+      if not m.StartBranch('default'):
+        print('fatal: cannot create default in manifest', file=sys.stderr)
+        sys.exit(1)
 
   def _LinkManifest(self, name):
     if not name:
-      print >>sys.stderr, 'fatal: manifest name (-m) is required.'
+      print('fatal: manifest name (-m) is required.', file=sys.stderr)
       sys.exit(1)
 
     try:
       self.manifest.Link(name)
-    except ManifestParseError, e:
-      print >>sys.stderr, "fatal: manifest '%s' not available" % name
-      print >>sys.stderr, 'fatal: %s' % str(e)
+    except ManifestParseError as e:
+      print("fatal: manifest '%s' not available" % name, file=sys.stderr)
+      print('fatal: %s' % str(e), file=sys.stderr)
       sys.exit(1)
 
-  def _PromptKey(self, prompt, key, value):
-    mp = self.manifest.manifestProject
-
+  def _Prompt(self, prompt, value):
     sys.stdout.write('%-10s [%s]: ' % (prompt, value))
     a = sys.stdin.readline().strip()
-    if a != '' and a != value:
-      mp.config.SetString(key, a)
+    if a == '':
+      return value
+    return a
+
+  def _ShouldConfigureUser(self):
+    gc = self.manifest.globalConfig
+    mp = self.manifest.manifestProject
+
+    # If we don't have local settings, get from global.
+    if not mp.config.Has('user.name') or not mp.config.Has('user.email'):
+      if not gc.Has('user.name') or not gc.Has('user.email'):
+        return True
+
+      mp.config.SetString('user.name', gc.GetString('user.name'))
+      mp.config.SetString('user.email', gc.GetString('user.email'))
+
+    print()
+    print('Your identity is: %s <%s>' % (mp.config.GetString('user.name'),
+                                         mp.config.GetString('user.email')))
+    print('If you want to change this, please re-run \'repo init\' with --config-name')
+    return False
 
   def _ConfigureUser(self):
     mp = self.manifest.manifestProject
 
-    print ''
-    self._PromptKey('Your Name', 'user.name', mp.UserName)
-    self._PromptKey('Your Email', 'user.email', mp.UserEmail)
+    while True:
+      print()
+      name  = self._Prompt('Your Name', mp.UserName)
+      email = self._Prompt('Your Email', mp.UserEmail)
+
+      print()
+      print('Your identity is: %s <%s>' % (name, email))
+      sys.stdout.write('is this correct [y/N]? ')
+      a = sys.stdin.readline().strip().lower()
+      if a in ('yes', 'y', 't', 'true'):
+        break
+
+    if name != mp.UserName:
+      mp.config.SetString('user.name', name)
+    if email != mp.UserEmail:
+      mp.config.SetString('user.email', email)
 
   def _HasColorSet(self, gc):
     for n in ['ui', 'diff', 'status']:
@@ -167,39 +352,83 @@ default.xml will be used.
         self._on = True
     out = _Test()
 
-    print ''
-    print "Testing colorized output (for 'repo diff', 'repo status'):"
+    print()
+    print("Testing colorized output (for 'repo diff', 'repo status'):")
 
-    for c in ['black','red','green','yellow','blue','magenta','cyan']:
+    for c in ['black', 'red', 'green', 'yellow', 'blue', 'magenta', 'cyan']:
       out.write(' ')
       out.printer(fg=c)(' %-6s ', c)
     out.write(' ')
     out.printer(fg='white', bg='black')(' %s ' % 'white')
     out.nl()
 
-    for c in ['bold','dim','ul','reverse']:
+    for c in ['bold', 'dim', 'ul', 'reverse']:
       out.write(' ')
       out.printer(fg='black', attr=c)(' %-6s ', c)
     out.nl()
 
-    sys.stdout.write('Enable color display in this user account (y/n)? ')
+    sys.stdout.write('Enable color display in this user account (y/N)? ')
     a = sys.stdin.readline().strip().lower()
     if a in ('y', 'yes', 't', 'true', 'on'):
       gc.SetString('color.ui', 'auto')
 
+  def _ConfigureDepth(self, opt):
+    """Configure the depth we'll sync down.
+
+    Args:
+      opt: Options from optparse.  We care about opt.depth.
+    """
+    # Opt.depth will be non-None if user actually passed --depth to repo init.
+    if opt.depth is not None:
+      if opt.depth > 0:
+        # Positive values will set the depth.
+        depth = str(opt.depth)
+      else:
+        # Negative numbers will clear the depth; passing None to SetString
+        # will do that.
+        depth = None
+
+      # We store the depth in the main manifest project.
+      self.manifest.manifestProject.config.SetString('repo.depth', depth)
+
+  def _DisplayResult(self):
+    if self.manifest.IsMirror:
+      init_type = 'mirror '
+    else:
+      init_type = ''
+
+    print()
+    print('repo %shas been initialized in %s'
+          % (init_type, self.manifest.topdir))
+
+    current_dir = os.getcwd()
+    if current_dir != self.manifest.topdir:
+      print('If this is not the directory in which you want to initialize '
+            'repo, please run:')
+      print('   rm -r %s/.repo' % self.manifest.topdir)
+      print('and try again.')
+
   def Execute(self, opt, args):
-    self._CheckGitVersion()
+    git_require(MIN_GIT_VERSION, fail=True)
+
+    if opt.reference:
+      opt.reference = os.path.expanduser(opt.reference)
+
+    # Check this here, else manifest will be tagged "not new" and init won't be
+    # possible anymore without removing the .repo/manifests directory.
+    if opt.archive and opt.mirror:
+      print('fatal: --mirror and --archive cannot be used together.',
+            file=sys.stderr)
+      sys.exit(1)
+
     self._SyncManifest(opt)
     self._LinkManifest(opt.manifest_name)
 
-    if os.isatty(0) and os.isatty(1) and not opt.mirror:
-      self._ConfigureUser()
+    if os.isatty(0) and os.isatty(1) and not self.manifest.IsMirror:
+      if opt.config_name or self._ShouldConfigureUser():
+        self._ConfigureUser()
       self._ConfigureColor()
 
-    if opt.mirror:
-      type = 'mirror '
-    else:
-      type = ''
+    self._ConfigureDepth(opt)
 
-    print ''
-    print 'repo %sinitialized in %s' % (type, self.manifest.topdir)
+    self._DisplayResult()

subcmds/list.py

@@ -0,0 +1,87 @@
+#
+# Copyright (C) 2011 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import sys
+
+from command import Command, MirrorSafeCommand
+
+class List(Command, MirrorSafeCommand):
+  common = True
+  helpSummary = "List projects and their associated directories"
+  helpUsage = """
+%prog [-f] [<project>...]
+%prog [-f] -r str1 [str2]..."
+"""
+  helpDescription = """
+List all projects; pass '.' to list the project for the cwd.
+
+This is similar to running: repo forall -c 'echo "$REPO_PATH : $REPO_PROJECT"'.
+"""
+
+  def _Options(self, p):
+    p.add_option('-r', '--regex',
+                 dest='regex', action='store_true',
+                 help="Filter the project list based on regex or wildcard matching of strings")
+    p.add_option('-g', '--groups',
+                 dest='groups',
+                 help="Filter the project list based on the groups the project is in")
+    p.add_option('-f', '--fullpath',
+                 dest='fullpath', action='store_true',
+                 help="Display the full work tree path instead of the relative path")
+    p.add_option('-n', '--name-only',
+                 dest='name_only', action='store_true',
+                 help="Display only the name of the repository")
+    p.add_option('-p', '--path-only',
+                 dest='path_only', action='store_true',
+                 help="Display only the path of the repository")
+
+  def Execute(self, opt, args):
+    """List all projects and the associated directories.
+
+    This may be possible to do with 'repo forall', but repo newbies have
+    trouble figuring that out.  The idea here is that it should be more
+    discoverable.
+
+    Args:
+      opt: The options.
+      args: Positional args.  Can be a list of projects to list, or empty.
+    """
+
+    if opt.fullpath and opt.name_only:
+      print('error: cannot combine -f and -n', file=sys.stderr)
+      sys.exit(1)
+
+    if not opt.regex:
+      projects = self.GetProjects(args, groups=opt.groups)
+    else:
+      projects = self.FindProjects(args)
+
+    def _getpath(x):
+      if opt.fullpath:
+        return x.worktree
+      return x.relpath
+
+    lines = []
+    for project in projects:
+      if opt.name_only and not opt.path_only:
+        lines.append("%s" % ( project.name))
+      elif opt.path_only and not opt.name_only:
+        lines.append("%s" % (_getpath(project)))
+      else:
+        lines.append("%s : %s" % (_getpath(project), project.name))
+
+    lines.sort()
+    print('\n'.join(lines))

subcmds/manifest.py

@@ -0,0 +1,85 @@
+#
+# Copyright (C) 2009 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import os
+import sys
+
+from command import PagedCommand
+
+class Manifest(PagedCommand):
+  common = False
+  helpSummary = "Manifest inspection utility"
+  helpUsage = """
+%prog [-o {-|NAME.xml} [-r]]
+"""
+  _helpDescription = """
+
+With the -o option, exports the current manifest for inspection.
+The manifest and (if present) local_manifest.xml are combined
+together to produce a single manifest file.  This file can be stored
+in a Git repository for use during future 'repo init' invocations.
+
+"""
+
+  @property
+  def helpDescription(self):
+    helptext = self._helpDescription + '\n'
+    r = os.path.dirname(__file__)
+    r = os.path.dirname(r)
+    fd = open(os.path.join(r, 'docs', 'manifest-format.md'))
+    for line in fd:
+      helptext += line
+    fd.close()
+    return helptext
+
+  def _Options(self, p):
+    p.add_option('-r', '--revision-as-HEAD',
+                 dest='peg_rev', action='store_true',
+                 help='Save revisions as current HEAD')
+    p.add_option('--suppress-upstream-revision', dest='peg_rev_upstream',
+                 default=True, action='store_false',
+                 help='If in -r mode, do not write the upstream field.  '
+                 'Only of use if the branch names for a sha1 manifest are '
+                 'sensitive.')
+    p.add_option('-o', '--output-file',
+                 dest='output_file',
+                 default='-',
+                 help='File to save the manifest to',
+                 metavar='-|NAME.xml')
+
+  def _Output(self, opt):
+    if opt.output_file == '-':
+      fd = sys.stdout
+    else:
+      fd = open(opt.output_file, 'w')
+    self.manifest.Save(fd,
+                       peg_rev = opt.peg_rev,
+                       peg_rev_upstream = opt.peg_rev_upstream)
+    fd.close()
+    if opt.output_file != '-':
+      print('Saved manifest to %s' % opt.output_file, file=sys.stderr)
+
+  def Execute(self, opt, args):
+    if args:
+      self.Usage()
+
+    if opt.output_file is not None:
+      self._Output(opt)
+      return
+
+    print('error: no operation to perform', file=sys.stderr)
+    print('error: see repo help manifest', file=sys.stderr)
+    sys.exit(1)

subcmds/overview.py

@@ -0,0 +1,84 @@
+#
+# Copyright (C) 2012 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+from color import Coloring
+from command import PagedCommand
+
+
+class Overview(PagedCommand):
+  common = True
+  helpSummary = "Display overview of unmerged project branches"
+  helpUsage = """
+%prog [--current-branch] [<project>...]
+"""
+  helpDescription = """
+The '%prog' command is used to display an overview of the projects branches,
+and list any local commits that have not yet been merged into the project.
+
+The -b/--current-branch option can be used to restrict the output to only
+branches currently checked out in each project.  By default, all branches
+are displayed.
+"""
+
+  def _Options(self, p):
+    p.add_option('-b', '--current-branch',
+                 dest="current_branch", action="store_true",
+                 help="Consider only checked out branches")
+
+  def Execute(self, opt, args):
+    all_branches = []
+    for project in self.GetProjects(args):
+      br = [project.GetUploadableBranch(x)
+            for x in project.GetBranches()]
+      br = [x for x in br if x]
+      if opt.current_branch:
+        br = [x for x in br if x.name == project.CurrentBranch]
+      all_branches.extend(br)
+
+    if not all_branches:
+      return
+
+    class Report(Coloring):
+      def __init__(self, config):
+        Coloring.__init__(self, config, 'status')
+        self.project = self.printer('header', attr='bold')
+        self.text = self.printer('text')
+
+    out = Report(all_branches[0].project.config)
+    out.text("Deprecated. See repo info -o.")
+    out.nl()
+    out.project('Projects Overview')
+    out.nl()
+
+    project = None
+
+    for branch in all_branches:
+      if project != branch.project:
+        project = branch.project
+        out.nl()
+        out.project('project %s/' % project.relpath)
+        out.nl()
+
+      commits = branch.commits
+      date = branch.date
+      print('%s %-33s (%2d commit%s, %s)' % (
+            branch.name == project.CurrentBranch and '*' or ' ',
+            branch.name,
+            len(commits),
+            len(commits) != 1 and 's' or ' ',
+            date))
+      for commit in commits:
+        print('%-35s   - %s' % ('', commit))

subcmds/prune.py

@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
 from color import Coloring
 from command import PagedCommand
 
@@ -24,11 +25,11 @@ class Prune(PagedCommand):
 """
 
   def Execute(self, opt, args):
-    all = []
+    all_branches = []
     for project in self.GetProjects(args):
-      all.extend(project.PruneHeads())
+      all_branches.extend(project.PruneHeads())
 
-    if not all:
+    if not all_branches:
       return
 
     class Report(Coloring):
@@ -36,13 +37,13 @@ class Prune(PagedCommand):
         Coloring.__init__(self, config, 'status')
         self.project = self.printer('header', attr='bold')
 
-    out = Report(all[0].project.config)
+    out = Report(all_branches[0].project.config)
     out.project('Pending Branches')
     out.nl()
 
     project = None
 
-    for branch in all:
+    for branch in all_branches:
       if project != branch.project:
         project = branch.project
         out.nl()
@@ -51,9 +52,9 @@ class Prune(PagedCommand):
 
       commits = branch.commits
       date = branch.date
-      print '%s %-33s (%2d commit%s, %s)' % (
+      print('%s %-33s (%2d commit%s, %s)' % (
             branch.name == project.CurrentBranch and '*' or ' ',
             branch.name,
             len(commits),
             len(commits) != 1 and 's' or ' ',
-            date)
+            date))

subcmds/rebase.py

@@ -0,0 +1,142 @@
+#
+# Copyright (C) 2010 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import sys
+
+from command import Command
+from git_command import GitCommand
+
+class Rebase(Command):
+  common = True
+  helpSummary = "Rebase local branches on upstream branch"
+  helpUsage = """
+%prog {[<project>...] | -i <project>...}
+"""
+  helpDescription = """
+'%prog' uses git rebase to move local changes in the current topic branch to
+the HEAD of the upstream history, useful when you have made commits in a topic
+branch but need to incorporate new upstream changes "underneath" them.
+"""
+
+  def _Options(self, p):
+    p.add_option('-i', '--interactive',
+                dest="interactive", action="store_true",
+                help="interactive rebase (single project only)")
+
+    p.add_option('-f', '--force-rebase',
+                 dest='force_rebase', action='store_true',
+                 help='Pass --force-rebase to git rebase')
+    p.add_option('--no-ff',
+                 dest='no_ff', action='store_true',
+                 help='Pass --no-ff to git rebase')
+    p.add_option('-q', '--quiet',
+                 dest='quiet', action='store_true',
+                 help='Pass --quiet to git rebase')
+    p.add_option('--autosquash',
+                 dest='autosquash', action='store_true',
+                 help='Pass --autosquash to git rebase')
+    p.add_option('--whitespace',
+                 dest='whitespace', action='store', metavar='WS',
+                 help='Pass --whitespace to git rebase')
+    p.add_option('--auto-stash',
+                 dest='auto_stash', action='store_true',
+                 help='Stash local modifications before starting')
+    p.add_option('-m', '--onto-manifest',
+                 dest='onto_manifest', action='store_true',
+                 help='Rebase onto the manifest version instead of upstream '
+                      'HEAD.  This helps to make sure the local tree stays '
+                      'consistent if you previously synced to a manifest.')
+
+  def Execute(self, opt, args):
+    all_projects = self.GetProjects(args)
+    one_project = len(all_projects) == 1
+
+    if opt.interactive and not one_project:
+      print('error: interactive rebase not supported with multiple projects',
+            file=sys.stderr)
+      if len(args) == 1:
+        print('note: project %s is mapped to more than one path' % (args[0],),
+            file=sys.stderr)
+      return -1
+
+    for project in all_projects:
+      cb = project.CurrentBranch
+      if not cb:
+        if one_project:
+          print("error: project %s has a detached HEAD" % project.relpath,
+                file=sys.stderr)
+          return -1
+        # ignore branches with detatched HEADs
+        continue
+
+      upbranch = project.GetBranch(cb)
+      if not upbranch.LocalMerge:
+        if one_project:
+          print("error: project %s does not track any remote branches"
+                % project.relpath, file=sys.stderr)
+          return -1
+        # ignore branches without remotes
+        continue
+
+      args = ["rebase"]
+
+      if opt.whitespace:
+        args.append('--whitespace=%s' % opt.whitespace)
+
+      if opt.quiet:
+        args.append('--quiet')
+
+      if opt.force_rebase:
+        args.append('--force-rebase')
+
+      if opt.no_ff:
+        args.append('--no-ff')
+
+      if opt.autosquash:
+        args.append('--autosquash')
+
+      if opt.interactive:
+        args.append("-i")
+
+      if opt.onto_manifest:
+        args.append('--onto')
+        args.append(project.revisionExpr)
+
+      args.append(upbranch.LocalMerge)
+
+      print('# %s: rebasing %s -> %s'
+            % (project.relpath, cb, upbranch.LocalMerge), file=sys.stderr)
+
+      needs_stash = False
+      if opt.auto_stash:
+        stash_args = ["update-index", "--refresh", "-q"]
+
+        if GitCommand(project, stash_args).Wait() != 0:
+          needs_stash = True
+          # Dirty index, requires stash...
+          stash_args = ["stash"]
+
+          if GitCommand(project, stash_args).Wait() != 0:
+            return -1
+
+      if GitCommand(project, args).Wait() != 0:
+        return -1
+
+      if needs_stash:
+        stash_args.append('pop')
+        stash_args.append('--quiet')
+        if GitCommand(project, stash_args).Wait() != 0:
+          return -1

subcmds/selfupdate.py

@@ -0,0 +1,62 @@
+#
+# Copyright (C) 2009 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+from optparse import SUPPRESS_HELP
+import sys
+
+from command import Command, MirrorSafeCommand
+from subcmds.sync import _PostRepoUpgrade
+from subcmds.sync import _PostRepoFetch
+
+class Selfupdate(Command, MirrorSafeCommand):
+  common = False
+  helpSummary = "Update repo to the latest version"
+  helpUsage = """
+%prog
+"""
+  helpDescription = """
+The '%prog' command upgrades repo to the latest version, if a
+newer version is available.
+
+Normally this is done automatically by 'repo sync' and does not
+need to be performed by an end-user.
+"""
+
+  def _Options(self, p):
+    g = p.add_option_group('repo Version options')
+    g.add_option('--no-repo-verify',
+                 dest='no_repo_verify', action='store_true',
+                 help='do not verify repo source code')
+    g.add_option('--repo-upgraded',
+                 dest='repo_upgraded', action='store_true',
+                 help=SUPPRESS_HELP)
+
+  def Execute(self, opt, args):
+    rp = self.manifest.repoProject
+    rp.PreSync()
+
+    if opt.repo_upgraded:
+      _PostRepoUpgrade(self.manifest)
+
+    else:
+      if not rp.Sync_NetworkHalf():
+        print("error: can't update repo", file=sys.stderr)
+        sys.exit(1)
+
+      rp.bare_git.gc('--auto')
+      _PostRepoFetch(rp,
+                     no_repo_verify = opt.no_repo_verify,
+                     verbose = True)

subcmds/smartsync.py

@@ -0,0 +1,33 @@
+#
+# Copyright (C) 2010 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from subcmds.sync import Sync
+
+class Smartsync(Sync):
+  common = True
+  helpSummary = "Update working tree to the latest known good revision"
+  helpUsage = """
+%prog [<project>...]
+"""
+  helpDescription = """
+The '%prog' command is a shortcut for sync -s.
+"""
+
+  def _Options(self, p):
+    Sync._Options(self, p, show_smart=False)
+
+  def Execute(self, opt, args):
+    opt.smart_sync = True
+    Sync.Execute(self, opt, args)

subcmds/stage.py

@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
 import sys
 
 from color import Coloring
@@ -48,19 +49,19 @@ The '%prog' command stages files to prepare the next commit.
       self.Usage()
 
   def _Interactive(self, opt, args):
-    all = filter(lambda x: x.IsDirty(), self.GetProjects(args))
-    if not all:
-      print >>sys.stderr,'no projects have uncommitted modifications'
+    all_projects = [p for p in self.GetProjects(args) if p.IsDirty()]
+    if not all_projects:
+      print('no projects have uncommitted modifications', file=sys.stderr)
       return
 
     out = _ProjectList(self.manifest.manifestProject.config)
     while True:
-      out.header('        %-20s %s', 'project', 'path')
+      out.header('        %s', 'project')
       out.nl()
 
-      for i in xrange(0, len(all)):
-        p = all[i]
-        out.write('%3d:    %-20s %s', i + 1, p.name, p.relpath + '/')
+      for i in range(len(all_projects)):
+        project = all_projects[i]
+        out.write('%3d:    %s', i + 1, project.relpath + '/')
         out.nl()
       out.nl()
 
@@ -93,15 +94,15 @@ The '%prog' command stages files to prepare the next commit.
       if a_index is not None:
         if a_index == 0:
           break
-        if 0 < a_index and a_index <= len(all):
-          _AddI(all[a_index - 1])
+        if 0 < a_index and a_index <= len(all_projects):
+          _AddI(all_projects[a_index - 1])
           continue
 
-      p = filter(lambda x: x.name == a or x.relpath == a, all)
-      if len(p) == 1:
-        _AddI(p[0])
+      projects = [p for p in all_projects if a in [p.name, p.relpath]]
+      if len(projects) == 1:
+        _AddI(projects[0])
         continue
-    print 'Bye.'
+    print('Bye.')
 
 def _AddI(project):
   p = GitCommand(project, ['add', '--interactive'], bare=False)

subcmds/start.py

@@ -13,30 +13,32 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
+import os
 import sys
+
 from command import Command
+from git_config import IsImmutable
 from git_command import git
+import gitc_utils
+from progress import Progress
+from project import SyncBuffer
 
 class Start(Command):
   common = True
   helpSummary = "Start a new branch for development"
   helpUsage = """
-%prog <newbranchname> [<project>...]
-
-This subcommand starts a new branch of development that is automatically
-pulled from a remote branch.
-
-It is equivalent to the following git commands:
-
-"git branch --track <newbranchname> m/<codeline>",
-or 
-"git checkout --track -b <newbranchname> m/<codeline>".
-
-All three forms set up the config entries that repo bases some of its
-processing on.  Use %prog or git branch or checkout with --track to ensure
-the configuration data is set up properly.
-
+%prog <newbranchname> [--all | <project>...]
 """
+  helpDescription = """
+'%prog' begins a new branch of development, starting from the
+revision specified in the manifest.
+"""
+
+  def _Options(self, p):
+    p.add_option('--all',
+                 dest='all', action='store_true',
+                 help='begin branch in all projects')
 
   def Execute(self, opt, args):
     if not args:
@@ -44,8 +46,72 @@ the configuration data is set up properly.
 
     nb = args[0]
     if not git.check_ref_format('heads/%s' % nb):
-      print >>sys.stderr, "error: '%s' is not a valid name" % nb
+      print("error: '%s' is not a valid name" % nb, file=sys.stderr)
       sys.exit(1)
 
-    for project in self.GetProjects(args[1:]):
-      project.StartBranch(nb)
+    err = []
+    projects = []
+    if not opt.all:
+      projects = args[1:]
+      if len(projects) < 1:
+        projects = ['.',]  # start it in the local project by default
+
+    all_projects = self.GetProjects(projects,
+                                    missing_ok=bool(self.gitc_manifest))
+
+    # This must happen after we find all_projects, since GetProjects may need
+    # the local directory, which will disappear once we save the GITC manifest.
+    if self.gitc_manifest:
+      gitc_projects = self.GetProjects(projects, manifest=self.gitc_manifest,
+                                       missing_ok=True)
+      for project in gitc_projects:
+        if project.old_revision:
+          project.already_synced = True
+        else:
+          project.already_synced = False
+          project.old_revision = project.revisionExpr
+        project.revisionExpr = None
+      # Save the GITC manifest.
+      gitc_utils.save_manifest(self.gitc_manifest)
+
+      # Make sure we have a valid CWD
+      if not os.path.exists(os.getcwd()):
+        os.chdir(self.manifest.topdir)
+
+    pm = Progress('Starting %s' % nb, len(all_projects))
+    for project in all_projects:
+      pm.update()
+
+      if self.gitc_manifest:
+        gitc_project = self.gitc_manifest.paths[project.relpath]
+        # Sync projects that have not been opened.
+        if not gitc_project.already_synced:
+          proj_localdir = os.path.join(self.gitc_manifest.gitc_client_dir,
+                                       project.relpath)
+          project.worktree = proj_localdir
+          if not os.path.exists(proj_localdir):
+            os.makedirs(proj_localdir)
+          project.Sync_NetworkHalf()
+          sync_buf = SyncBuffer(self.manifest.manifestProject.config)
+          project.Sync_LocalHalf(sync_buf)
+          project.revisionId = gitc_project.old_revision
+
+      # If the current revision is immutable, such as a SHA1, a tag or
+      # a change, then we can't push back to it. Substitute with
+      # dest_branch, if defined; or with manifest default revision instead.
+      branch_merge = ''
+      if IsImmutable(project.revisionExpr):
+        if project.dest_branch:
+          branch_merge = project.dest_branch
+        else:
+          branch_merge = self.manifest.default.revisionExpr
+
+      if not project.StartBranch(nb, branch_merge=branch_merge):
+        err.append(project)
+    pm.end()
+
+    if err:
+      for p in err:
+        print("error: %s/: cannot start %s" % (p.relpath, nb),
+              file=sys.stderr)
+      sys.exit(1)

subcmds/status.py

@@ -15,13 +15,180 @@
 
 from command import PagedCommand
 
+try:
+  import threading as _threading
+except ImportError:
+  import dummy_threading as _threading
+
+import glob
+
+import itertools
+import os
+
+from color import Coloring
+import platform_utils
+
 class Status(PagedCommand):
   common = True
   helpSummary = "Show the working tree status"
   helpUsage = """
 %prog [<project>...]
 """
+  helpDescription = """
+'%prog' compares the working tree to the staging area (aka index),
+and the most recent commit on this branch (HEAD), in each project
+specified.  A summary is displayed, one line per file where there
+is a difference between these three states.
+
+The -j/--jobs option can be used to run multiple status queries
+in parallel.
+
+The -o/--orphans option can be used to show objects that are in
+the working directory, but not associated with a repo project.
+This includes unmanaged top-level files and directories, but also
+includes deeper items.  For example, if dir/subdir/proj1 and
+dir/subdir/proj2 are repo projects, dir/subdir/proj3 will be shown
+if it is not known to repo.
+
+# Status Display
+
+The status display is organized into three columns of information,
+for example if the file 'subcmds/status.py' is modified in the
+project 'repo' on branch 'devwork':
+
+  project repo/                                   branch devwork
+   -m     subcmds/status.py
+
+The first column explains how the staging area (index) differs from
+the last commit (HEAD).  Its values are always displayed in upper
+case and have the following meanings:
+
+ -:  no difference
+ A:  added         (not in HEAD,     in index                     )
+ M:  modified      (    in HEAD,     in index, different content  )
+ D:  deleted       (    in HEAD, not in index                     )
+ R:  renamed       (not in HEAD,     in index, path changed       )
+ C:  copied        (not in HEAD,     in index, copied from another)
+ T:  mode changed  (    in HEAD,     in index, same content       )
+ U:  unmerged; conflict resolution required
+
+The second column explains how the working directory differs from
+the index.  Its values are always displayed in lower case and have
+the following meanings:
+
+ -:  new / unknown (not in index,     in work tree                )
+ m:  modified      (    in index,     in work tree, modified      )
+ d:  deleted       (    in index, not in work tree                )
+
+"""
+
+  def _Options(self, p):
+    p.add_option('-j', '--jobs',
+                 dest='jobs', action='store', type='int', default=2,
+                 help="number of projects to check simultaneously")
+    p.add_option('-o', '--orphans',
+                 dest='orphans', action='store_true',
+                 help="include objects in working directory outside of repo projects")
+    p.add_option('-q', '--quiet', action='store_true',
+                 help="only print the name of modified projects")
+
+  def _StatusHelper(self, project, clean_counter, sem, quiet):
+    """Obtains the status for a specific project.
+
+    Obtains the status for a project, redirecting the output to
+    the specified object. It will release the semaphore
+    when done.
+
+    Args:
+      project: Project to get status of.
+      clean_counter: Counter for clean projects.
+      sem: Semaphore, will call release() when complete.
+      output: Where to output the status.
+    """
+    try:
+      state = project.PrintWorkTreeStatus(quiet=quiet)
+      if state == 'CLEAN':
+        next(clean_counter)
+    finally:
+      sem.release()
+
+  def _FindOrphans(self, dirs, proj_dirs, proj_dirs_parents, outstring):
+    """find 'dirs' that are present in 'proj_dirs_parents' but not in 'proj_dirs'"""
+    status_header = ' --\t'
+    for item in dirs:
+      if not platform_utils.isdir(item):
+        outstring.append(''.join([status_header, item]))
+        continue
+      if item in proj_dirs:
+        continue
+      if item in proj_dirs_parents:
+        self._FindOrphans(glob.glob('%s/.*' % item) +
+            glob.glob('%s/*' % item),
+            proj_dirs, proj_dirs_parents, outstring)
+        continue
+      outstring.append(''.join([status_header, item, '/']))
 
   def Execute(self, opt, args):
-    for project in self.GetProjects(args):
-      project.PrintWorkTreeStatus()
+    all_projects = self.GetProjects(args)
+    counter = itertools.count()
+
+    if opt.jobs == 1:
+      for project in all_projects:
+        state = project.PrintWorkTreeStatus(quiet=opt.quiet)
+        if state == 'CLEAN':
+          next(counter)
+    else:
+      sem = _threading.Semaphore(opt.jobs)
+      threads = []
+      for project in all_projects:
+        sem.acquire()
+
+        t = _threading.Thread(target=self._StatusHelper,
+                              args=(project, counter, sem, opt.quiet))
+        threads.append(t)
+        t.daemon = True
+        t.start()
+      for t in threads:
+        t.join()
+    if not opt.quiet and len(all_projects) == next(counter):
+      print('nothing to commit (working directory clean)')
+
+    if opt.orphans:
+      proj_dirs = set()
+      proj_dirs_parents = set()
+      for project in self.GetProjects(None, missing_ok=True):
+        proj_dirs.add(project.relpath)
+        (head, _tail) = os.path.split(project.relpath)
+        while head != "":
+          proj_dirs_parents.add(head)
+          (head, _tail) = os.path.split(head)
+      proj_dirs.add('.repo')
+
+      class StatusColoring(Coloring):
+        def __init__(self, config):
+          Coloring.__init__(self, config, 'status')
+          self.project = self.printer('header', attr = 'bold')
+          self.untracked = self.printer('untracked', fg = 'red')
+
+      orig_path = os.getcwd()
+      try:
+        os.chdir(self.manifest.topdir)
+
+        outstring = []
+        self._FindOrphans(glob.glob('.*') +
+            glob.glob('*'),
+            proj_dirs, proj_dirs_parents, outstring)
+
+        if outstring:
+          output = StatusColoring(self.manifest.globalConfig)
+          output.project('Objects not within a project (orphans)')
+          output.nl()
+          for entry in outstring:
+            output.untracked(entry)
+            output.nl()
+        else:
+          print('No orphan files or directories')
+
+      finally:
+        # Restore CWD.
+        os.chdir(orig_path)

subcmds/upload.py

@@ -13,62 +13,227 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
+import copy
 import re
 import sys
 
 from command import InteractiveCommand
 from editor import Editor
-from error import UploadError
+from error import HookError, UploadError
+from git_command import GitCommand
+from project import RepoHook
+
+from pyversion import is_python3
+if not is_python3():
+  input = raw_input
+else:
+  unicode = str
+
+UNUSUAL_COMMIT_THRESHOLD = 5
+
+def _ConfirmManyUploads(multiple_branches=False):
+  if multiple_branches:
+    print('ATTENTION: One or more branches has an unusually high number '
+          'of commits.')
+  else:
+    print('ATTENTION: You are uploading an unusually high number of commits.')
+  print('YOU PROBABLY DO NOT MEAN TO DO THIS. (Did you rebase across '
+        'branches?)')
+  answer = input("If you are sure you intend to do this, type 'yes': ").strip()
+  return answer == "yes"
 
 def _die(fmt, *args):
   msg = fmt % args
-  print >>sys.stderr, 'error: %s' % msg
+  print('error: %s' % msg, file=sys.stderr)
   sys.exit(1)
 
+def _SplitEmails(values):
+  result = []
+  for value in values:
+    result.extend([s.strip() for s in value.split(',')])
+  return result
+
 class Upload(InteractiveCommand):
   common = True
   helpSummary = "Upload changes for code review"
-  helpUsage="""
-%prog [<project>]...
+  helpUsage = """
+%prog [--re --cc] [<project>]...
 """
   helpDescription = """
-The '%prog' command is used to send changes to the Gerrit code
-review system.  It searches for changes in local projects that do
-not yet exist in the corresponding remote repository.  If multiple
-changes are found, '%prog' opens an editor to allow the
-user to choose which change to upload.  After a successful upload,
-repo prints the URL for the change in the Gerrit code review system.
+The '%prog' command is used to send changes to the Gerrit Code
+Review system.  It searches for topic branches in local projects
+that have not yet been published for review.  If multiple topic
+branches are found, '%prog' opens an editor to allow the user to
+select which branches to upload.
+
+'%prog' searches for uploadable changes in all projects listed at
+the command line.  Projects can be specified either by name, or by
+a relative or absolute path to the project's local directory. If no
+projects are specified, '%prog' will search for uploadable changes
+in all projects listed in the manifest.
+
+If the --reviewers or --cc options are passed, those emails are
+added to the respective list of users, and emails are sent to any
+new users.  Users passed as --reviewers must already be registered
+with the code review system, or the upload will fail.
+
+# Configuration
+
+review.URL.autoupload:
+
+To disable the "Upload ... (y/N)?" prompt, you can set a per-project
+or global Git configuration option.  If review.URL.autoupload is set
+to "true" then repo will assume you always answer "y" at the prompt,
+and will not prompt you further.  If it is set to "false" then repo
+will assume you always answer "n", and will abort.
+
+review.URL.autoreviewer:
+
+To automatically append a user or mailing list to reviews, you can set
+a per-project or global Git option to do so.
+
+review.URL.autocopy:
+
+To automatically copy a user or mailing list to all uploaded reviews,
+you can set a per-project or global Git option to do so. Specifically,
+review.URL.autocopy can be set to a comma separated list of reviewers
+who you always want copied on all uploads with a non-empty --re
+argument.
+
+review.URL.username:
+
+Override the username used to connect to Gerrit Code Review.
+By default the local part of the email address is used.
+
+The URL must match the review URL listed in the manifest XML file,
+or in the .git/config within the project.  For example:
+
+  [remote "origin"]
+    url = git://git.example.com/project.git
+    review = http://review.example.com/
+
+  [review "http://review.example.com/"]
+    autoupload = true
+    autocopy = johndoe@company.com,my-team-alias@company.com
+
+review.URL.uploadtopic:
+
+To add a topic branch whenever uploading a commit, you can set a
+per-project or global Git option to do so. If review.URL.uploadtopic
+is set to "true" then repo will assume you always want the equivalent
+of the -t option to the repo command. If unset or set to "false" then
+repo will make use of only the command line option.
+
+# References
+
+Gerrit Code Review:  https://www.gerritcodereview.com/
 
-'%prog' searches for uploadable changes in all projects listed
-at the command line.  Projects can be specified either by name, or
-by a relative or absolute path to the project's local directory. If
-no projects are specified, '%prog' will search for uploadable
-changes in all projects listed in the manifest.
 """
 
-  def _SingleBranch(self, branch):
+  def _Options(self, p):
+    p.add_option('-t',
+                 dest='auto_topic', action='store_true',
+                 help='Send local branch name to Gerrit Code Review')
+    p.add_option('--re', '--reviewers',
+                 type='string',  action='append', dest='reviewers',
+                 help='Request reviews from these people.')
+    p.add_option('--cc',
+                 type='string',  action='append', dest='cc',
+                 help='Also send email to these email addresses.')
+    p.add_option('--br',
+                 type='string',  action='store', dest='branch',
+                 help='Branch to upload.')
+    p.add_option('--cbr', '--current-branch',
+                 dest='current_branch', action='store_true',
+                 help='Upload current git branch.')
+    p.add_option('-d', '--draft',
+                 action='store_true', dest='draft', default=False,
+                 help='If specified, upload as a draft.')
+    p.add_option('--ne', '--no-emails',
+                 action='store_false', dest='notify', default=True,
+                 help='If specified, do not send emails on upload.')
+    p.add_option('-p', '--private',
+                 action='store_true', dest='private', default=False,
+                 help='If specified, upload as a private change.')
+    p.add_option('-w', '--wip',
+                 action='store_true', dest='wip', default=False,
+                 help='If specified, upload as a work-in-progress change.')
+    p.add_option('-o', '--push-option',
+                 type='string', action='append', dest='push_options',
+                 default=[],
+                 help='Additional push options to transmit')
+    p.add_option('-D', '--destination', '--dest',
+                 type='string', action='store', dest='dest_branch',
+                 metavar='BRANCH',
+                 help='Submit for review on this target branch.')
+
+    # Options relating to upload hook.  Note that verify and no-verify are NOT
+    # opposites of each other, which is why they store to different locations.
+    # We are using them to match 'git commit' syntax.
+    #
+    # Combinations:
+    # - no-verify=False, verify=False (DEFAULT):
+    #   If stdout is a tty, can prompt about running upload hooks if needed.
+    #   If user denies running hooks, the upload is cancelled.  If stdout is
+    #   not a tty and we would need to prompt about upload hooks, upload is
+    #   cancelled.
+    # - no-verify=False, verify=True:
+    #   Always run upload hooks with no prompt.
+    # - no-verify=True, verify=False:
+    #   Never run upload hooks, but upload anyway (AKA bypass hooks).
+    # - no-verify=True, verify=True:
+    #   Invalid
+    p.add_option('--no-cert-checks',
+                 dest='validate_certs', action='store_false', default=True,
+                 help='Disable verifying ssl certs (unsafe).')
+    p.add_option('--no-verify',
+                 dest='bypass_hooks', action='store_true',
+                 help='Do not run the upload hook.')
+    p.add_option('--verify',
+                 dest='allow_all_hooks', action='store_true',
+                 help='Run the upload hook without prompting.')
+
+  def _SingleBranch(self, opt, branch, people):
     project = branch.project
     name = branch.name
-    date = branch.date
-    list = branch.commits
+    remote = project.GetBranch(name).remote
 
-    print 'Upload project %s/:' % project.relpath
-    print '  branch %s (%2d commit%s, %s):' % (
-                  name,
-                  len(list),
-                  len(list) != 1 and 's' or '',
-                  date)
-    for commit in list:
-      print '         %s' % commit
+    key = 'review.%s.autoupload' % remote.review
+    answer = project.config.GetBoolean(key)
 
-    sys.stdout.write('(y/n)? ')
-    answer = sys.stdin.readline().strip()
-    if answer in ('y', 'Y', 'yes', '1', 'true', 't'):
-      self._UploadAndReport([branch])
+    if answer is False:
+      _die("upload blocked by %s = false" % key)
+
+    if answer is None:
+      date = branch.date
+      commit_list = branch.commits
+
+      destination = opt.dest_branch or project.dest_branch or project.revisionExpr
+      print('Upload project %s/ to remote branch %s%s:' %
+            (project.relpath, destination, ' (draft)' if opt.draft else ''))
+      print('  branch %s (%2d commit%s, %s):' % (
+                    name,
+                    len(commit_list),
+                    len(commit_list) != 1 and 's' or '',
+                    date))
+      for commit in commit_list:
+        print('         %s' % commit)
+
+      sys.stdout.write('to %s (y/N)? ' % remote.review)
+      answer = sys.stdin.readline().strip().lower()
+      answer = answer in ('y', 'yes', '1', 'true', 't')
+
+    if answer:
+      if len(branch.commits) > UNUSUAL_COMMIT_THRESHOLD:
+        answer = _ConfirmManyUploads()
+
+    if answer:
+      self._UploadAndReport(opt, [branch], people)
     else:
       _die("upload aborted by user")
 
-  def _MultipleBranches(self, pending):
+  def _MultipleBranches(self, opt, pending, people):
     projects = {}
     branches = {}
 
@@ -80,18 +245,22 @@ changes in all projects listed in the manifest.
 
       b = {}
       for branch in avail:
+        if branch is None:
+          continue
         name = branch.name
         date = branch.date
-        list = branch.commits
+        commit_list = branch.commits
 
         if b:
           script.append('#')
-        script.append('#  branch %s (%2d commit%s, %s):' % (
+        destination = opt.dest_branch or project.dest_branch or project.revisionExpr
+        script.append('#  branch %s (%2d commit%s, %s) to remote branch %s:' % (
                       name,
-                      len(list),
-                      len(list) != 1 and 's' or '',
-                      date))
-        for commit in list:
+                      len(commit_list),
+                      len(commit_list) != 1 and 's' or '',
+                      date,
+                      destination))
+        for commit in commit_list:
           script.append('#         %s' % commit)
         b[name] = branch
 
@@ -99,6 +268,11 @@ changes in all projects listed in the manifest.
       branches[project.name] = b
     script.append('')
 
+    script = [ x.encode('utf-8')
+             if issubclass(type(x), unicode)
+             else x
+             for x in script ]
+
     script = Editor.EditString("\n".join(script)).split("\n")
 
     project_re = re.compile(r'^#?\s*project\s*([^\s]+)/:$')
@@ -127,55 +301,206 @@ changes in all projects listed in the manifest.
         todo.append(branch)
     if not todo:
       _die("nothing uncommented for upload")
-    self._UploadAndReport(todo)
 
-  def _UploadAndReport(self, todo):
+    many_commits = False
+    for branch in todo:
+      if len(branch.commits) > UNUSUAL_COMMIT_THRESHOLD:
+        many_commits = True
+        break
+    if many_commits:
+      if not _ConfirmManyUploads(multiple_branches=True):
+        _die("upload aborted by user")
+
+    self._UploadAndReport(opt, todo, people)
+
+  def _AppendAutoList(self, branch, people):
+    """
+    Appends the list of reviewers in the git project's config.
+    Appends the list of users in the CC list in the git project's config if a
+    non-empty reviewer list was found.
+    """
+    name = branch.name
+    project = branch.project
+
+    key = 'review.%s.autoreviewer' % project.GetBranch(name).remote.review
+    raw_list = project.config.GetString(key)
+    if not raw_list is None:
+      people[0].extend([entry.strip() for entry in raw_list.split(',')])
+
+    key = 'review.%s.autocopy' % project.GetBranch(name).remote.review
+    raw_list = project.config.GetString(key)
+    if not raw_list is None and len(people[0]) > 0:
+      people[1].extend([entry.strip() for entry in raw_list.split(',')])
+
+  def _FindGerritChange(self, branch):
+    last_pub = branch.project.WasPublished(branch.name)
+    if last_pub is None:
+      return ""
+
+    refs = branch.GetPublishedRefs()
+    try:
+      # refs/changes/XYZ/N --> XYZ
+      return refs.get(last_pub).split('/')[-2]
+    except (AttributeError, IndexError):
+      return ""
+
+  def _UploadAndReport(self, opt, todo, original_people):
     have_errors = False
     for branch in todo:
       try:
-        branch.UploadForReview()
+        people = copy.deepcopy(original_people)
+        self._AppendAutoList(branch, people)
+
+        # Check if there are local changes that may have been forgotten
+        changes = branch.project.UncommitedFiles()
+        if changes:
+          key = 'review.%s.autoupload' % branch.project.remote.review
+          answer = branch.project.config.GetBoolean(key)
+
+          # if they want to auto upload, let's not ask because it could be automated
+          if answer is None:
+            sys.stdout.write('Uncommitted changes in ' + branch.project.name)
+            sys.stdout.write(' (did you forget to amend?):\n')
+            sys.stdout.write('\n'.join(changes) + '\n')
+            sys.stdout.write('Continue uploading? (y/N) ')
+            a = sys.stdin.readline().strip().lower()
+            if a not in ('y', 'yes', 't', 'true', 'on'):
+              print("skipping upload", file=sys.stderr)
+              branch.uploaded = False
+              branch.error = 'User aborted'
+              continue
+
+        # Check if topic branches should be sent to the server during upload
+        if opt.auto_topic is not True:
+          key = 'review.%s.uploadtopic' % branch.project.remote.review
+          opt.auto_topic = branch.project.config.GetBoolean(key)
+
+        destination = opt.dest_branch or branch.project.dest_branch
+
+        # Make sure our local branch is not setup to track a different remote branch
+        merge_branch = self._GetMergeBranch(branch.project)
+        if destination:
+          full_dest = 'refs/heads/%s' % destination
+          if not opt.dest_branch and merge_branch and merge_branch != full_dest:
+            print('merge branch %s does not match destination branch %s'
+                  % (merge_branch, full_dest))
+            print('skipping upload.')
+            print('Please use `--destination %s` if this is intentional'
+                  % destination)
+            branch.uploaded = False
+            continue
+
+        branch.UploadForReview(people,
+                               auto_topic=opt.auto_topic,
+                               draft=opt.draft,
+                               private=opt.private,
+                               notify=None if opt.notify else 'NONE',
+                               wip=opt.wip,
+                               dest_branch=destination,
+                               validate_certs=opt.validate_certs,
+                               push_options=opt.push_options)
+
         branch.uploaded = True
-      except UploadError, e:
+      except UploadError as e:
         branch.error = e
         branch.uploaded = False
         have_errors = True
 
-    print >>sys.stderr, ''
-    print >>sys.stderr, '--------------------------------------------'
+    print(file=sys.stderr)
+    print('----------------------------------------------------------------------', file=sys.stderr)
 
     if have_errors:
       for branch in todo:
         if not branch.uploaded:
-          print >>sys.stderr, '[FAILED] %-15s %-15s  (%s)' % (
+          if len(str(branch.error)) <= 30:
+            fmt = ' (%s)'
+          else:
+            fmt = '\n       (%s)'
+          print(('[FAILED] %-15s %-15s' + fmt) % (
                  branch.project.relpath + '/', \
                  branch.name, \
-                 branch.error)
-      print >>sys.stderr, ''
+                 str(branch.error)),
+                 file=sys.stderr)
+      print()
 
     for branch in todo:
-        if branch.uploaded:
-          print >>sys.stderr, '[OK    ] %-15s %s' % (
-                 branch.project.relpath + '/',
-                 branch.name)
-          print >>sys.stderr, '%s' % branch.tip_url
-          print >>sys.stderr, '(as %s)' % branch.owner_email
-          print >>sys.stderr, ''
+      if branch.uploaded:
+        print('[OK    ] %-15s %s' % (
+               branch.project.relpath + '/',
+               branch.name),
+               file=sys.stderr)
 
     if have_errors:
       sys.exit(1)
 
+  def _GetMergeBranch(self, project):
+    p = GitCommand(project,
+                   ['rev-parse', '--abbrev-ref', 'HEAD'],
+                   capture_stdout = True,
+                   capture_stderr = True)
+    p.Wait()
+    local_branch = p.stdout.strip()
+    p = GitCommand(project,
+                   ['config', '--get', 'branch.%s.merge' % local_branch],
+                   capture_stdout = True,
+                   capture_stderr = True)
+    p.Wait()
+    merge_branch = p.stdout.strip()
+    return merge_branch
+
   def Execute(self, opt, args):
     project_list = self.GetProjects(args)
     pending = []
+    reviewers = []
+    cc = []
+    branch = None
+
+    if opt.branch:
+      branch = opt.branch
 
     for project in project_list:
-      avail = project.GetUploadableBranches()
+      if opt.current_branch:
+        cbr = project.CurrentBranch
+        up_branch = project.GetUploadableBranch(cbr)
+        if up_branch:
+          avail = [up_branch]
+        else:
+          avail = None
+          print('ERROR: Current branch (%s) not uploadable. '
+                'You may be able to type '
+                '"git branch --set-upstream-to m/master" to fix '
+                'your branch.' % str(cbr),
+                file=sys.stderr)
+      else:
+        avail = project.GetUploadableBranches(branch)
       if avail:
         pending.append((project, avail))
 
     if not pending:
-      print >>sys.stdout, "no branches ready for upload"
-    elif len(pending) == 1 and len(pending[0][1]) == 1:
-      self._SingleBranch(pending[0][1][0])
+      print("no branches ready for upload", file=sys.stderr)
+      return
+
+    if not opt.bypass_hooks:
+      hook = RepoHook('pre-upload', self.manifest.repo_hooks_project,
+                      self.manifest.topdir,
+                      self.manifest.manifestProject.GetRemote('origin').url,
+                      abort_if_user_denies=True)
+      pending_proj_names = [project.name for (project, available) in pending]
+      pending_worktrees = [project.worktree for (project, available) in pending]
+      try:
+        hook.Run(opt.allow_all_hooks, project_list=pending_proj_names,
+                 worktree_list=pending_worktrees)
+      except HookError as e:
+        print("ERROR: %s" % str(e), file=sys.stderr)
+        return
+
+    if opt.reviewers:
+      reviewers = _SplitEmails(opt.reviewers)
+    if opt.cc:
+      cc = _SplitEmails(opt.cc)
+    people = (reviewers, cc)
+
+    if len(pending) == 1 and len(pending[0][1]) == 1:
+      self._SingleBranch(opt, pending[0][1][0], people)
     else:
-      self._MultipleBranches(pending)
+      self._MultipleBranches(opt, pending, people)

subcmds/version.py

@@ -0,0 +1,44 @@
+#
+# Copyright (C) 2009 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import sys
+from command import Command, MirrorSafeCommand
+from git_command import git
+from git_refs import HEAD
+
+class Version(Command, MirrorSafeCommand):
+  wrapper_version = None
+  wrapper_path = None
+
+  common = False
+  helpSummary = "Display the version of repo"
+  helpUsage = """
+%prog
+"""
+
+  def Execute(self, opt, args):
+    rp = self.manifest.repoProject
+    rem = rp.GetRemote(rp.remote.name)
+
+    print('repo version %s' % rp.work_git.describe(HEAD))
+    print('       (from %s)' % rem.url)
+
+    if Version.wrapper_path is not None:
+      print('repo launcher version %s' % Version.wrapper_version)
+      print('       (from %s)' % Version.wrapper_path)
+
+    print(git.version().strip())
+    print('Python %s' % sys.version)

tests/fixtures/gitc_config

@@ -0,0 +1 @@
+gitc_dir=/test/usr/local/google/gitc

tests/fixtures/test.gitconfig

@@ -0,0 +1,3 @@
+[section]
+	empty
+	nonempty = true

tests/test_git_config.py

@@ -0,0 +1,52 @@
+import os
+import unittest
+
+import git_config
+
+def fixture(*paths):
+  """Return a path relative to test/fixtures.
+  """
+  return os.path.join(os.path.dirname(__file__), 'fixtures', *paths)
+
+class GitConfigUnitTest(unittest.TestCase):
+  """Tests the GitConfig class.
+  """
+  def setUp(self):
+    """Create a GitConfig object using the test.gitconfig fixture.
+    """
+    config_fixture = fixture('test.gitconfig')
+    self.config = git_config.GitConfig(config_fixture)
+
+  def test_GetString_with_empty_config_values(self):
+    """
+    Test config entries with no value.
+
+    [section]
+        empty
+
+    """
+    val = self.config.GetString('section.empty')
+    self.assertEqual(val, None)
+
+  def test_GetString_with_true_value(self):
+    """
+    Test config entries with a string value.
+
+    [section]
+        nonempty = true
+
+    """
+    val = self.config.GetString('section.nonempty')
+    self.assertEqual(val, 'true')
+
+  def test_GetString_from_missing_file(self):
+    """
+    Test missing config file
+    """
+    config_fixture = fixture('not.present.gitconfig')
+    config = git_config.GitConfig(config_fixture)
+    val = config.GetString('empty')
+    self.assertEqual(val, None)
+
+if __name__ == '__main__':
+  unittest.main()

tests/test_wrapper.py

@@ -0,0 +1,75 @@
+#
+# Copyright (C) 2015 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import unittest
+
+import wrapper
+
+def fixture(*paths):
+  """Return a path relative to tests/fixtures.
+  """
+  return os.path.join(os.path.dirname(__file__), 'fixtures', *paths)
+
+class RepoWrapperUnitTest(unittest.TestCase):
+  """Tests helper functions in the repo wrapper
+  """
+  def setUp(self):
+    """Load the wrapper module every time
+    """
+    wrapper._wrapper_module = None
+    self.wrapper = wrapper.Wrapper()
+
+  def test_get_gitc_manifest_dir_no_gitc(self):
+    """
+    Test reading a missing gitc config file
+    """
+    self.wrapper.GITC_CONFIG_FILE = fixture('missing_gitc_config')
+    val = self.wrapper.get_gitc_manifest_dir()
+    self.assertEqual(val, '')
+
+  def test_get_gitc_manifest_dir(self):
+    """
+    Test reading the gitc config file and parsing the directory
+    """
+    self.wrapper.GITC_CONFIG_FILE = fixture('gitc_config')
+    val = self.wrapper.get_gitc_manifest_dir()
+    self.assertEqual(val, '/test/usr/local/google/gitc')
+
+  def test_gitc_parse_clientdir_no_gitc(self):
+    """
+    Test parsing the gitc clientdir without gitc running
+    """
+    self.wrapper.GITC_CONFIG_FILE = fixture('missing_gitc_config')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/something'), None)
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/test'), 'test')
+
+  def test_gitc_parse_clientdir(self):
+    """
+    Test parsing the gitc clientdir
+    """
+    self.wrapper.GITC_CONFIG_FILE = fixture('gitc_config')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/something'), None)
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/test'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/test/'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/test/extra'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/test'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/test/'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/test/extra'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/'), None)
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/'), None)
+
+if __name__ == '__main__':
+  unittest.main()

trace.py

@@ -13,13 +13,23 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-class Remote(object):
-  def __init__(self, name,
-               fetch=None,
-               review=None,
-               projectName=None):
-    self.name = name
-    self.fetchUrl = fetch
-    self.reviewUrl = review
-    self.projectName = projectName
-    self.requiredCommits = []
+from __future__ import print_function
+import sys
+import os
+REPO_TRACE = 'REPO_TRACE'
+
+try:
+  _TRACE = os.environ[REPO_TRACE] == '1'
+except KeyError:
+  _TRACE = False
+
+def IsTrace():
+  return _TRACE
+
+def SetTrace():
+  global _TRACE
+  _TRACE = True
+
+def Trace(fmt, *args):
+  if IsTrace():
+    print(fmt % args, file=sys.stderr)

wrapper.py

@@ -0,0 +1,30 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2014 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import imp
+import os
+
+
+def WrapperPath():
+  return os.path.join(os.path.dirname(__file__), 'repo')
+
+_wrapper_module = None
+def Wrapper():
+  global _wrapper_module
+  if not _wrapper_module:
+    _wrapper_module = imp.load_source('wrapper', WrapperPath())
+  return _wrapper_module