Commit Graph

467 Commits

Author SHA1 Message Date
Andrew Wheeler
4d5bb68d58 status: add -q/--quiet option
The --quiet option reduces the output to just
a list of projects with modified workspaces (and
orphans if -o is specified)

A common use case is when performing a full-workspace
merge.  The integrator will kick-off a merge via:

    repo forall -c git merge <some tag>

And then produce a short list of conflicted projects via:

    repo status -q

The integrator can then iteratively fix and clean up all conficted
components.  The merge is complete when:

    repo status -q

    returns no output.

Change-Id: Ibbba8713eac35befd8287c95948874e23fd5c7e2
2016-10-17 15:24:09 -05:00
Steve Rae
d648045366 implement optional 'pushurl' in the manifest file
Allow the 'remote' element in the manifest file to define an optional
'pushurl' attribute which is passed into the .git/config file.

Change-Id: If342d299d371374aedc4440645798888869c9714
Signed-off-by: Steve Rae <steve.rae@raedomain.com>
2016-09-20 15:31:20 +00:00
David Pursehouse
e01ee026e6 Merge "Fix submodule checkout error when using sync-s option" 2016-09-14 07:45:50 +00:00
Jonathan Nieder
71e4cea6de RepoHook: do not list options twice during hash based approval
Instead of

  Do you want to allow this script to run (yes/yes-never-ask-again/NO)? (yes/always/NO)?

ask

  Do you want to allow this script to run (yes/always/NO)?

Change-Id: I5f5a2d0e88086a8d85e54fb8623a62d74a20956a
Signed-off-by: Jonathan Nieder <jrn@google.com>
2016-08-18 17:06:26 -07:00
David Pursehouse
183c52ab02 Merge "project: Set config option to skip lfs smudge filter" 2016-08-17 01:00:12 +00:00
Mike Frysinger
40252c20f7 RepoHook: allow users to approve hooks via manifests
The constant prompting when registered hooks change can be tedious and
has a large multiplication factor when the project is large (e.g. the
AOSP).  It gets worse as people want to write more checks, hooks, docs,
and tests (or fix bugs), but every CL that goes in will trigger a new
prompt to approve.

Let's tweak our trust model when it comes to hooks.  Since people start
off by calling `repo init` with a URL to a manifest, and that manifest
defines all the hooks, anchor trust in that.  This requires that we get
the manifest over a trusted link (e.g. https or ssh) so that it can't
be MITM-ed.  If the user chooses to use an untrusted link (e.g. git or
http), then we'll fallback to the existing hash based approval.

Bug: Issue 226
Change-Id: I77be9e4397383f264fcdaefb582e345ea4069a13
2016-08-16 13:02:52 -04:00
David Pursehouse
76a4a9df86 project: Set config option to skip lfs smudge filter
During sync, repo runs `git read-tree --reset -u -v HEAD` which causes
git-lfs's smudge filter to run. However this fails because git-lfs does
not work with bare repositories.

Add lfs.filter configuration to the project config as suggested in the
comments on the upstream git-lfs client issue [1]. This prevents the
smudge filter from running, and the sync completes successfully.

For any projects that have LFS objects, `git lfs pull` must be executed.

[1] https://github.com/github/git-lfs/issues/1422

Bug: Issue 224
Change-Id: I091ff37998131e2e6bbc59aa37ee352fe12d7fcd
2016-08-16 21:55:36 +09:00
David Pursehouse
8016f60a46 Merge "repo: Repo does not always handle '.' parameter correctly" 2016-08-14 08:50:28 +00:00
Cheuk Leung
8ac0c96537 Fix removing broken symlink in reference dir
Re-ordered to first create the symlink before checking the source
file and remove the destination if the source does not exists.

Change-Id: Iae923ba2ef0ba5a8dc1b8e42d8cc3f3708f773af
2016-06-29 05:43:01 +00:00
David Pursehouse
a36af0767b Merge "Fix variable assignment" 2016-06-29 01:43:41 +00:00
Aymen Bouaziz
037040f73e Fix variable assignment
If upstream string is empty, current_branch_only variable will be assigned
to an empty string.

This is not what we expect here as this variable is a boolean.

Change-Id: Ibba935e25a74c2be1e50c88b4b403cf394ba365e
2016-06-28 12:31:25 +02:00
Aymen Bouaziz
2598ed06f1 Fix submodule checkout error when using sync-s option
When sync-s="true" option is used, the checkout of a submodule will try
to use the revision attribute of the parent project.

If this revision is a named reference, the checkout will fail if there
is no reference with this name in the submodule.

The proposed solution is to use the git commit id as revisionExpr for
submodules.

Change-Id: Ie8390a11957fd6a9c61289c6861d13cb3fa11678
2016-06-27 20:16:45 +02:00
Dan Willemsen
8b39fb4bc0 Merge "Fix XmlManifest.Save with remotes that have 'alias' set" 2016-04-22 18:28:57 +00:00
Dan Willemsen
96c2d65489 Fix XmlManifest.Save with remotes that have 'alias' set
When the alias attribute is set for a remote, the RemoteSpec attached to
a Project only contains the alias name used by git, not the original
name used in the manifest. But that's not enough information to
reconstruct the manifest, so save off the original manifest name as
another RemoteSpec parameter, only used to write the manifest out.

Bug: Issue 181
Bug: Issue 219
Change-Id: Id7417dfd6ce5572e4e5fe14f22924fdf088ca4f3
2016-04-22 10:32:06 +09:00
Sebastian Schuberth
7ecccf6225 diffmanifests: support custom git pretty format strings
Change-Id: I29f4f1351c421f393328514d145df1a96aed9ee2
2016-04-21 18:36:11 +00:00
Mark E. Hamilton
f9fe3e14d2 repo: Repo does not always handle '.' parameter correctly
The repo script allows a manifest to specify a '.' as the path the
top-level directory, which co-locates the .git and .repo directories,
and places files from the git repository at the top-level:

  <project name="proj_name" path="." />
  <project name="sierra.other.git" path="other" />

Most commands work correctly with this setup. Some commands, however,
fail to find the project. For instance, 'repo sync' works, and 'repo sync .'
works in a sub-project ('other' in this case) but 'repo sync .' in the
top-level directory fails with the error:

error: project . not found

There are two reasons for this:

1. The self.worktree attribute of the Project object is not normalized,
so with a '.' for path its value would be '/my/project/root/.'. This is
fine when used as a path, since it's the same path as '/my/project/root',
but when used in a string comparison it fails. This commit applies
os.path.normpath() to that value before storing it.

2. The _GetProjectByPath method in command.py was not checking the path
against manifest.topdir, so even once it was normalized the project was
not found. This commit adds a check against manifest.topdir if the
loop drops out without finding a project.

Change-Id: Ic84d053f1bbb5a357cad566805d5a326ae8246d2
2016-04-08 00:07:52 +00:00
Dan Willemsen
bdb866ea76 Fix symlinking of new projects
We weren't copying these lists, so the += was actually changing the
underlying lists.

When a new project was added to the manifest, we run _CheckDirReference
against the manifest project with share_refs=True, which added the
working_tree_* to the shareable_* lists. Then, when we load the new
manifest and create the new project, it uses the lists that already
contain the working_tree_* files, even though we passed
share_refs=False.

This happens reliably under the above conditions, but doesn't seem to
happen when syncing a fresh tree. So we've got a mixture of links that
may need to be cleaned up later. This patch will just stop it from
happening in the future.

Change-Id: Ib7935bfad78af1e494a75e55134ec829f13c2a41
2016-04-05 17:44:09 -07:00
David Pursehouse
936d6185eb Merge changes from topic 'pylint-pep8-cleanup'
* changes:
  command.py: Cleaned up pylint/pep8 violations
  project.py: Cleaned up pylint/pep8 violations
2016-03-15 00:31:12 +00:00
Mike Frysinger
4aa4b211c6 RepoHook: set __file__ when running the hook
A common design pattern is to use __file__ to find the location of the
active python module to assist in output or loading of related assets.
The current hook systems runs the pre-upload.py hook in a context w/out
that set leading to runtime errors:

$ repo upload --cbr .
ERROR: Traceback (most recent call last):
  File ".../repo/project.py", line 481, in _ExecuteHook
    self._script_fullpath, 'exec'), context)
  File ".../repohooks/pre-upload.py", line 32, in <module>
    path = os.path.dirname(os.path.realpath(__file__))
NameError: name '__file__' is not defined

Define this variable in this context so code can safely use it.

Change-Id: If6331312445fa61d9351b59f83abcc1c99ae6748
2016-03-05 21:52:31 +00:00
Mark E. Hamilton
30b0f4e022 project.py: Cleaned up pylint/pep8 violations
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a number of violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the project.py script.

This script now has a pylint score of 10.0, and no violations reported
by pep8.

Change-Id: I1462fd84f5b6b4c0dc893052671373e7ffd838f1
2016-03-02 09:05:40 -07:00
Dan Willemsen
1a799d14b7 Fix prune when bare git has detached head
We don't really use HEAD much in the bare git repositories, but there
have been reports of errors in git-symbolic-ref:

  symbolic-ref: fatal: Refusing to point HEAD outside of refs/

That happen when the bare git repo is in the detached head state. It's
possible that previous operations were killed while we were pruning
branches.

Use DetachHead instead of SetHead if we're restoring the repo into a
detached head state.

Change-Id: I9062e8957bc70367d3ded399685ac026fbb421fc
2015-12-15 14:22:40 -08:00
Dan Willemsen
e1e0bd1f75 Check for broken links when updating linkfiles
If a linkfile is a broken link (destination does not exist), and it
needs to be updated, we didn't notice that it needed to be removed
first. Use lexists instead of exists to check for this condition.

Change-Id: I1f6a1f0193d3fd2b9f7a647836044997f6ab32eb
2015-11-18 16:51:51 -08:00
David Pursehouse
74cfd2709b Sync: Add option to prune refs during sync
By passing --prune to the sync command, the --prune option is
given to the `git fetch`, causing refs that no longer exist on
the remote to be removed.

Change-Id: I3cedacce14276d96ac2d5aabf2d07fd05e92bc02
2015-10-27 03:04:17 +00:00
Dan Willemsen
029eaf3bac _CopyAndLinkFiles even if the sources haven't changed
The source or destination attributes may have changed even if the source
didn't, so we need to make sure that these are up to date.

Change-Id: I266ef3598ddda7e8c23bc9c6a049905ddc586348
2015-09-03 12:54:06 -07:00
Simran Basi
b9a1b73425 GITC: Add repo start support.
Add repo start support for GITC checkouts. If the user is in
the GITC FS view, they can now run repo start to check out
the sources and create a new working branch.

When "repo start" is called on a GITC project, the revision
tag is set to an empty string and saved in a new tag:
old-revision. This tells the GITC filesystem to display the
local copy of the sources when being viewed. The local copy
is created by pulling the project sources and the new branch
is created based off the original project revision.

Updated main.py to setup each command's gitc_manifest when
appropriate.

Updated repo sync's logic to sync opened projects and
updating the GITC manifest file for the rest.

Change-Id: I7e4809d1c4fc43c69b26f2f1bebe45aab0cae628
2015-08-28 10:53:05 -07:00
David Pursehouse
dc2545cad6 project.py: Improve message shown when hook is not replaced
If a hook file has been modified locally, it will not be replaced.

Improve the message to make this clearer.

Also change it from an error to a warning.

Change-Id: I62c635390f24d2868db17717c247861b0381c99f
2015-08-25 05:40:46 +00:00
David Pursehouse
f33929d014 project.py: Consistently use the _error method to print error messages
Use the _error method instead of directly calling `print`.

Also add a new _warn convenience method.

Change-Id: Ia332c14ef8d9d1fe2df128dbf36b5521802ccdf1
2015-08-25 14:39:06 +09:00
Dan Willemsen
7c9263bce0 Merge "Support smart-sync through persistent-http[s]" 2015-08-19 23:36:35 +00:00
Dan Willemsen
dab9e99f0f Merge "Fix formatting of message when retrying clone" 2015-08-19 17:54:50 +00:00
Dan Willemsen
c5f15bf7c0 Merge "Include project path in --force-sync error message" 2015-08-19 17:53:28 +00:00
Dan Willemsen
0745bb2657 Support smart-sync through persistent-http[s]
Use the same cookies and proxy that git traffic goes through for
persistent-http[s] to support authentication for smart-sync.

Change-Id: I20f4a281c259053a5a4fdbc48b1bca48e781c692
2015-08-19 10:22:11 -07:00
David Pursehouse
25857b8988 Fix formatting of message when retrying clone
Passing the force_sync variable into the string formatting results in
the message:

  "Retrying clone after deleting None"

or

  "Retrying clone after deleting True".

Pass the name of the git directory instead.

Also, move the print inside the if-block so it's only displayed
when the retry is actually going to be attempted.

Change-Id: I76d9ecc176cecee4ad512d13e9d1f6bd36aacbbb
2015-08-19 13:03:13 +00:00
Simon Ruggier
f9b7683a3b Include project path in --force-sync error message
For projects that have been cloned outside of the repo command (or
cloned a long time ago), commit abaa7f312f
introduced an error message to invite the user to use --force-sync.
However, due to the risk of data loss, it's useful to know which
project's git directory is being replaced before deciding whether or not
to provide --force-sync.

This change updates the exception's associated value to include the
project's relative path and explain to the user how they can resolve the
issue. A previous version of this commit used the project name. However,
for projects that have multiple work trees, the name can be ambiguous,
while the path clearly identifies which git directory will be replaced.

Change-Id: If717e66fda4d19accc0a8e889a91f4cd4ff14dff
2015-08-04 18:41:20 -04:00
Dan Willemsen
eeab6860f1 Fix shallow clone behavior
The existing code here makes sure that switching clone-depth from on to
off actually causes the history to be fully restored. Unfortunately, it
does this by fetching the full history every time the fetch spec
changes. Switching between two clone-depth="1" branches will fetch far
more than the top commit.

Instead, when not using clone-depth, pass --depth=2147483647 to git
fetch so that it ensures that we have the entire history. That is
slightly less efficient, so limit it to only when there are shallow
objects in the project by checking for the existance of the 'shallow'
file.

Change-Id: Iee0cfc9c6992c208344b1d9123769992412db67b
2015-08-03 16:54:16 -07:00
Dan Willemsen
2a3e15217a Fix _ReferenceGitDir symlinking
This fixes these errors:

  ...
  File ".repo/repo/project.py", line 2371, in _ReferenceGitDir
    os.symlink(os.path.relpath(src, os.path.dirname(dst)), dst)
  OSError: [Errno 17] File exists

Which was happening for checkouts that were created before v1.12.8, when
project-objects was created. Nothing had yet been forcing these
checkouts to use project-objects, until the recent verification changes.

In this OSError case, we already created the symlink, so src == dst, and
the directory did not exist. This caused us to run os.makedirs the
os.symlink on the same file.

dst really should be the file in gitdir, not the target of that symlink
if it exists. So just use realpath for the dotgit portion of the path.

Change-Id: Iff5396a2093de91029c42cf38aa57131fd22981c
2015-07-30 21:29:53 -07:00
Kevin Degi
abaa7f312f Add option to correct gitdir when syncing
In some cases, a user may wish to continue with a sync even though
it would require overwriting an existing git directory. This behavior
is not safe as a default because it could result in the loss of some
user data, but as an optional flag it allows the user more flexibility.

To support this, add a --force-sync flag to the sync command that will
attempt to overwrite the existing git dir if it is specified and the
existing git dir points to the wrong obj dir.

Change-Id: Ieddda8ad54e264a1eb4a9d54881dd6ebc8a03833
2015-07-29 14:44:46 -06:00
Conley Owens
7cccfb2cf0 Merge "InitGitDir: Clean up created directories" 2015-07-29 18:49:12 +00:00
Kevin Degi
b1a07b8276 InitGitDir: Clean up created directories
If _InitGitDir fails, it leaves any progress it had made on the file
system. This can cause subsequent calls to repo sync to behave
differently. This is especially evident when _CheckDirReference() fails,
since it will not be invoked when sync is retried because both the
source and destination directories already exist.

To address this, have _InitGitDir() clean up any directories it has created
if it catches an exception. Also behave the same way for _InitWorkTree().

Change-Id: Ic16bb3feea649e115b59bd44be294e89e3692aeb
2015-07-27 13:33:43 -06:00
David Pursehouse
4e16c24981 Revert "Add --prune option to fetch when syncing a mirror repo"
For some users it is not desirable to remove refs that don't exist
on the remote server when syncing a mirror repo.

This reverts commit b4d43b9f66.

Change-Id: Ie849b66682138ef88da6cd1a5fbb27e993197dd7
2015-07-20 22:31:04 +09:00
Conley Owens
b3d6e67196 Merge "Fail if gitdir does not point to objdir during sync" 2015-07-15 19:30:41 +00:00
Kevin Degi
679bac4bf3 project.RemoteFetch: Handle depth cases more robustly
The fetch logic for the case where depth is set and revision is a
SHA1 has several failure modes that are not handled well by the
current logic.

1) 'git fetch <SHA1>' requires git version >= 1.8.3
2) 'git fetch <SHA1>' can be prevented by a configuration option on the server.
3) 'git fetch --depth=<N> <refspec>' can fail to contain a SHA1 specified by
   the manifest.

Each of these cases cause infinite recursion when _RemoteFetch() tries to call
itself with current_branch_only=False because current_branch_only is set to
True when depth != None.

To try to prevent the infinite recursion, we set self.clone_depth to None
before the first retry of _RemoteFetch(). This will allow the Fetch to
eventually succeed in the case where clone-depth is specified in the manifest.
A user specified depth from the init command will still recurse infinitely.

In addition, never try to fetch a SHA1 directly if the git version being used
is not at least 1.8.3.

Change-Id: I802fc17878c0929cfd63fff611633c1d3b54ecd3
2015-07-15 15:53:14 +00:00
Wink Saville
4c426ef1d4 Teach _LinkFile._Link to handle globs.
This allows a project to use globs in the linkfile src attribute. When
a glob is used in the src the dest field must be a directory. Then
_LinkFile._Link(self) calls will create symbolic links in the dest
directory to all of the entries in the src as defined by the glob
specification.

Below all of the entries in master-configs/ will have symbolic links
in <root dir>/configs directory:

  <project name="helloworld.git" path="apps/helloworld">
      <linkfile src="master-configs/*" dest="configs"/>
  </project>

Change-Id: Idfed8fa47c83d2ca6e2b8e867731b8e2f9e2eb47
2015-06-03 08:05:17 -07:00
David Pursehouse
472ce9f5fa Merge changes I32da12c2,Ie4a65b3e
* changes:
  Skip sleep and retry if git remote update exits with a signal
  Catch exceptions in project list generator
2015-06-02 00:14:43 +00:00
Colin Cross
0184dcc510 Make linkfile symlinks relative
The source (target) of the symlink is specified relative to a project
within a tree, and the destination is specified relative to the top
of the tree, so it should always be possible to create a relative symlink
to the target file.  Relative symlinks will allow moving an entire tree
without breaking the symlink, and copying a tree (with -p) without leaving
a symlink to the old tree.

Change-Id: I16492a8b59a137d2abe43ca78e3b212e2c835599
2015-06-01 01:24:38 +00:00
Colin Cross
c4b301f988 Skip sleep and retry if git remote update exits with a signal
Pressing ctrl-c during repo sync often hangs for 30 to 45 seconds
due to the time.sleep and retry in _RemoteFetch.  If git exits with
a signal, for example -2 for SIGINT triggered by ctrl-c, skip the
sleep and retry.

Change-Id: I32da12c2dcc96d9cc0b12a066e824b12ebfb52a0
2015-05-13 18:11:34 +00:00
Kevin Degi
384b3c5948 Fail if gitdir does not point to objdir during sync
There are a set of cases that can cause the git directory in
.repo/projects to point to a directory in .repo/project-objects that
is not the one specified in the manifest. This results in a tree that
is not sane, and so should cause a failure.

In order to reproduce the failure case:
1) Sync to any manifest
2) Change the 'name' of a project to a different repository. Leave the
   'path' the same.
3) Resync the modified project. The project-objects directory will not
   be created, and the projects directory will remain pointed at the old
   project-objects.

Change-Id: Ie6711b1c773508850c5c9f748a27ff72d65e2bf2
2015-05-12 09:15:53 -06:00
David Pursehouse
35de228f33 Merge "Don't attempt to create "fully qualified names" for SHA1s" 2015-05-11 09:20:54 +00:00
David Pursehouse
ace097c36e Merge "Add option on sync to avoid fetching from remotes for existing sha1" 2015-05-01 07:51:52 +00:00
David Pursehouse
b155354034 Add option on sync to avoid fetching from remotes for existing sha1
In 2fb6466f79 an optimisation was
added to avoid fetching from remotes if the project is fixed to
a revision and the revision is already available locally.

This causes problems for users who expect all objects to be
fetched by default.

Change the logic so that the optimized behaviour is only enabled if
an option is explicitly given to repo sync.

Change-Id: I3b2794ddd8e0071b1787e166463cd8347ca9e24f
2015-04-30 14:29:02 +00:00
Alexandre Boeglin
382582728e Don't attempt to create "fully qualified names" for SHA1s
Doing so breaks "repo init -b <SHA1>".

Change-Id: Ic071a1b099a9125db22ea446d7e92e7854d69b37
2015-04-30 14:54:47 +02:00
David Pursehouse
b4d43b9f66 Add --prune option to fetch when syncing a mirror repo
When syncing a mirror repo, add the --prune option to the fetch
command to force removal of stale refs from the mirror.

Change-Id: I4b43b2a5c86b9915627887c16f6569066f3ab978
2015-04-30 10:32:37 +09:00
David Pursehouse
403b64edf4 Don't append branch to fetch spec when syncing to a mirror
Appending the branch to the fetch spec causes sync of a mirror to
fail for projects that don't have an explicit revision specified,
and don't have a branch of the same name as the default revision.

For example, a manifest defining a default revision:

 <default revision="master">

having a project without an explicit revision:

 <project name="path/to/project">

and not having a branch named "master", will cause repo sync to
fail for that project with the error:

 Couldn't find remote ref refs/heads/master

Modify the logic to not append the branch onto the fetch spec when
syncing to a mirror.

Change-Id: I5c4457bd125519abf27abe682dea62ad708978c9
2015-04-27 10:56:27 +09:00
Conley Owens
04f2f0e186 Maintain fully qualified tracking branches
When running repo branch, the git merge line (in many circumstances)
is set to the revision of the project specified in the manifest.  If
this is a branch name that is not fully-qualified, we will end up with
something like "merge = master" instead of "merge = refs/heads/master".
This change examines the revision if we are going to use that and
changes branch short names to fully qualified branch names.

Change-Id: Ie1be94fb8d45df8eeac44a47f729a3819a05fa81
2015-04-01 17:43:36 +00:00
Anthony King
23ff7df6a7 use the max depth instead of unshallow
This allows the use of older versions of git

Change-Id: I88ea685066603af19896a791829355ddbfa91ffe
2015-03-30 21:54:26 +00:00
John L. Villalovos
9c76f67f13 Always capture output for GitCommand
Switch the GitCommand program to always capture the output for stdout
and stderr.  And by default print the output while running.

The options capture_stdout and capture_stderr have effectively become
options to supress the printing of stdout and stderr.

Update the 'git fetch' to use '--progress' so that the progress messages
will be displayed.  git checks if the output location isatty() and if it
is not a TTY it will by default not print the progress messages.

Change-Id: Ifdae138e008f80a59195f9f43c911a1a5210ec60
2015-03-26 11:43:55 -07:00
Jonathan Nieder
9371979628 Revert "Implementation of manifest defined githooks"
This reverts commit 38e4387f8e.

A "repo init" followed by "repo sync" is meant to be as safe as
"git clone".  In particular it should not run arbitrary code provided
by the manifest owner.

It would still be nice to have support for manifest-defined git hooks
--- they'd just need a prompt like the upload RepoHook has.  Hopefully
a later change can bring them back.

Change-Id: I5ecd90fb5c2ed64f103d856d1ffcba38a47b062d
Signed-off-by: Jonathan Nieder <jrn@google.com>
2015-03-17 11:29:58 -07:00
Jakub Vrana
0402cd882a Add space between project path and branch in repo status.
Currently, paths longer than 39 chars have no space after them so it looks
like this:

project path/branch master

Change-Id: I4c1bb13648ac099ade8a8d4ebafa04131571f842
2015-03-11 07:42:17 +00:00
Conley Owens
85e8267031 Merge "Implementation of manifest defined githooks" 2015-03-05 20:52:30 +00:00
John L. Villalovos
e30f46b957 Print stderr output from git command for RemoteFetch
The stderr output generated by git during a RemoteFetch was not being
printed.  This information is useful so print it.

Change-Id: I6e6ce12c4a57e5ca2359f76ce14f2fcbbc37a5ef
2015-02-25 14:29:28 -08:00
Conley Owens
e4978cfbe3 Ensure the repo project is never fetched with partial depth
If the repo project is synced with partial depth, then the tags
won't be fetched and users will be told the newest sha1 in the
stable branch isn't signed.

Change-Id: I107df97b4836b928c76aa33a700fa35d1705ae09
2015-02-10 14:44:05 -08:00
John L. Villalovos
126e298214 Handle case where 'git remote prune' needs to be run
Handle the case when this error occurs:
    error: some local refs could not be updated; try running
     'git remote prune origin' to remove any old, conflicting branches

This is usually caused by a reference getting changed from a file to a
directory.

For example:
  Initially someone creates a branch 'foo' and it is stored as:
    .git/refs/remotes/origin/foo

  Then later on it is decided to change the layout structure where 'foo'
  is a directory with branches below it:
    .git/refs/remotes/origin/foo/master

  The problem occurs when someone still has
  '.git/refs/remotes/origin/foo' on their system and does a repo sync.
  When this occurs the error message for needing to do a
  'git remote prune origin' occurs.

Now when doing a 'git fetch' if the error message from git says that a
'git remote prune' is needed, it will do the prune and then retry the
fetch.

Change-Id: I4c6f5aa6bd932f0ef7a39134400bedd52e82f633
Signed-off-by: John L. Villalovos <john.l.villalovos@intel.com>
2015-02-03 13:49:51 -08:00
Jimmie Wester
38e4387f8e Implementation of manifest defined githooks
When working within a team or corporation it is often
useful/required to use predefined git templates. This
change teaches repo to use a per-remote git hook template
structure.

The implementation is done as a continuation of the
existing projecthook functionality. The terminology is
therefore defined as projecthooks.

The downloaded projecthooks are stored in the .repo
directory as a metaproject separating them from the users
project forest.

The projecthooks are downloaded and set up when doing a
repo init and updated for each new repo init.

When downloading a mirror the projecthooks gits are
not added to the bare forest since the intention is to
ensure that the latest are used (allows for company policy
enforcement).

The projecthooks are defined in the manifest file in the
remote element as a subnode, the name refers to the
project name on the server referred to in the remote.
<remote name="myremote ..>
   <projecthook name="myprojecthookgit" revision="myrevision"/>
</remote>

The hooks found in the projecthook revision supersede
the stock hooks found in repo. This removes the need for
updating the projecthook gits for repo stock hook changes.

Change-Id: I6796b7b0342c1f83c35f4b3e46782581b069a561
Signed-off-by: Patrik Ryd <patrik.ryd@stericsson.com>
Signed-off-by: Ian Kumlien <ian.kumlien@gmail.com>
2015-02-03 16:01:15 +09:00
Conley Owens
db6f1b0884 Merge "Use depth flag when fetching" 2015-01-30 19:36:06 +00:00
Conley Owens
ee69084421 Merge "Handle shallow checkout of SHA1 pinned repos" 2015-01-28 20:29:37 +00:00
Conley Owens
d37d43f036 Merge "Don't delete hooks in .git/hooks" 2015-01-28 20:29:05 +00:00
Anthony King
7bdac71087 pylint fixes for project.py
Fix all the formatting warnings and unused variables

Change-Id: I17d88a23572303879530077f3a80451de5417fbb
2015-01-22 04:20:21 +00:00
Conley Owens
f97e8383a3 Use depth flag when fetching
Currently, we only use the depth flag when cloning.  The result is that when
new project history has merges, the entire history of the merged branch is
brought in and the project becomes unshallow very quickly.  --depth and
clone-depth are often used to save on space, not just network load, so this
seems less than ideal.

This change uses --depth on every fetch (when the user has depth specified),
not just the initial clone.  The result is that the given project stays
consistently shallow as opposed to growing over time, especially when merges
are involved.

Change-Id: Iac706cfdad4a555c72f9d9f1119195d38d91df12
2015-01-22 01:20:22 +00:00
Bertrand SIMONNET
3000cdad22 Handle shallow checkout of SHA1 pinned repos
When doing a shallow checkout SHA1 pinned repos with repo init --depth=1 and
repo sync -c, repo would try to fetch only some reference and fail if the exact
SHA1 repo was missing.
Instead, when depth is set, fetch only the specific commit.

Change-Id: If3f799d0e78c03faea47f796380bb5e367b11998
2015-01-21 14:14:23 -08:00
Mitchel Humpherys
b9d9efd394 Don't delete hooks in .git/hooks
We currently delete all hooks in .git/hooks for each project before
symlink'ing in the standard project hooks.  This can be annoying for
users who have installed custom git hooks.

There's no reason to delete all existing hooks.  Just rip out the
deletion code.

Change-Id: I5062a6cd20af700f6d6a17b11ad6c94853987c57
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
2015-01-15 22:49:08 -08:00
Dave Borowitz
497bde4de5 Respect --quiet when looking up bundle cookie file
Change-Id: I02a244132c49e4bb50ecda978974d6d2b220f6d1
2015-01-02 13:58:05 -08:00
Dave Borowitz
4abf8e6ef8 Save cookies back to jar when fetching clone.bundle
Change-Id: I3ef71b5e7f8ee1cda66057e46ae234866c7258c4
2015-01-02 13:57:14 -08:00
Dave Borowitz
137d0131bf Hold persistent proxy connection open while fetching clone.bundle
The persistent proxy may choose to present a per-process cookie file
that gets cleaned up after the process exits, to help with the fact
that libcurl cannot save cookies atomically when a cookie file is
shared across processes. We were letting this cleanup happen
immediately by closing stdin as soon as we read the configuration
option, resulting in a nonexistent cookie file by the time we use the
config option.

Work around this by converting the cookie logic to a context manager
method, which closes the process only when we're done with the cookie
file.

Change-Id: I12a88b25cc19621ef8161337144c1b264264211a
2015-01-02 13:57:13 -08:00
Kris Giesing
c8d882ae2a Silence warnings about invalid clone.bundle files when quieted
The invalid clone.bundle file warning is not typically user actionable,
and can be confusing. So don't show it when -q flag is in effect.

Change-Id: If9fef4085391acf54b63c75029ec0e161c38eb86
2014-12-24 10:23:24 +09:00
Conley Owens
3eb87cec5c Revert "Check for existence of refs upon initial fetch"
This reverts commit 565480588d.

We are reverting this change for 2 reasons:

1) It introduced a bug for users using sync -c with a reference mirror.
2) The fetch specs have recently changed to cause git to properly fail
when we request a non-existent branch of a manifest, removing the need
for this change.

Change-Id: I0f63da9bfb40cf5ffafb7979f1b8c929a738fc7b
2014-11-10 23:49:32 +00:00
Conley Owens
7893b85509 Merge changes I1f71be22,I5b119f11
* changes:
  Always fetch the specific revision given
  Support specifying non-HEADS refs as upstream
2014-10-22 00:23:18 +00:00
Vadim Bendebury
14e134da02 upload: report names of uncommitted files
When there are uncommitted files in the tree, 'repo upload' stops to
ask if it is OK to continue, but does not report the actual names of
uncommitted files.

This patch adds plumbing to have the outstanding file names reported
if desired.

BUG=None
TEST=verified that 'repo upload' properly operates with the following
    conditions present in the tree:
    . file(s) modified locally
    . file(s) added to index, but not committed
    . files not known to git
    . no modified files (the upload proceeds as expected)

Change-Id: If65d5f8e8bcb3300c16d85dc5d7017758545f80d
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Signed-off-by: Vadim Bendebury <vbendeb@google.com>
2014-10-14 11:20:05 -07:00
Nasser Grainawi
04e52d6166 Always fetch the specific revision given
Don't assume the revision is in refs/heads/.

Change-Id: I1f71be222ed3ed940d2265aad43d1f2d601fc03a
2014-10-09 13:41:56 -06:00
Nasser Grainawi
909d58b2e2 Support specifying non-HEADS refs as upstream
While not typical, some users might have an upstream that isn't in
the usual refs/heads/* namespace. There's no reason not to use
those refs as the value for the upstream attribute, so support
doing so.

Change-Id: I5b119f1135c3268c20e7c4084682e860d3ee1fb1
2014-10-09 13:41:51 -06:00
Chirayu Desai
303a82f33a Don't open non-binary files as binary
* Don't pen the git config file, and the git ".lock" file as binary.

Change-Id: I7b3939658456f2fd0a0500443cdd8d1ee1a4459d
2014-08-19 23:05:44 +05:30
Conley Owens
80b87fe6c1 Use fetch --unshallow when appropriate.
If a user reinits to a different manifest or the manifest updates so
that a project no longer has a fixed depth, we need to use --unshallow
when we fetch.

Change-Id: I6d3f15e5464b5eaad9205654bc24354947a78aea
2014-05-09 18:47:35 -07:00
Conley Owens
e9f75b1782 Merge "Enable remotes to define their own revision" 2014-05-08 18:38:33 +00:00
Conley Owens
5196805fa2 Merge "Use exec() rather than execfile()" 2014-05-07 18:18:56 +00:00
Anthony King
36ea2fb6ee Enable remotes to define their own revision
Some projects use multiple remotes.
In some cases these remotes have different naming conventions.
Add an option to define a revision in the remote configuration.

The `project` revision takes precedence over `remote` and `default`.
The `remote` revision takes precedence over `default`.
The `default` revision acts as a fall back as it originally did.

Change-Id: I2b376160d45d48b0bab840c02a3eef1a1e32cf6d
2014-05-07 08:29:30 +00:00
Anthony King
2cd1f0452e Use next(iterator) rather than iterator.next()
iterator.next() was replaced with iterator.__next__() in Python 3.
Use next(iterator) instead which will select the correct method for
returning the next item.

Change-Id: I6d0c89c8b32e817e5897fe87332933dacf22027b
2014-05-07 08:44:20 +01:00
Anthony King
70f6890352 Use exec() rather than execfile()
execfile() is not in Python 3.

Change-Id: I5af222340f13c1e8edaa820e7675d3e4d62a1689
2014-05-05 23:41:07 +01:00
Conley Owens
666d534636 Ensure HEAD is correct when skipping remote fetch
A recent optimization (2fb6466f79) skips
performing a remote fetch if we already know we have the sha1 we want.
However, that optimization skipped initialization steps that ensure HEAD
points to the correct sha1.  This change makes sure not to skip those
steps.

Here is an example of how to test this change:

"""""""""
url=<manifest url>
branch1=<branch name>
branch2=<branch name>
project=<project with revision set to different sha1 in each branch>

repo init -u $url -b $branch1 --mirror
repo sync $project
first=$(cd $project.git; git rev-parse HEAD)

repo init -b $branch2
repo sync $project
second=$(cd platform/build.git; git rev-parse HEAD)

if [[ $first == $second ]]
then
    echo 'problem!'
else
    echo 'no problem!'
fi
"""""""""
2014-05-01 13:20:32 -07:00
Conley Owens
f2af756425 Add 'shallow' gitfile to symlinks
This fixes the bug that kept clients from doing things like `git log`
in projects using the clone-depth feature.

Change-Id: Ib4024a7b82ceaa7eb7b8935b007b3e8225e0aea8
2014-04-30 11:34:00 -07:00
Conley Owens
544e7b0a97 Merge "Ignore clone-depth attribute when fetching to a mirror" 2014-04-24 21:21:02 +00:00
Jeff Hamilton
e0df232da7 Add linkfile support.
It's just like copyfile and runs at the same time as copyfile but
instead of copying it creates a symlink instead.  This is needed
because copyfile copies the target of the link as opposed to the
symlink itself.

Change-Id: I7bff2aa23f0d80d9d51061045bd9c86a9b741ac5
2014-04-22 14:35:47 -05:00
David Pursehouse
9bc422f130 Ignore clone-depth attribute when fetching to a mirror
If a manifest includes projects with a clone-depth=1 attribute, and a
workspace is initialised from that manifest using the --mirror option,
any workspaces initialised and synced from the mirror will fail with:

  fatal: attempt to fetch/clone from a shallow repository

on the projects that had the clone-depth.

Ignore the clone-depth attribute when fetching from the remote to a
mirror workspace. Thus the mirror will be synched with a complete
clone of all the repositories.

Change-Id: I638b77e4894f5eda137d31fa6358eec53cf4654a
2014-04-16 11:00:40 +09:00
Mitchel Humpherys
26c45a7958 Make --no-tags work with -c
Currently, the --no-tags option is ignored if the user asks to only
fetch the current branch. There is no reason for this restriction. Fix
it.

Change-Id: Ibaaeae85ebe9955ed49325940461d630d794b990
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
2014-03-12 16:34:53 +09:00
David Pursehouse
68425f4da8 Fix indentation in project.py
Change-Id: I81c630536eaa54d5a25b9cb339a96c28619815ea
2014-03-11 14:55:52 +09:00
Dan Sandler
53e902a19b More verbose errors for NoManifestExceptions.
The old "manifest required for this command -- please run
init" is replaced by a more helpful message that lists the
command repo was trying to execute (with arguments) as well
as the str() of the NoManifestException. For example:

> error: in `sync`: [Errno 2] No such file or directory:
> 	'path/to/.repo/manifests/.git/HEAD'
> error: manifest missing or unreadable -- please run init

Other failure points in basic command parsing and dispatch
are more clearly explained in the same fashion.

Change-Id: I6212e5c648bc5d57e27145d55a5391ca565e4149
2014-03-11 05:33:43 +00:00
Chris AtLee
2fb6466f79 Don't fetch from remotes if commit id exists locally
In existing workspaces where the manifest specifies a commit id in the
manifest, we can avoid doing a fetch from the remote if we have the
commit locally. This substantially improves sync times for fully
specified manifests.

Change-Id: Ide216f28a545e00e0b493ce90ed0019513c61613
2014-03-03 10:17:03 +00:00
Julien Campergue
dd6542268a Add the "diffmanifests" command
This command allows a deeper diff between two manifest projects.
In addition to changed projects, it displays the logs of the
commits between both revisions for each project.

Change-Id: I86d30602cfbc654f8c84db2be5d8a30cb90f1398
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
2014-02-17 11:20:11 +00:00
Conley Owens
565480588d Check for existence of refs upon initial fetch
When we do an initial fetch and have not specified any branch etc,
the following fetch command will not error:
git fetch origin --tags +refs/heads/*:refs/remotes/origin/*

In this change we make sure something got fetched and if not we report
an error.

This fixes the bug that occurs when we init using a bad manifest url and
then are unable to init again (because a manifest project has been
inited with no manifest).

Change-Id: I6f8aaefc83a1837beb10b1ac90bea96dc8e61156
2014-02-12 09:11:00 -08:00
Conley Owens
c9877c7cf6 Merge "Only fetch current branch on shallow clients" 2014-01-29 21:12:34 +00:00
Shawn Pearce
69e04d8953 Only fetch current branch on shallow clients
Fetching a new branch on a shallow client may download the entire
project history, as the depth parameter is not passed to git
fetch. Force the fetch to only download the current branch.

Change-Id: Ie17ce8eb5e3487c24d90b2cae8227319dea482c8
2014-01-29 12:48:54 -08:00
Conley Owens
e695338e21 Merge "repo: Support multiple branches for the same project." 2014-01-10 01:20:13 +00:00
Conley Owens
bd80f7eedd Merge "Canonicalize project hooks path before use" 2014-01-09 02:11:10 +00:00
Julien Campergue
335f5ef4ad Add --archive option to init to sync using git archive
This significantly reduces sync time and used brandwidth as only
a tar of each project's revision is checked out, but git is not
accessible from projects anymore.

This is relevant when git is not needed in projects but sync
speed/brandwidth may be important like on CI servers when building
several versions from scratch regularly for example.

Archive is not supported over http/https.

Change-Id: I48c3c7de2cd5a1faec33e295fcdafbc7807d0e4d
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
2013-12-10 08:27:07 +00:00
Jesse Hall
672cc499b9 Canonicalize project hooks path before use
If the top-level .repo directory is moved somewhere else (e.g. a
different drive) and replaced with a symlink, _InitHooks() will create
broken symlinks. Resolving symlinks before computing the relative path
for the symlink keeps the path within the repo tree, so the tree can
be moved anywhere.

Change-Id: Ifa5c07869e3477186ddd2c255c6c607f547bc1fe
2013-12-03 09:02:16 -08:00
Conley Owens
cbc0798f67 Fix print of git-remote-persistent-https error
If git-remote-persistent-https fails, we use an iter() and then
subsequently a .read() on stderr.  Python doesn't like this and
gives the following error message:
ValueError: Mixing iteration and read methods would lose data

This change removes the use of iter() to avoid the issue.

Change-Id: I980659b83229e2a559c20dcc7b116f8d2476abd5
2013-11-21 10:38:03 -08:00
Chirayu Desai
0eb35cbe50 Fix some python3 encoding issues
* Add .decode('utf-8') where needed
* Add 'b' to `open` where needed, and remove where unnecessary

Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
2013-11-21 06:03:22 +00:00
David James
8d20116038 repo: Support multiple branches for the same project.
It is often useful to be able to include the same project more than
once, but with different branches and placed in different paths in the
workspace. Add this feature.

This CL adds the concept of an object directory. The object directory
stores objects that can be shared amongst several working trees. For
newly synced repositories, we set up the git repo now to share its
objects with an object repo.

Each worktree for a given repo shares objects, but has an independent
set of references and branches. This ensures that repo only has to
update the objects once; however the references for each worktree are
updated separately. Storing the references separately is needed to
ensure that commits to a branch on one worktree will not change the
HEAD commits of the others.

One nice side effect of sharing objects between different worktrees is
that you can easily cherry-pick changes between the two worktrees
without needing to fetch them.

Bug: Issue 141
Change-Id: I5e2f4e1a7abb56f9d3f310fa6fd0c17019330ecd
2013-10-14 15:34:32 -07:00
Conley Owens
edd0151a26 Accept all UTF-8 committer names
Change-Id: I7d9d49a8bacf2dc332614d26cdfcc905be7a5290
2013-09-27 00:35:35 +00:00
Dave Borowitz
0836a22d38 Wait for git-remote-persistent-https -print_config to exit
Change-Id: I5ab96e7c8575682217d440ddc52ecfdc8c35f179
2013-09-25 17:46:01 -07:00
Chirayu Desai
033a7e91de DownloadPatchSet: fetch the change only, and nothing else.
* Previously, it would run `git fetch <remote.name> <change refspec>
  <remote.fetch>, which would fetch all the branches, even if 'sync-c'
  was set to true in the manifest.
* Fix that, since all it needs to fetch is the change that was asked
  for, and nothing else.
* For some more info, refer to the discussion on:
  I42a9d419b51f5da03f20a640ea68993cda4b6500

Change-Id: Ibc801695d56fc16e56f999e0f61393f54461785f
2013-06-13 21:14:48 +05:30
Dave Borowitz
74c1f3d5e6 Read cookie file from git-remote-persistent-https if applicable
git-remote-persistent-https proxy implementations may pass cookie file
configuration to git-remote-https. When fetching bundles for
persistent-http(s) URLs, use the -print_config flag (if supported) to
extract this information from the proxy binary and pass it to curl,
overriding http.cookiefile from .gitconfig.

This adds a few ms overhead per clone.bundle fetch, which should be
acceptable since it happens only on the initial clone, which takes
much longer anyway.

Change-Id: I03be8ebaf8d3005855d33998cd8ecd412e8ec287
2013-06-04 00:12:15 +00:00
Dave Borowitz
91f3ba5a3f Ensure clone.bundle files have proper header
Server auth middleware may return a 200 from a clone.bundle request
that is not a bundle file, but instead a login or access denied page.
Instead of just checking the file size, actually check the first few
bytes of the file to ensure it is a bundle file before proceeding.

Change-Id: Icea07567c568a24fd838e5cf974c58f9e4abd7c0
2013-06-04 00:12:01 +00:00
Bryan Jacobs
f609f91b72 Send reviews to a different branch from fetch
This adds the ability to have reviews pushed to a different branch
from the one on which changes are based. This is useful for "gateway"
systems without smartsync.

Change-Id: I3a8a0fabcaf6055e62d3fb55f89c944e2f81569f
2013-05-24 12:17:22 -04:00
David Pursehouse
59bbb580e3 Move Python version checking to a separate module
Add a new module with methods for checking the Python version.

Instead of handling Python3 imports with try...except blocks, first
check the python version and then import the relevant modules.  This
makes the code a bit cleaner and will result in less diff when/if we
remove support for Python < 3 later.

Use the same mechanism to handle `input` vs. `raw_input` and add
suppression of pylint warnings caused by redefinition of the built-in
method `input`.

Change-Id: Ia403e525b88d77640a741ac50382146e7d635924
Also-by: Chirayu Desai <cdesai@cyanogenmod.org>
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
2013-05-23 07:28:53 +00:00
Victor Boivie
2b30e3aaba Use reference also for manifest git
When running 'repo init --reference=<mirror>', the mirror will be
used for all projects except the manifest project. This is because
the _InitGitDir function uses the 'repo.reference' git config
value specified in the manifest git, which has no effect when
creating the manifest git as that value will be set after the git
has been successfully cloned.

Information about where the manifest git is located on the server
is only known when performing the 'repo init', so that information
has to be provided when cloning the git in order for it to set up
a proper mapping.

Change-Id: I47a2c8b3267a4065965058718ce1def4ecb34d5a
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
2013-05-12 17:49:28 +05:30
David Pursehouse
793f90cdc0 Merge "Re-initialise repos git hooks when updating the forest" 2013-05-09 08:36:12 +00:00
Jimmie Wester
2f992cba32 Repo should not fetch tags for shallow projects
Fetching all tags for a shallow git results in an
inconstent git and forces git to fetch more than
the depth specified.

This change teaches repo not to fetch any tags in a
repository initialised with the depth option.

Change-Id: I557ead9f88fa0d6a19b1cb55b23bba1100fcb8f2
Signed-off-by: Patrik Ryd <patrik.ryd@stericsson.com>
2013-05-08 06:54:10 +01:00
Chirayu Desai
217ea7d274 Some fixes for supporting python3
* Fix imports.
* Use python3 syntax.
* Wrap map() calls with list().
* Use list() only wherever needed.
  (Thanks Conley!)
* Fix dictionary iteration methods
  (s/iteritems/items/).
* Make use of sorted() in appropriate places
* Use iterators directly in the loop.
* Don't use .keys() wherever it isn't needed.
* Use sys.maxsize instead of sys.maxint

TODO:
* Make repo work fully with python3. :)

Some of this was done by the '2to3' tool [1], by
applying the needed fixes in a way that doesn't
break compatibility with python2.

Links:
[1]: http://docs.python.org/2/library/2to3.html

Change-Id: Ibdf3bf9a530d716db905733cb9bfef83a48820f7
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
2013-04-18 21:35:49 +05:30
David Pursehouse
ede7f12d4a Allow clone depth to be specified per project
If the clone-depth attribute is set on a project, its value will
be used to set the depth when fetching the git.  The value, if
given, must be a positive integer.

The value in the clone-depth attribute overrides any value given to
repo init via the --depth command line option.

Change-Id: I273015b3724213600b63e40cca4cafaa9f782ddf
2013-04-10 09:17:50 +09:00
David Holmer
0a1c6a1c16 Special handling for manifest group "default"
Change Details:
* Make "default" a special manifest group that matches any project that
  does not have the special project group "notdefault"
* Use "default" instead of "all,-notdefault" when user does not specify
  manifest group
* Expand -g option help to include example usage of manifest groups

Change Benefits:
* Allow a more intuitive and expressive manifest groups specification:
  * "default" instead of "all,-notdefault"
  * "default,foo" instead of "all,-notdefault,foo"
  * "default,-foo" instead of "all,-notdefault,-foo"
  * "foo,-default" which has no equivalent
* Default manifest groups behavior can be restored by the command
  'repo init -g default'. This is significantly more intuitive than the
  current equivalent command 'repo init -g all,-notdefault'.

Change-Id: I6d0673791d64a650110a917c248bcebb23b279d3
2013-04-03 22:27:45 +00:00
Jimmie Wester
a0444584cb Re-initialise repos git hooks when updating the forest
Repo now re-initialises the git-hooks reference directory
when updating the forest. This allows for any new template
files to be made available throughout the project forest
when updating the forest. Previous functionality required
the user to recreate the forest.

Change-Id: I9051265028a9e77d6139791547fff095bc195077
Signed-off-by: Patrik Ryd <patrik.ryd@stericsson.com>
2013-03-08 09:09:04 +01:00
Shawn Pearce
45d21685b9 upload: support --re and --cc options over HTTP
HTTP can't use the older style of passing options as part of
the git receive-pack command line. Use the new style as defined
by https://gerrit-review.googlesource.com/42652 when connecting
over HTTP.

If the Gerrit server is too old to understand the % option
syntax used here one of two outcomes is possible:

- If no topic name was sent the server will fail with an error
  message. This happens because the user tried to do an upload to
  "refs/for/master%r=alice", and the branch does not exist.
  The user can delete the options and retry the upload.

- If a topic was set the options will be read as part of the
  topic string and shown on the change page in the topic field.

Either outcome is slightly better than the current behavior of
just dropping the data on the floor and forgetting whatever the
user tried to supply.

Change-Id: Ib2df62679e5bf3ee93d6b18c12ab6474f96d9106
2013-02-28 12:10:31 -08:00
Mitchel Humpherys
597868b4c4 Add --no-tags option to prevent fetching of tags
Add an option to pass `--no-tags' to `git fetch'.

Change-Id: I4158cc369773e08e55a167091c38ca304a197587
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
2013-02-27 11:00:49 +09:00
Torne (Richard Coles)
ed68d0e852 Check for a cookie file when fetching clone.bundle.
If the user's git config specifies a cookie file for HTTP, use it when
fetching clone.bundle, as it may contain the required login credentials
to get access (e.g. when used with Compute Engine service accounts).

Change-Id: I12ee9e38694822ef1de4ed62138c3876c43f431b
2013-01-11 16:22:54 +00:00
Che-Liang Chiou
b2bd91c99b Represent git-submodule as nested projects, take 2
(Previous submission of this change broke Android buildbot due to
 incorrect regular expression for parsing git-config output.  During
 investigation, we also found that Android, which pulls Chromium, has a
 workaround for Chromium's submodules; its manifest includes Chromium's
 submodules.  This new change, in addition to fixing the regex, also
 take this type of workarounds into consideration; it adds a new
 attribute that makes repo not fetch submodules unless submodules have a
 project element defined in the manifest, or this attribute is
 overridden by a parent project element or by the default element.)

We need a representation of git-submodule in repo; otherwise repo will
not sync submodules, and leave workspace in a broken state.  Of course
this will not be a problem if all projects are owned by the owner of the
manifest file, who may simply choose not to use git-submodule in all
projects.  However, this is not possible in practice because manifest
file owner is unlikely to own all upstream projects.

As git submodules are simply git repositories, it is natural to treat
them as plain repo projects that live inside a repo project.  That is,
we could use recursively declared projects to denote the is-submodule
relation of git repositories.

The behavior of repo remains the same to projects that do not have a
sub-project within.  As for parent projects, repo fetches them and their
sub-projects as normal projects, and then checks out subprojects at the
commit specified in parent's commit object.  The sub-project is fetched
at a path relative to parent project's working directory; so the path
specified in manifest file should match that of .gitmodules file.

If a submodule is not registered in repo manifest, repo will derive its
properties from itself and its parent project, which might not always be
correct.  In such cases, the subproject is called a derived subproject.

To a user, a sub-project is merely a git-submodule; so all tips of
working with a git-submodule apply here, too.  For example, you should
not run `repo sync` in a parent repository if its submodule is dirty.

Change-Id: I4b8344c1b9ccad2f58ad304573133e5d52e1faef
2012-11-19 10:45:21 -08:00
Conley Owens
75ee0570da Raise a NoManifestException when the manifest DNE
When a command (eg, `repo forall`) expects the manifest project to
exist, but there is no manifest, an IOException gets raised.  This
change defines a new Exception type to be raised in these cases and
raises it when project.py fails to read the manifest.

Change-Id: Iac576c293a37f7d8f60cd4f6aa95b2c97f9e7957
2012-11-15 18:50:11 -08:00
David Pursehouse
8f62fb7bd3 Tidy up code formatting a bit more
Enable the following Pylint warnings:

  C0322: Operator not preceded by a space
  C0323: Operator not followed by a space
  C0324: Comma not followed by a space

And make the necessary fixes.

Change-Id: I74d74283ad5138cbaf28d492b18614eb355ff9fe
2012-11-14 12:09:38 +09:00
David Pursehouse
c1b86a2323 Fix inconsistent indentation
The repo coding style is to indent at 2 characters, but there are
many places where this is not followed.

Enable pylint warning "W0311: Bad indentation" and make sure all
indentation is at multiples of 2 characters.

Change-Id: I68f0f64470789ce2429ab11104d15d380a63e6a8
2012-11-14 11:38:57 +09:00
David Pursehouse
98ffba1401 Fix: "Statement seems to have no effect"
Pylint raises an error on the call:

  print

Change it to:

 print()

Change-Id: I507e1b3dd928fa6c32ea7e86260fb3d7b1428e6f
2012-11-14 11:38:57 +09:00
Sarah Owens
cecd1d864f Change print statements to work in python3
This is part of a series of changes to introduce Python3 support.

Change-Id: I373be5de7141aa127d7debdbce1df39148dbec32
2012-11-13 17:33:56 -08:00
David Pursehouse
7e6dd2dff0 Fix pylint warning W0108: Lambda may not be necessary
Remove unnecessary usage of lambda.

Change-Id: I06d41933057d60d15d307ee800cca052a44754c6
2012-11-07 08:39:57 +09:00
Dave Borowitz
b42b4746af project: Require git >= 1.7.2 for setting config on command line
This option causes the git call to fail, which probably indicates a
programming error; callers should check the git version and change the
call appropriately if -c is not available. Failing loudly is preferable
to failing silently in the general case.

For an example of correctly checking at the call site, see I8fd313dd.
If callers prefer to fail silently, they may set GIT_CONFIG_PARAMETERS
in the environment rather than using the config kwarg to pass
configuration.

Change-Id: I0de18153d44d3225cd3031e6ead54461430ed334
2012-10-31 12:27:27 -07:00
David Pursehouse
1d947b3034 Even more coding style cleanup
Fixing some more pylint warnings:

W1401: Anomalous backslash in string
W0623: Redefining name 'name' from outer scope
W0702: No exception type(s) specified
E0102: name: function already defined line n

Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
2012-10-30 10:28:20 +09:00
Shawn O. Pearce
cd81dd6403 Revert "Represent git-submodule as nested projects"
This reverts commit 69998b0c6f.
Broke Android's non-gitmodule use case.

Conflicts:
	project.py
	subcmds/sync.py

Change-Id: I68ceeb63d8ee3b939f85a64736bdc81dfa352aed
2012-10-26 12:24:57 -07:00
Shawn Pearce
dc96476af3 Merge "project: Support config args in git command callables" 2012-10-25 17:36:04 -07:00
Che-Liang Chiou
ab8f911a67 Fix pylint warnings introduced by the submodule patch
"69998b0 Represent git-submodule as nested projects" has introduced a
few pylint warnings.

W0612:1439,8:Project._GetSubmodules.get_submodules: Unused variable 'sub_gitdir'
W0613:1424,36:Project._GetSubmodules.get_submodules: Unused argument 'path'
W0612:1450,25:Project._GetSubmodules.parse_gitmodules: Unused variable 'e'
W0622:516,8:Sync.Execute: Redefining built-in 'all'

Change-Id: I84378e2832ed1b5ab023e394d53b22dcea799ba4
2012-10-25 13:55:49 -07:00
Conley Owens
608aff7f62 Merge "Use modern Python exception syntax" 2012-10-25 10:03:37 -07:00
Dave Borowitz
091f893625 project: Support config args in git command callables
Change-Id: I9d4d0d2b1aeebe41a6b24a339a154d258af665eb
2012-10-24 14:52:08 -07:00
Sarah Owens
a5be53f9c8 Use modern Python exception syntax
"except Exception as e" instead of "except Exception, e"

This is part of a transition to supporting Python 3.  Python >= 2.6
support "as" syntax.

Note: this removes Python 2.5 support.

Change-Id: I309599f3981bba2b46111c43102bee38ff132803
2012-10-23 21:35:59 -07:00
Che-Liang Chiou
69998b0c6f Represent git-submodule as nested projects
We need a representation of git-submodule in repo; otherwise repo will
not sync submodules, and leave workspace in a broken state.  Of course
this will not be a problem if all projects are owned by the owner of the
manifest file, who may simply choose not to use git-submodule in all
projects.  However, this is not possible in practice because manifest
file owner is unlikely to own all upstream projects.

As git submodules are simply git repositories, it is natural to treat
them as plain repo projects that live inside a repo project.  That is,
we could use recursively declared projects to denote the is-submodule
relation of git repositories.

The behavior of repo remains the same to projects that do not have a
sub-project within.  As for parent projects, repo fetches them and their
sub-projects as normal projects, and then checks out subprojects at the
commit specified in parent's commit object.  The sub-project is fetched
at a path relative to parent project's working directory; so the path
specified in manifest file should match that of .gitmodules file.

If a submodule is not registered in repo manifest, repo will derive its
properties from itself and its parent project, which might not always be
correct.  In such cases, the subproject is called a derived subproject.

To a user, a sub-project is merely a git-submodule; so all tips of
working with a git-submodule apply here, too.  For example, you should
not run `repo sync` in a parent repository if its submodule is dirty.

Change-Id: I541e9e2ac1a70304272dbe09724572aa1004eb5c
2012-10-23 16:08:58 -07:00
David Pursehouse
5c6eeac8f0 More coding style cleanup
Fixing more issues found with pylint.  Some that were supposed to
have been fixed in the previous sweep (Ie0db839e) but were missed:

C0321: More than one statement on a single line
W0622: Redefining built-in 'name'

And some more:

W0631: Using possibly undefined loop variable 'name'
W0223: Method 'name' is abstract in class 'name' but is not overridden
W0231: __init__ method from base class 'name' is not called

Change-Id: Ie119183708609d6279e973057a385fde864230c3
2012-10-22 12:30:14 +09:00
Mickaël Salaün
2f6ab7f5b8 Rename "dir" variables
The variable name "dir" conflicts with the name of a Python built-in
function: http://docs.python.org/library/functions.html#dir

Change-Id: I850f3ec8df7563dc85e21f2876fe5e6550ca2d8f
2012-10-10 08:30:15 +02:00
David Pursehouse
8a68ff9605 Coding style cleanup
Fix the following issues reported by pylint:

C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
W0612: Unused variable 'name'
W0613: Unused argument 'name'
W0102: Dangerous default value 'value' as argument
W0105: String statement has no effect

Also fixed a few cases of inconsistent indentation.

Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
2012-10-09 12:45:30 +02:00
Brian Harring
14a6674e32 manifest: record the original revision when in -r mode.
Currently when doing a sync against a revision locked manifest,
sync has no option but to fall back to sync'ing the entire refs space;
it doesn't know which ref to ask for that contains the sha1 it wants.

This sucks if we're in -c mode; thus when we generate a revision
locked manifest, record the originating branch- and try syncing that
branch first.  If the sha1 is found within that branch, this saves
us having to pull down the rest of the repo- a potentially heavy
saving.

If that branch doesn't have the desired sha1, we fallback to sync'ing
everything.

Change-Id: I99a5e44fa1d792dfcada76956a2363187df94cf1
2012-09-28 22:31:27 -07:00
Matt Gumbel
2dc810c2e4 Fix errors when clone.bundle missing on server
Catch curl failures to download clone.bundle; don't let git try to parse
the 404 page as a bundle file (was causing much user confusion).

This should eliminate false error messages from init and sync such as:
  error: '.repo/manifests.git/clone.bundle' does not look like a v2 bundle file
  fatal: Could not read bundle '.repo/manifests.git/clone.bundle'.
  error: RPC failed; result=22, HTTP code = 400

Signed-off-by: Matt Gumbel <matthew.k.gumbel@intel.com>
Change-Id: I7994f7c0baecfb45bb5a5850c48bd2a0ffabe773
2012-09-06 10:54:46 -07:00
Conley Owens
bb1b5f5f86 Allow projects to be specified as notdefault
Instead of every group being in the group "default", every project
is now in the group "all".   A group that should not be downloaded
by default may be added to the group "notdefault".

This allows all group names to be positive (instead of removing groups
directly in the manifest with -default) and offers a clear way of
selecting every project (--groups all).

Change-Id: I99cd70309adb1f8460db3bbc6eff46bdcd22256f
2012-09-05 11:46:48 -07:00
David Pursehouse
e15c65abc2 Remove unused imports
There are several imports that are not used.  Remove them.

Change-Id: I2ac3be66827bd68d3faedcef7d6bbf30ea01d3f2
2012-08-23 12:15:26 +02:00
Mickaël Salaün
b9477bc2dd project.py: Replace the relpath function with os.path.relpath
Change-Id: Ib313340344968211cecfc0a718f6072e41da1a91
2012-08-06 23:51:43 +02:00
Shawn O. Pearce
5e7127d00b Use curl command line tool for clone.bundle
urllib2 is not thread safe and may be causing sync to lock up or
not work correctly on various platforms. Instead use the command
line curl program.

Change-Id: I36eaf18bb4df089d26ea99d533cb015e7c616eb0
2012-08-02 15:18:10 -07:00
Shawn O. Pearce
e0904f721b Fix unsupported operand type(s) for +: 'int' and 'str'
Change-Id: I88455107d63daaa60c3b33c010aa8c730a590c70
2012-08-01 20:44:23 -07:00
Shawn O. Pearce
9830553748 Fix percent done on resumed /clone.bundle
The Content-Length when resuming is the number of bytes that
remain in the file. To compute the total size as expected by
the progress meter, we must add the bytes already stored.

While we are in this method fix uses of % operator to ensure
a tuple is always supplied.

Change-Id: Ic899231b5bc0ab43b3ddb1d29845f6390e820115
2012-08-01 17:41:26 -07:00
Brian Harring
435370c6f0 upload: add --draft option.
Change-Id: I6967ff2f8163cd4116027b3f15ddb36875942af4
2012-07-28 15:44:05 -07:00
Torne (Richard Coles)
e8f75fa368 Don't delete the branch config when switching branches.
The fix for issue #46 in 5d016502eb appears to break syncing in some
situations: the branch is deleted after the point where it's been
configured, which deletes part of its configuration and causes the
config to change each time you call `repo init`, alternating between a
configuration that works and one that doesn't.

Instead of deleting the branch with git branch -D, use git update-ref -d
which just deletes the ref (to avoid the rebase) without touching the
configuration for the branch that was set up during the first repo init.

This appears to ensure the config is left in a valid state all the time
no matter what combination of repo init commands you run, without
reintroducing the rebasing issue.

Change-Id: Iaadaa6e56a46840bbc593fa5b35cb5b34cd3ce69
2012-07-20 15:33:17 +01:00
Florian Vallee
7cf1b36bcd Detach branch even when already on the latest revision using sync -d
This patch fixes repo behaviour when running sync -d with unmodified
topic branches.

Prior to this patch sync -d would see the latest revision is already
checked out, thus staying on the branch. Since "-d" means detach we
should follow git's behaviour and actually detach from the branch in
that case.

Basic test case - after a fresh repo init + sync -
        * repo start --all testdetach
        * repo sync -d
        * repo status
-> status shows active topic branch "testdetach",
   should show :
nothing to commit (working directory clean)

Change-Id: Ic1351e6b5721b76557a51ab09f9dd42c38a4b415
2012-06-13 10:36:17 -07:00
Florian Vallee
5d016502eb Fix switching manifest branches using repo init -b
See repo issue #46 :
	https://code.google.com/p/git-repo/issues/detail?id=46

When using repo init -b on an already existing repository,
the next sync will try to rebase changes coming from the old manifest
branch onto the new, leading in the best case scenario to conflicts
and in the worst case scenario to an incorrect "mixed up" manifest.

This patch fixes this by deleting the "default" branch in the local
manifest repository when the -d init switch is used, thus forcing
repo to perform a fresh checkout of the new manifest branch

Change-Id: I379e4875ec5357d8614d1197b6afbe58f9606751
2012-06-13 10:00:57 -07:00
Colin Cross
eca119e5d6 Allow projects with groups=None
Mirror manifest and repo projects are outside the manifest and
have no groups.  Allow project groups to be None for these
projects.

Change-Id: I3e1c4add894fe1c43aa4e77a1fc1558aa10dd191
2012-05-24 15:40:05 -07:00
Shawn O. Pearce
6ba6ba0ef3 Fix initial sync broken by sync-c option
Change-Id: I308753da8944e6ce5c46e3bfee1bcd41d5b7e292
2012-05-24 09:46:50 -07:00
Pierre Tardy
3d125940f6 repo download: add --ff-only option
Allows to ff-only a gerrit patch
This patch is necessary to automatically ensure that the patch will
be correctly submitted on ff-only gerrit projects

You can now use:
repo download (--ff-only|-f) project changeid/patchnumber

This is useful to automate verification of fast forward status of a patch
in the context of build automation, and commit gating (e.g. buildbot)

Change-Id: I403a667557a105411a633e62c8eec23d93724b43
Signed-off-by: Erwan Mahe <erwan.mahe@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
2012-05-24 09:04:20 -07:00
Erwan Mahe
a94f162b9f repo download: add --revert option
BZ: 4779
Allows to revert a gerrit patch
This patch is necessary for the on-demand creation of
engineering builds using buildbot

You can now use:
repo download [--revert|-r project changeid/patchnumber

This is useful to automate reverting of a patch
in the context of build automation, and regression bisection

Change-Id: I3985e80e4b2a230f83526191ea1379765a54bdcf
Signed-off-by: Erwan Mahe <erwan.mahe@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
2012-05-24 09:03:10 -07:00
Pierre Tardy
e5a2122e64 repo download: add --cherry-pick option
default option uses git checkout, and thus overwrite the previous
checkouts.  this is a problem for automated builds of several
changesets in the same project for daily builds of pending submission

You can now use:
repo download [--cherry-pick|-c] project changeid/patchnumber

This will parse the manifest, cd to the corresponding project
download the changes to FETCH_HEAD and cherry-pick the result.

This is useful to automate cherry-picking of a patch
in the context of build automation, and commit gating (e.g. buildbot)

Change-Id: Ib638afd87677f1be197afb7b0f73c70fb98909fe
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
2012-05-24 09:02:38 -07:00
Anatol Pomazau
79770d269e Add sync-c option to manifest
There are use-cases when fetching all branch is impractical and
we really need to fetch only one branch/tag.
e.g. there is a large project with binaries and every update of a
binary file is put to a separate branch.
The whole project history might be too large to allow users fetch it.

Add 'sync-c' option to 'project' and 'default' tags to make it possible
to configure 'sync-c' behavior at per-project and per-manifest level.

Note that currently there is no possibility to revert boolean flag from
command line. If 'sync-c' is set in manifest then you cannot make
full fetch by providing a repo tool argument.

Change-Id: Ie36fe5737304930493740370239403986590f593
2012-04-23 14:10:52 -07:00
Conley Owens
971de8ea7b Refine groups functionality
Every project is in group "default".  "-default" does not remove
it from this project.  All group names specified in the manifest
are positive names as opposed to a mix of negative and positive.

Specified groups are resolved in order.  If init is supplied with
--groups="group1,-group2", the following describes the project
selection when syncing:

  * all projects in "group1" will be added, and
  * all projects in "group2" will be removed.

Change-Id: I1df3dcdb64bbd4cd80d675f9b2d3becbf721f661
2012-04-23 12:39:05 -07:00
James W. Mills
24c1308840 Add project annotation handling to repo
Allow the optional addition of "annotation" nodes nested under
projects.  Each annotation node must have "name" and "value"
attributes.  These name/value pairs will be exported into the
environment during any forall command, prefixed with "REPO__"

In addition, an optional "keep" attribute with case insensitive "true"
or "false" values can be included to determine whether the annotation
will be exported with 'repo manifest'

Change-Id: Icd7540afaae02c958f769ce3d25661aa721a9de8
Signed-off-by: James W. Mills <jameswmills@gmail.com>
2012-04-23 12:35:08 -07:00
Anatol Pomazau
b962a1f5e0 Check if SHA1 presents in repository
Previously repo had incorrect code that did not really check
if sha1 presents in a project. It worked for tags though.

Check if a revision (either tag or sha1) is present by using
'git rev_parse' functionality.

Change-Id: I1787f3348573948573948753987394839487572b
2012-04-23 11:09:17 -07:00
Colin Cross
5acde75e5d Add manifest groups
Allows specifying a list of groups with a -g argument to repo init.
The groups act on a group= attribute specified on projects in the
manifest.
All projects are implicitly labelled with "default" unless they are
explicitly labelled "-default".
Prefixing a group with "-" removes matching projects from the list
of projects to sync.
If any non-inverted manifest groups are specified, the default label
is ignored.

Change-Id: I3a0dd7a93a8a1756205de1d03eee8c00906af0e5
Reviewed-on: https://gerrit-review.googlesource.com/34570
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
2012-04-13 09:46:00 -07:00
pelya
d67872d2f4 Option for 'repo diff' to generate output suitable for 'patch' cmd
The -u option causes 'repo diff' to generate diff output
with file paths relative to the repository root,
so the output can be applied to the Unix 'patch' command.
The name '-u' was selected for convenience, because
both 'diff' and 'git diff' accept the option with the same name
to generate an 'unified diff' output suitable for 'patch' command.

Change-Id: I79c8356db4ed20ecaccc258b3ba139db76666fe0
Reviewed-on: https://gerrit-review.googlesource.com/34380
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
2012-04-13 09:20:10 -07:00
Shawn O. Pearce
c3d2f2b76f Ignore /clone.bundle on HTTP 401, 403 and 404
401: Unauthorized, authentication may be required. This is usually
     handled internally by the HTTP client in Python. If it reaches
     our code in repo, the Python HTTP client didn't find a password
     in ~/.netrc that it could use.

403: Authentication was supplied, but is incorrect. It might be
     that the CDN doesn't want to offer this clone.bundle file
     to the client, but the Git fetch operation would still be
     successful. This might arise if branch level read controls
     were used in Gerrit Code Review and the /clone.bundle file
     contained branches not visible to the client.

404: The server has no /clone.bundle file available.

In all of these cases, sliently ignore the /clone.bundle file HTTP
error and let the Git operation take over.

Change-Id: I1787f3cac86b017351952bbb793fe5874d83c72b
2012-03-22 14:18:40 -07:00
Anatol Pomazau
cd7c5deca0 Do not change branch.foo.merge in case of manifest sync
In case of manifest/smart sync repo changes ".merge" config
option from branch to SHA. Doing 'repo upload' fails as
repo tries to upload to a remote branch that looks like SHA
(e.g. refs/for/23423423423423423423423)

Do not update the .merge in case if revision is SHA.

Change-Id: I9139708fa17f21eec5a7e23c3255333626bf529e
2012-03-20 14:11:56 -07:00
Shawn O. Pearce
e02ac0af2e sync: --no-clone-bundle disables the clone bundle support
Change-Id: Ia9ed7da8451b273c1be620c3dd0dcad777b29096
2012-03-14 15:38:28 -07:00
Shawn O. Pearce
898e12a2d9 Permit - in URL schemes for special URLs
Clients might be using their own special git-remote-* helper that
has a hypen in its name. Permit - in the scheme part of the URL
when trying to decide if it is an SSH URL and assume it is *not*
SSH if the URL matches "foo-bar://" style.

Change-Id: I7ba2d810a614f6e605a441d5972902c4a14e73fd
2012-03-14 15:28:22 -07:00
Ali Utku Selen
76abcc1d1e repo status to print project name on clean gits
repo status just prints "# on branch oprofile" if you have branched
in clean status. This doesn't really tell which branch is meant.

Instead we can use the same syntax with modified gits which will
give us detailed information.

Change-Id: I55fe5154d278e10a814281dd2ba501ec6e956730
2012-03-12 12:25:40 -07:00
Mike Pontillo
d315382572 Add 'rebase="false"' attribute to the <project/> XML.
This new attribute can prevent 'repo sync' from automatically rebasing.

I hit a situation in where one of the git repositories I was tracking
was actually an external repository that I wanted to pull commits
into and merge myself. (NOT rebase, since that would lose the merge
history.) In this case, I'm not using 'repo upload', I'm manually
managing the merges to and from this repository.

Everything was going great until I typed 'repo sync' and it rebased
my manually-merged tree. Hence the option to skip it.

Change-Id: I965e0dd1acb87f4a56752ebedc7e2de1c502dbf8
2012-03-12 12:24:22 -07:00
Conley Owens
43bda84362 Avoid missing content-length header in project.py
Occassionally, the content-length may be missing when using urlib
in python 2.6 and 2.7.  This change assumes the value of the header is
0 if it doesn't exist

Change-Id: Iaf1c8a796bc667823d4d7c30f9b617644b271d00
2012-03-12 12:13:15 -07:00
Shawn O. Pearce
c9571423f8 upload: Support uploading to Gerrit over https://
If SSH is not available, Gerrit returns NOT_AVAILABLE to the /ssh_info
query made by repo upload. In this case fallback to the /p/$PROJECT URL
that Gerrit also exports and use that for uploads.

Change-Id: I1e3e39ab709ecc0a692614a41a42446426f39c08
2012-01-11 16:18:40 -08:00
Anatol Pomazau
2a32f6afa6 Fix typo
Change-Id: Idd68ad0a34fcf4bd4e18b0248f50187a539d610a
2011-11-29 12:09:35 -08:00
Anatol Pomazau
53d6f4d17e Add a sync flag that fetches only current branch
There is also shortcuts in case if the "current branch" is
a persistent revision such as tag or sha1. We check if the
persistent revision is present locally and if it does - do
no fetch anything from the server.

This greately reduces sync time and size of the on-disk repo

Change-Id: I23c6d95185474ed6e1a03c836a47f489953b99be
2011-11-03 13:08:27 -07:00
Shawn O. Pearce
df5ee52050 Fix Python 2.4 support
Change-Id: I89521ae52fa564f0d849cc51e71fee65b3c47bab
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 14:06:11 -07:00
Shawn O. Pearce
fab96c68e3 Work around Python 2.7 urllib2 bug
If the remote is using authenticated HTTP, but does not have
$GIT_URL/clone.bundle files in each repository, an initial sync
would fail around 8 projects in due to the library not resetting
the number of failures after getting a 404.

Work around this by updating the retry counter ourselves.

The urllib2 library is also not thread-safe. Make it somewhat
safer by wrapping the critical section with a lock.

Change-Id: I886e2750ef4793cbe2150c3b5396eb9f10974f7f
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 12:18:07 -07:00
Shawn O. Pearce
bf1fbb20ab Fix AttributeError: 'HTTPError' object has no attribute 'reason'
Not every version of urllib2 supplies a reason object on the
HTTPError exception that it throws from urlopen().  Work around
this by using str(e) instead and hope the string formatting includes
sufficient information.

Change-Id: I0f4586dba0aa7152691b2371627c951f91fdfc8d
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 09:31:58 -07:00
Shawn O. Pearce
29472463ba Work around Python 2.7 failure to initialize base class
urllib2 returns a malformed HTTPError object in certain situations.
For example, urllib2 has a couple of places where it creates an
HTTPError object with no fp:

  if self.retried > 5:
    # retry sending the username:password 5 times before failing.
    raise HTTPError(req.get_full_url(), 401, "basic auth failed",
                    headers, None)

When it does that, HTTPError's ctor doesn't call through to
addinfourl's ctor:

  # The addinfourl classes depend on fp being a valid file
  # object.  In some cases, the HTTPError may not have a valid
  # file object.  If this happens, the simplest workaround is to
  # not initialize the base classes.
  if fp is not None:
    self.__super_init(fp, hdrs, url, code)

Which means the 'headers' slot in addinfourl is not initialized and
info() fails.  It is completely insane that urllib2 decides not to
initialize its own base class sometimes.

Change-Id: I32a0d738f71bdd7d38d86078b71d9001e26f1ec3
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-11 09:24:07 -07:00
Shawn O. Pearce
c325dc35f6 sync: Fetch after applying bundle and retry after errors
After a $GIT_URL/clone.bundle has been applied to the new local
repository, perform an incremental fetch using `git fetch` to ensure
the local repository is up-to-date. This allows the hosting server
to offer stale /clone.bundle files to bootstrap a new client.

If a single git fetch fails, it may succeed again after a short
delay.  Transient failures are typical in environments where the
remote Git server happens to have limits on how many requests it
can serve at once (the anonymous git daemon, or an HTTP server).
Wait a randomized delay between 30 and 45 seconds and retry the
failed project once.  This delay gives the site time to recover
from a transient traffic spike, and the randomization makes it less
likely that a spike occurs again from all of the same clients.

Change-Id: I97fb0fcb33630fb78ac1a21d1a4a3e2268ab60c0
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-10-03 08:30:24 -07:00
Shawn O. Pearce
f322b9abb4 sync: Support downloading bundle to initialize repository
An HTTP (or HTTPS) based remote server may now offer a 'clone.bundle'
file in each repository's Git directory. Over an http:// or https://
remote repo will first ask for '$URL/clone.bundle', and if present
download this to bootstrap the local client, rather than relying
on the native Git transport to initialize the new repository.

Bundles may be hosted elsewhere. The client automatically follows a
HTTP 302 redirect to acquire the bundle file. This allows servers
to direct clients to cached copies residing on content delivery
networks, where the bundle may be closer to the end-user.

Bundle downloads are resumeable from where they last left off,
allowing clients to initialize large repositories even when the
connection gets interrupted.

If a bundle does not exist for a repository (a HTTP 404 response
code is returned for '$URL/clone.bundle'), the native Git transport
is used instead. If the client is performing a shallow sync, the
bundle transport is not used, as there is no way to embed shallow
data into the bundle.

Change-Id: I05dad17792fd6fd20635a0f71589566e557cc743
Signed-off-by: Shawn O. Pearce <sop@google.com>
2011-09-28 10:07:36 -07:00
Victor Boivie
65e0f35fda Add commit-msg hook also for manifest project
The manifest project has - by design - not a review URL associated
with it. It is actually not even a 'project' in repo's sense.

This will prevent the commit-msg hook from being added, which is
not necessarily wanted as the project is managed in gerrit.

This commit will enable the commit-msg hook, which in turn will
add the Change-Id-line to every new commit in it. This simplifies
replacing patch sets (by git push ... refs/for/...).

Change-Id: I42d0f6fd79e6282d9d47074a3819e68d968999a7
Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
2011-07-20 07:34:23 -07:00
Doug Anderson
30d452905f Add a --depth option to repo init.
Change-Id: Id30fb4a85f4f8a1847420b0b51a86060041eb5bf
2011-06-09 16:48:23 -07:00
Mandeep Singh Baines
d6c93a28ca Add branch support to repo upload
This commit adds a --br=<branch> option to repo upload.

repo currently examines every non-published branch. This is problematic
for my workflow. I have many branches in my kernel tree. Many of these
branches are based off of upstream remotes (I have many remotes) and
will never be uploaded (they'll get sent upstream as a patch).

Having repo scan these branches adds to my upload processing time
and clutters the branch selection buffer. I've also seen repo get
confused when one of my branches is 1000s of commits different from
m/master.

Change-Id: I68fa18951ea59ba373277b57ffcaf8cddd7e7a40
2011-05-26 10:49:39 -07:00
Doug Anderson
3ba5f95b46 Fixed repo checkout error message when git reports errors.
In the current version of repo checkout, we often get the error:
  error: no project has branch xyzzy

...even when the actual error was something else.  This fixes it
to only report the 'no project has branch' when that is actually true.

This fix is very similar to one made for 'repo abandon':
  https://review.source.android.com/#change,22207

The repo checkout error is filed as: <http://crosbug.com/6514>

TEST=manual

A sample creating a case where 'git checkout' will fail:

  $ repo start branch1 .
  $ repo start branch2 .
  $ touch bogusfile
  $ git add bogusfile
  $ git commit -m "create bogus file"
  [branch2 f8b6b08] create bogus file
   0 files changed, 0 insertions(+), 0 deletions(-)
   create mode 100644 bogusfile
  $ echo "More" >> bogusfile
  $ repo checkout branch1 .
  error: chromite/: cannot checkout branch1

A sample case showing that we still fail if no project has a branch:

  $ repo checkout xyzzy .
  error: no project has branch xyzzy

Change-Id: I48a8e258fa7a9c1f2800dafc683787204bbfcc63
2011-04-07 16:55:35 -04:00
Doug Anderson
dafb1d68d3 Fixed repo abandon to give better messages.
The main fix is to give an error message if nothing was actually
abandoned.  See <http://crosbug.com/6041>.

The secondary fix is to list projects where the abandon happened.
This could be done in a separate CL or dropped altogether if requested.

TEST=manual

$ repo abandon dougabc; echo $?
Abandon dougabc: 100% (127/127), done.
Abandoned in 2 project(s):
  chromite
  src/platform/init
0

$ repo abandon dougabc; echo $?
Abandon dougabc: 100% (127/127), done.
error: no project has branch dougabc
1

$ repo abandon dougabc; echo $?
Abandon dougabc: 100% (127/127), done.
error: chromite/: cannot abandon dougabc
1

Change-Id: I79520cc3279291acadc1a24ca34a761e9de04ed4
2011-04-07 16:49:23 -04:00
Terence Haddock
4655e81a75 Add option to check status of projects in parallel.
Change-Id: I6ac653f88573def8bb3d96031d3570ff966251ad
2011-04-07 16:36:42 -04:00
Victor Boivie
0960b5b53d Creating rr-cache
If git-rerere is enabled, it uses the rr-cache directory that
repo currently creates a symlink from, but doesn't create the
destination directory (inside the project's directory). Git
will then complain during merges and rebases.

This commit creates the rr-cache directory inside the project.

Change-Id: If8b57a04f022fc6ed6a7007d05aa2e876e6611ee
2011-03-17 09:19:51 -07:00
Doug Anderson
37282b4b9c Support repo-level pre-upload hook and prep for future hooks.
All repo-level hooks are expected to live in a single project at the
top level of that project.  The name of the hooks project is provided
in the manifest.xml.  The manifest also lists which hooks are enabled
to make it obvious if a file somehow failed to sync down (or got
deleted).

Before running any hook, we will prompt the user to make sure that it
is OK.  A user can deny running the hook, allow once, or allow
"forever" (until hooks change).  This tries to keep with the git
spirit of not automatically running anything on the user's computer
that got synced down.  Note that individual repo commands can add
always options to avoid these prompts as they see fit (see below for
the 'upload' options).

When hooks are run, they are loaded into the current interpreter (the
one running repo) and their main() function is run.  This mechanism is
used (instead of using subprocess) to make it easier to expand to a
richer hook interface in the future.  During loading, the
interpreter's sys.path is updated to contain the directory containing
the hooks so that hooks can be split into multiple files.

The upload command has two options that control hook behavior:
  - no-verify=False, verify=False (DEFAULT):
    If stdout is a tty, can prompt about running upload hooks if needed.
    If user denies running hooks, the upload is cancelled.  If stdout is
    not a tty and we would need to prompt about upload hooks, upload is
    cancelled.
  - no-verify=False, verify=True:
    Always run upload hooks with no prompt.
  - no-verify=True, verify=False:
    Never run upload hooks, but upload anyway (AKA bypass hooks).
  - no-verify=True, verify=True:
    Invalid

Sample bit of manifest.xml code for enabling hooks (assumes you have a
project named 'hooks' where hooks are stored):
  <repo-hooks in-project="hooks" enabled-list="pre-upload" />

Sample main() function in pre-upload.py in hooks directory:
  def main(project_list, **kwargs):
    print ('These projects will be uploaded: %s' %
           ', '.join(project_list))
    print ('I am being a good boy and ignoring anything in kwargs\n'
           'that I don\'t understand.')
    print 'I fail 50% of the time.  How flaky.'
    if random.random() <= .5:
      raise Exception('Pre-upload hook failed.  Have a nice day.')

Change-Id: I5cefa2cd5865c72589263cf8e2f152a43c122f70
2011-03-11 11:53:23 -08:00
Skyler Kaufman
835cd6888f Post-nonexistent-revision crash sidestepped
Fix for the bug that leaves a fractional .git directory after attempting to
perform an initial sync to a nonexistent revision. Moved the initialization of
the working directory to after the revision ID has already been checked. Now,
no project/.git directory gets created at all if the revision ID is bad.

Change-Id: I0c9b2a59573410f1d11de7661591bf02e4ce326b
2011-03-08 13:48:24 -08:00
Doug Anderson
8ced8641c8 Renamed 'repo_hooks' function to '_ProjectHooks'.
This renaming was done for two reasons:
1. The hooks are actually project-level hooks, not repo-level
   hooks.  Since we are talking about adding repo-level hooks,
   It keeps things less confusing if we name the existing hooks
   to be "ProjectHooks"
2. The function is a private function in project.py and so
   should have capitalization to match.

I also added a docstring describing this function.

Change-Id: I1d30f5de08e8f9f99f78146e68c76f906782d97e
2011-02-01 09:57:29 -08:00
Doug Anderson
2536f80625 Fixed bug identifying 'commit-msg' files.
There was a minor typo that would cause repo to (I believe)
mistakenly identify any file that contained a substring of the
word 'commit-msg' as a commit message hook.  For example, the file
'mit' or the file 'msg' would be treated as a commit message hook.
I believe that it was intended that repo only recognize files
named exactly 'commit-msg'.

Change-Id: I93edbddf3da3cf0935641e6efb19b0a8ee6e2308
2011-02-01 09:53:56 -08:00
Shawn O. Pearce
0ce6ca9c7b Fix mirror clients with no worktree
Commit "Make path references OS independent" (df14a70c45)
broke mirror clients by trying to invoke replace() on None
when there is no worktree.

Change-Id: Ie0a187058358f7dcdf83119e45cc65409c980f11
2011-01-10 13:26:34 -08:00
Anthony Newnam
df14a70c45 Make path references OS independent
Change-Id: I5573995adfd52fd54bddc62d1d1ea78fb1328130
(cherry picked from commit b0f9a02394)

Conflicts:

	command.py
2011-01-09 17:39:19 -08:00
Ficus Kirkpatrick
a0de6e8eab upload: Remove --replace option
It hasn't been necessary for a long time, and its
functionality can be accomplished with 'git push'.

Change-Id: Ic00d3adbe4cee7be3955117489c69d6e90106559
2010-10-29 12:12:56 -07:00
Shawn O. Pearce
16614f86b3 sync --quiet: be more quiet
Change-Id: I5e8363c7b32e4546d1236cfc5a32e01c3e5ea8e6
Signed-off-by: Shawn O. Pearce <sop@google.com>
2010-10-29 12:08:57 -07:00
Shawn O. Pearce
88443387b1 sync: Enable use of git clone --reference
Use git clone to initialize a new repository, and when possible
allow callers to use --reference to reuse an existing checkout as
the initial object storage area for the new checkout.

Change-Id: Ie27f760247f311ce484c6d3e85a90d94da2febfc
Signed-off-by: Shawn O. Pearce <sop@google.com>
2010-10-29 12:08:50 -07:00
Shawn O. Pearce
a5ece0e050 upload -t: Automatically include local branch name
If the -t flag is given to upload, the local branch name is
automatically sent to Gerrit Code Review as the topic branch name
for the change(s).  This requires the server to be Gerrit Code
Review v2.1.3-53-gd50c94e or later, which isn't widely deployed
right now, so the default is opt-out.

Change-Id: I034fcacb405b7cb909147152db427fe69dd7bcbf
Signed-off-by: Shawn O. Pearce <sop@google.com>
2010-07-15 16:52:42 -07:00
Anthony Newnam
cc50bac8c7 Warn users before uploading if there are local changes
Change-Id: I231d7b6a3211e9f5ec71a542a0109b0c195d5e40
Signed-off-by: Shawn O. Pearce <sop@google.com>
2010-07-15 16:43:58 -07:00
Julius Gustavsson
0cb1b3f687 sync: Try fetching a tag as a last resort before giving up
If a tagged commit is not reachable by the fetch refspec configured
for the git (usually refs/heads/*) it will not be downloaded by
'git fetch'.  The tag can however be downloaded with 'git fetch
--tags' or 'git fetch tag <tag>'.

This patch fixes the situation when a tag is not found after a
'git fetch'. Repo will issue 'git fetch tag <tag>' before giving
up completely.

Change-Id: I87796a5e1d51fcf398f346a274b7a069df37599a
Signed-off-by: Shawn O. Pearce <sop@google.com>
2010-07-15 16:38:08 -07:00
Matthew Buckett
2daf66740b Allow files to be copied into new folders
Change-Id: I7f169e32be5a4328bb87ce7c2ff4b6529e925126
2010-05-27 18:05:26 -07:00
Shawn O. Pearce
9452e4ec09 Automatically install Gerrit Code Review's commit-msg hook
Most users of repo are also using Gerrit Code Review, and will want
the commit-msg hook to be automatically installed into their local
projects so that Change-Ids are assigned when commits are created,
not when they are first uploaded.

(cherry picked from commit a949fa5d20
 but squashed with latest hook script from version 2.1.2)

Change-Id: Ie68b2d60ac85d8c2285d2e1e6a4536eb76695547
Signed-off-by: Shawn O. Pearce <sop@google.com>
2010-03-06 19:21:00 -08:00
Daniel Sandler
4c50deea28 Fail sync when encountering "N commits behind."
This is almost always something the user needs to address
before continuing work, so promoting it to a failure (rather
than simply an informational message) seems the right way to
go. As a side-effect, repo will now exit with a non-zero
status code in this situation, so pipelines of the form
`repo sync && make` will fail if there are branches that
are stalled due to uploaded but unmerged patches.
2010-03-04 11:56:38 -05:00
Nico Sallembien
d63060fc95 Check that we are not overwriting a local repository when syncing.
If a local git repository exists within the same folder as a new project that
is added, when the user syncs the repo, the sync will overwrite the local
files under the project's .git repository with its own symlinks. Make sure
that we do not overwrite 'normal' files in repo and throw an error when
that happens.
2010-01-20 10:27:50 -08:00
Shawn O. Pearce
aa4982e4c9 sync: Fix split call on malformed email addresses
If an email address in a commit object contains a space, like a few
malformed ones on the Linux kernel, we still want to split only on
the first space.

Unfortunately my brain was too damaged by Perl and originally wrote
the split asking for 2 results; in Python split's argument is how
many splits to perform.  Here we want only 1 split, to break apart
the commit identity from the email address on the same line.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-12-30 18:38:27 -08:00
Shawn O. Pearce
da88ff4411 Silence 'Current branch %s is up to date' during sync
We accidentally introduced this message during 1.6.8 by always
invoking `git rebase` when there were no new commits from the
upstream, but the user had local commits.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-06-03 11:09:31 -07:00
Shawn O. Pearce
fe200eeb52 Fix unnecessary self in project.py
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-06-01 15:28:21 -07:00
Shawn O. Pearce
3c8dea1f8d Change project.revision to revisionExpr and revisionId
The revisionExpr field now holds an expression from the manifest,
such as "refs/heads/master", while revisionId holds the current
commit-ish SHA-1 of the revisionExpr.  Currently that is only
filled in if the manifest points directly to a SHA-1.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-05-29 18:45:20 -07:00
Shawn O. Pearce
8ad8a0e61d Change DWIMery hack for dealing with rewound remote branch
The trick of looking at the reflog for the remote tracking branch
and only going back one commit works some of the time, but not all of
the time.  Its sort of relying on the fact that the user didn't use
`repo sync -n` or `git fetch` to only update the tracking branches
and skip the working directory update.

Doing this right requires looking through the history of the SHA-1
source (what the upstream used to be) and finding a spot where the
DAG diveraged away suddenly, and consider that to be the rewind
point.  That's really difficult to do, as we don't have a clear
picture of what that old point was.

A close approximation is to list all of the commits that are in
HEAD, but not the new upstream, and rebase all of those where the
committer email address is this user's email address.  In most cases,
this will effectively rebase only the user's new original work.

If the user is the project maintainer and rewound the branch
themselves, and they don't want all of the commits they have created
to be rebased onto the new upstream, they should handle the rebase
on their own, after the sync is complete.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-05-29 18:45:17 -07:00
Shawn O. Pearce
d1f70d9929 Refactor how projects parse remotes so it can be replaced
We now feed Project a RemoteSpec, instead of the Remote directly
from the XmlManifest.  This way the RemoteSpec already has the
full project URL, rather than just the base, permitting other
types of manifests to produce the URL in their own style.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-05-29 09:31:28 -07:00
Shawn O. Pearce
242b52690d Remove support for the extra <remote> definitions in manifests
These aren't that widely used, and actually make it difficult for
users to fully mirror a forest of repositories, and then permit
someone else to clone off that forest, rather then the original
upstream servers.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-05-19 13:01:52 -07:00
Ficus Kirkpatrick
bc7ef67d9b Automatically guess Gerrit change number in "repo upload --replace"
This feature only works if you have one commit to replace right now
(the common case).
2009-05-05 15:01:18 -07:00
Shawn O. Pearce
9360966bd2 Perform copy file activity when creating a new work directory
Performance improvements in repo sync caused us to skip out of the
initial Sync_LocalHalf without ever running CopyFiles, so we didn't
create the top level Makefile in new clients whose manifest request
one with a <copyfile> element.

Now we run CopyFiles after the initial read-tree that populates
the project working directory.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-21 10:54:59 -07:00
Shawn O. Pearce
05f66b6836 Fix 'repo sync' rebase logic on a published branch
If the current branch is published, but all published commits are
merged into the manifest revision, but there is also at least one
unpublished commit on the current branch, we should rebase the
unpublished commit, rather than creating a merge commit.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-21 08:28:06 -07:00
Shawn O. Pearce
fb2316146f Automatically use SSH control master support during sync
By creating a background ssh "control master" process which lives
for the duration of our sync cycle we can easily cut the time for
a no-op sync of 132 projects from 60s to 18s.

Bug: REPO-11
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 16:50:47 -07:00
Shawn O. Pearce
3d2cdd0ea5 Highlight projects which still have sync failures during 'repo status'
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 15:26:10 -07:00
Shawn O. Pearce
552ac89929 Modify 'repo abandon' to be more like 'repo checkout' and 'repo start'
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 15:15:24 -07:00
Shawn O. Pearce
89e717d948 Improve checkout performance for the common unmodified case
Most projects will have their branch heads matching in all branches,
so switching between them should be just a matter of updating the
work tree's HEAD symref.  This can be done in pure Python, saving
quite a bit of time over forking 'git checkout'.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 15:04:41 -07:00
Shawn O. Pearce
0f0dfa3930 Add progress meter to 'repo start'
This is mostly useful if the number of projects to switch is many
(e.g. all of Android) and a large number of them are behind the
current manifest revision.  We wind up needing to run git just to
make the working tree match, and that often makes the command take
a couple of seconds longer than we'd like.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 14:53:39 -07:00
Shawn O. Pearce
76ca9f8145 Make usage of open safer by setting binary mode and closing fds
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 14:48:03 -07:00
Shawn O. Pearce
accc56d82b Speed up 'repo start' by removing some forks
Its quite common for most projects to be matching the current
manifest revision, as most developers only modify one or two projects
at any one time.  We can speed up `repo start foo` (that impacts
the entire client) by performing most of the branch creation and
switch operations in pure Python, and thus avoid 4 forks per project.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 14:45:51 -07:00
Shawn O. Pearce
9bb9617858 Remove unused methods from project.ReviewableBranch
These used to be used back when we had Gerrit 1.x support and used
HTTP based uploads to transmit changes for review.  Since we moved
entirely to Gerrit 2.x, these are no longer called.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 10:53:27 -07:00
Shawn O. Pearce
f690687671 Only fetch repo once-per-day under normal 'repo sync' usage
Its unlikely that a new version of repo will be delivered in any
given day, so we now check only once every 24 hours to see if repo
has been updated.  This reduces the sync cost, as we no longer need
to contact the repo distribution servers every time we do a sync.

repo selfupdate can still be used to force a check.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 10:49:00 -07:00
Shawn O. Pearce
336f7bd0ed Avoid git fork on the common case of repo not changing
Usually repo is upgraded only once a week, if that often.  Most of
the time we invoke HasChanges on the repo project (or even on the
manifest project) the current HEAD will resolve to the same SHA-1
as the remote tracking ref, and there are therefore no changes.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-18 10:39:28 -07:00
Shawn O. Pearce
0f3dd233ec Avoid unnecessary git symbolic-ref calls during repo sync
If the m/BRANCH ref is already pointing at the value set in the
manifest there is no reason to set it again.  Leave it alone,
thus saving a full fork+exec call.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-17 21:03:45 -07:00
Shawn O. Pearce
fbcde472ca Improve repo sync performance by avoid git forks
By resolving the current HEAD and the manifest revision using pure
Python, we can in the common case of "no changes" avoid a lot of
git operations and directly jump out of the local sync method.

This reduces the no-op `repo sync -l` time for Android's 114 projects
from more than 6s to under 0.8s.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-17 21:03:45 -07:00
Shawn O. Pearce
d237b69865 Implement git ref reading purely in Python
Its much faster to read the refs from 114 projects when the reader
is pure Python and just doing file IO than forking 114 git commands
and parsing their output.

The reader caches refs based upon file mtimes.  If any single ref
file has been modified since the last read, we re-read the entire
repository's ref namespace.  This simplifies the code as we don't
need to worry about shooting down symbolic-refs, but it may cause
more IO than is necessary if only one ref gets updated.

This change drops `repo branches` in Android from 1.658s to 0.206s.
Likewise, `repo sync` improves dramatically as well.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-17 21:03:41 -07:00
Shawn O. Pearce
5b23f24881 Implement 'git symbolic-ref HEAD' in Python
This is invoked once per project in `repo sync`.  Taking it out
saves about 1/114 of a second, so on a large set of projects like
Android it can save up to a full second of sync time.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-17 20:59:44 -07:00
Shawn O. Pearce
006734b798 Remove confusing message from repo sync output
Someone pointed out this message isn't always the truth; so we
shouldn't print it.  The code path is executed when there are
published commits, yet our output talks about unpublished ones.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-17 10:28:25 -07:00
Shawn O. Pearce
350cde4c4b Change repo sync to be more friendly when updating the tree
We now try to sync all projects that can be done safely first, before
we start rebasing user commits over the upstream.  This has the nice
effect of making the local tree as close to the upstream as possible
before the user has to start resolving merge conflicts, as that extra
information in other projects may aid in the conflict resolution.

Informational output is buffered and delayed until calculation for
all projects has been done, so that the user gets one concise list
of notice messages, rather than it interrupting the progress meter.

Fast-forward output is now prefixed with the project header, so the
user can see which project that update is taking place in, and make
some relation of the diffstat back to the project name.

Rebase output is now prefixed with the project header, so that if
the rebase fails, the user can see which project we were operating
on and can try to address the failure themselves.

Since rebase sits on a detached HEAD, we now look for an in-progress
rebase during sync, so we can alert the user that the given project
is in a state we cannot handle.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-16 11:21:18 -07:00
Shawn O. Pearce
48244781c2 Refactor error message display in project.py
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-16 08:25:57 -07:00
Shawn O. Pearce
19a83d8085 Use default rebase during sync instead of rebase -i
rebase interactive (aka rebase -i) has changed in newer versions
of git, and doesn't always generate the sequence of commits the
same way it used to.  It also doesn't handle having a previously
applied commit try to be applied again.

The default rebase algorithm is better suited to our needs.
It uses --ignore-if-in-upstream when generating the patch series
for git-am, and git-am with its 3-way fallback is able to handle
a rename case just as well as the cherry-pick variant used by -m.
Its also a generally faster implementation.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-16 08:23:29 -07:00
Shawn O. Pearce
161f445a4d status: tell the user the working tree is clean
If there is nothing output at all, tell the user the working tree is
completely clean.  It just gives them a bit more of a warm-fuzzy
feeling knowing repo and until the end.  It also more closely
matches with the output of git status.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-10 19:01:08 -07:00
Shawn O. Pearce
3e768c9dc7 Add 'repo sync -d' to detach projects from their current topic
The -d flag moves the project back to a detached HEAD state,
matching what is listed in the manifest.  This can be useful to
set a client to something stable (or at least well-known), such as
before a sequence of 'repo download' commands are used to get some
changes for testing.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-10 17:08:02 -07:00
Shawn O. Pearce
0a389e94de Make 'repo start' restartable upon failures
If `repo start foo` fails due to uncommitted and unmergeable changes
in a single project, we have switched half of the projects over to
the new target branches, but didn't on the one that failed to move.

This change improves the situation by doing three things differently:

- We keep going when we encounter an error, so other projects
  that can successfully switch still switch.

- We ignore projects whose current branch is already on the
  requested name; they are logically already setup.

- We checkout the branch if it already exists, rather than
  trying to recreate the branch.

Bug: REPO-22
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-10 16:21:18 -07:00
Shawn O. Pearce
2675c3f8b5 Don't capture stdout during 'repo checkout'
There isn't any great value in buffering stdout into memory
coming from git checkout.  So don't bother doing it.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-10 16:20:25 -07:00
Shawn O. Pearce
27b07327bc Add a repo branches subcommand to describe current branches
We now display a summary of the available topic branches in this
client, based upon a sorted union of all existing projects.

Bug: REPO-21
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-04-10 16:02:48 -07:00
Wink Saville
02d7945eb8 Add checkout command.
Teach repo how to checkout a branch in all projects or a list
of specific projects.

Bug: REPO-21
2009-04-10 13:01:24 -07:00
Shawn O. Pearce
c7a4eefa7e Add repo manifest -o to save a manifest
This can be useful to create a new manifest from an existing client,
especially if the client wants to use the "-r" option to set each
project's revision to the current commit SHA-1, making a sort of a
tag file that can be used to recreate this exact state elsewhere.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-03-05 10:32:38 -08:00
Shawn O. Pearce
2816d4f387 Set core.bare to true on mirror repositories
When creating a mirror repository we will always be using a bare
repository.  Setting $GIT_DIR/config to have core.bare = true is
reasonable and helps Git to recognize the environment it is in.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-03-03 17:53:18 -08:00
Shawn O. Pearce
6a5644d392 Get rid of the horrible android import work around hack
Months ago when the Android Open Source Project launched we had some
import errors that had to be fixed and worked over.  These hacks
were here to help users update their clients to newer versions of
the imported code.

Its very likely all clients have either been deleted, or have been
updated and have the fixed imports.  So we don't need this hack in
repo anymore.

If a very ancient client still existed, it would need to be created
from scratch anyway, due to the Android cupcake branch merging
into master and the manifest changes not being able to be handled
correctly by repo.  A new client wouldn't have the incorrectly
imported code in it, and thus wouldn't need this hack.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-03-03 13:52:20 -08:00
Shawn O. Pearce
fe08675956 Fix repo status when there are renamed/copied files
I missed a parameter in the format string, but still provided the
value in the parameter list, so the format failed to produce an
output message.

Bug: REPO-15
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-03-03 13:49:48 -08:00
Shawn O. Pearce
559b846b17 Report better errors when a project revision is invalid
If a manifest specifies an invalid revision property, give the
user a better error message detaling the problem, instead of an
ugly Python traceback with a strange Git error message.

Bug: REPO-2
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-03-02 12:56:08 -08:00
Shawn O. Pearce
7c6c64d463 Fix repo prune output to sort by branch name
We didn't always sort the output.  Now we do.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-03-02 12:38:45 -08:00
Shawn O. Pearce
3778f9d47e Fix repo prune to work on git 1.6.1-rc3~5 and later
Prior to git 1.6.1-rc3~5 the output of 'git branch -d' matched:

  Deleted branch (.*)\.

where the subgroup grabbed the branch name. In v1.6.1-rc3~5 (aka
a126ed0a01e265d7f3b2972a34e85636e12e6d34) Brandon Casey changed
the output to include the SHA-1 of the branch name, now matching
the pattern:

  Deleted branch (.*) \([0-9a-f]*\)\.

Instead of parsing the output of git branch we now re-obtain the
list of branches after the deletion attempt and perform a set
difference in memory to determine which branches we were able to
successfully delete.

Bug: REPO-9
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-03-02 12:38:36 -08:00
Shawn O. Pearce
370e3fa666 Remove the protobuf based HTTP upload code path
Now that Gerrit2 has been released and the only supported upload
protocol is direct git push over SSH we no longer need the large
and complex protobuf client library, or the upload chunking logic
in gerrit_upload.py.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-01-26 10:55:39 -08:00
Shawn O. Pearce
b54a392c9a Support Gerrit2's ssh:// based upload
In Gerrit2 uploads are sent over "git push ssh://...", as this
is a more efficient transport and is easier to code from external
scripts and/or direct command line usage by an end-user.

Gerrit1's HTTP POST based format is assumed if the review server
does not have the /ssh_info URL available on it.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-01-05 16:34:27 -08:00
Marcelo E. Magallon
21f7385400 Remove astray comma
There's an extra "," at the end of the line, which is causing
trouble when the manifest file specifies a revision for a
project.  Since the default manifest file doesn't specify
revisions for the projects, the problem has gone unnoticed.

Thanks to Barry Silverman <barry@disus.com> for spotting the
issue and providing a patch.

Signed-off-by: Marcelo E. Magallon <marcelo.magallon@gmail.com>
2008-12-31 04:45:04 +00:00
Joe Onorato
2896a79120 Add --review and --cc flags to repo upload, so you can
assign reviewers when you upload changes.
2008-11-19 11:55:06 -05:00
Shawn O. Pearce
c99883fee9 Teach 'repo upload --replace' how to add replacement patch sets
Users are prompted with the list of known changes we are about
to upload, and they can fill out the current change numbers for
any changes which already exist in the data store.  For each of
those changes the change number and commit id is sent as part of
the upload request, so Gerrit can insert the new commit as a new
patch set of the existing change, rather than make a new change.

This facility permits developers to replace a patch so they can
address comments made on a prior version of the same change.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2008-11-12 09:12:19 -08:00
Shawn O. Pearce
35f2596c27 Refactor part of GetUploadableBranches to lookup one specific branch
This way project.GetUploadableBranch(project.CurrentBranch) can tell
us how (if at all) to upload the currently checked out branch.

Signed-off-by: Shawn O. Pearce <sop@google.com>
2008-11-12 09:12:17 -08:00