Commit Graph

310 Commits

Author SHA1 Message Date
David Pursehouse
8016f60a46 Merge "repo: Repo does not always handle '.' parameter correctly" 2016-08-14 08:50:28 +00:00
Cheuk Leung
8ac0c96537 Fix removing broken symlink in reference dir
Re-ordered to first create the symlink before checking the source
file and remove the destination if the source does not exists.

Change-Id: Iae923ba2ef0ba5a8dc1b8e42d8cc3f3708f773af
2016-06-29 05:43:01 +00:00
David Pursehouse
a36af0767b Merge "Fix variable assignment" 2016-06-29 01:43:41 +00:00
Aymen Bouaziz
037040f73e Fix variable assignment
If upstream string is empty, current_branch_only variable will be assigned
to an empty string.

This is not what we expect here as this variable is a boolean.

Change-Id: Ibba935e25a74c2be1e50c88b4b403cf394ba365e
2016-06-28 12:31:25 +02:00
Aymen Bouaziz
2598ed06f1 Fix submodule checkout error when using sync-s option
When sync-s="true" option is used, the checkout of a submodule will try
to use the revision attribute of the parent project.

If this revision is a named reference, the checkout will fail if there
is no reference with this name in the submodule.

The proposed solution is to use the git commit id as revisionExpr for
submodules.

Change-Id: Ie8390a11957fd6a9c61289c6861d13cb3fa11678
2016-06-27 20:16:45 +02:00
Dan Willemsen
8b39fb4bc0 Merge "Fix XmlManifest.Save with remotes that have 'alias' set" 2016-04-22 18:28:57 +00:00
Dan Willemsen
96c2d65489 Fix XmlManifest.Save with remotes that have 'alias' set
When the alias attribute is set for a remote, the RemoteSpec attached to
a Project only contains the alias name used by git, not the original
name used in the manifest. But that's not enough information to
reconstruct the manifest, so save off the original manifest name as
another RemoteSpec parameter, only used to write the manifest out.

Bug: Issue 181
Bug: Issue 219
Change-Id: Id7417dfd6ce5572e4e5fe14f22924fdf088ca4f3
2016-04-22 10:32:06 +09:00
Sebastian Schuberth
7ecccf6225 diffmanifests: support custom git pretty format strings
Change-Id: I29f4f1351c421f393328514d145df1a96aed9ee2
2016-04-21 18:36:11 +00:00
Mark E. Hamilton
f9fe3e14d2 repo: Repo does not always handle '.' parameter correctly
The repo script allows a manifest to specify a '.' as the path the
top-level directory, which co-locates the .git and .repo directories,
and places files from the git repository at the top-level:

  <project name="proj_name" path="." />
  <project name="sierra.other.git" path="other" />

Most commands work correctly with this setup. Some commands, however,
fail to find the project. For instance, 'repo sync' works, and 'repo sync .'
works in a sub-project ('other' in this case) but 'repo sync .' in the
top-level directory fails with the error:

error: project . not found

There are two reasons for this:

1. The self.worktree attribute of the Project object is not normalized,
so with a '.' for path its value would be '/my/project/root/.'. This is
fine when used as a path, since it's the same path as '/my/project/root',
but when used in a string comparison it fails. This commit applies
os.path.normpath() to that value before storing it.

2. The _GetProjectByPath method in command.py was not checking the path
against manifest.topdir, so even once it was normalized the project was
not found. This commit adds a check against manifest.topdir if the
loop drops out without finding a project.

Change-Id: Ic84d053f1bbb5a357cad566805d5a326ae8246d2
2016-04-08 00:07:52 +00:00
Dan Willemsen
bdb866ea76 Fix symlinking of new projects
We weren't copying these lists, so the += was actually changing the
underlying lists.

When a new project was added to the manifest, we run _CheckDirReference
against the manifest project with share_refs=True, which added the
working_tree_* to the shareable_* lists. Then, when we load the new
manifest and create the new project, it uses the lists that already
contain the working_tree_* files, even though we passed
share_refs=False.

This happens reliably under the above conditions, but doesn't seem to
happen when syncing a fresh tree. So we've got a mixture of links that
may need to be cleaned up later. This patch will just stop it from
happening in the future.

Change-Id: Ib7935bfad78af1e494a75e55134ec829f13c2a41
2016-04-05 17:44:09 -07:00
David Pursehouse
936d6185eb Merge changes from topic 'pylint-pep8-cleanup'
* changes:
  command.py: Cleaned up pylint/pep8 violations
  project.py: Cleaned up pylint/pep8 violations
2016-03-15 00:31:12 +00:00
Mike Frysinger
4aa4b211c6 RepoHook: set __file__ when running the hook
A common design pattern is to use __file__ to find the location of the
active python module to assist in output or loading of related assets.
The current hook systems runs the pre-upload.py hook in a context w/out
that set leading to runtime errors:

$ repo upload --cbr .
ERROR: Traceback (most recent call last):
  File ".../repo/project.py", line 481, in _ExecuteHook
    self._script_fullpath, 'exec'), context)
  File ".../repohooks/pre-upload.py", line 32, in <module>
    path = os.path.dirname(os.path.realpath(__file__))
NameError: name '__file__' is not defined

Define this variable in this context so code can safely use it.

Change-Id: If6331312445fa61d9351b59f83abcc1c99ae6748
2016-03-05 21:52:31 +00:00
Mark E. Hamilton
30b0f4e022 project.py: Cleaned up pylint/pep8 violations
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a number of violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the project.py script.

This script now has a pylint score of 10.0, and no violations reported
by pep8.

Change-Id: I1462fd84f5b6b4c0dc893052671373e7ffd838f1
2016-03-02 09:05:40 -07:00
Dan Willemsen
1a799d14b7 Fix prune when bare git has detached head
We don't really use HEAD much in the bare git repositories, but there
have been reports of errors in git-symbolic-ref:

  symbolic-ref: fatal: Refusing to point HEAD outside of refs/

That happen when the bare git repo is in the detached head state. It's
possible that previous operations were killed while we were pruning
branches.

Use DetachHead instead of SetHead if we're restoring the repo into a
detached head state.

Change-Id: I9062e8957bc70367d3ded399685ac026fbb421fc
2015-12-15 14:22:40 -08:00
Dan Willemsen
e1e0bd1f75 Check for broken links when updating linkfiles
If a linkfile is a broken link (destination does not exist), and it
needs to be updated, we didn't notice that it needed to be removed
first. Use lexists instead of exists to check for this condition.

Change-Id: I1f6a1f0193d3fd2b9f7a647836044997f6ab32eb
2015-11-18 16:51:51 -08:00
David Pursehouse
74cfd2709b Sync: Add option to prune refs during sync
By passing --prune to the sync command, the --prune option is
given to the `git fetch`, causing refs that no longer exist on
the remote to be removed.

Change-Id: I3cedacce14276d96ac2d5aabf2d07fd05e92bc02
2015-10-27 03:04:17 +00:00
Dan Willemsen
029eaf3bac _CopyAndLinkFiles even if the sources haven't changed
The source or destination attributes may have changed even if the source
didn't, so we need to make sure that these are up to date.

Change-Id: I266ef3598ddda7e8c23bc9c6a049905ddc586348
2015-09-03 12:54:06 -07:00
Simran Basi
b9a1b73425 GITC: Add repo start support.
Add repo start support for GITC checkouts. If the user is in
the GITC FS view, they can now run repo start to check out
the sources and create a new working branch.

When "repo start" is called on a GITC project, the revision
tag is set to an empty string and saved in a new tag:
old-revision. This tells the GITC filesystem to display the
local copy of the sources when being viewed. The local copy
is created by pulling the project sources and the new branch
is created based off the original project revision.

Updated main.py to setup each command's gitc_manifest when
appropriate.

Updated repo sync's logic to sync opened projects and
updating the GITC manifest file for the rest.

Change-Id: I7e4809d1c4fc43c69b26f2f1bebe45aab0cae628
2015-08-28 10:53:05 -07:00
David Pursehouse
dc2545cad6 project.py: Improve message shown when hook is not replaced
If a hook file has been modified locally, it will not be replaced.

Improve the message to make this clearer.

Also change it from an error to a warning.

Change-Id: I62c635390f24d2868db17717c247861b0381c99f
2015-08-25 05:40:46 +00:00
David Pursehouse
f33929d014 project.py: Consistently use the _error method to print error messages
Use the _error method instead of directly calling `print`.

Also add a new _warn convenience method.

Change-Id: Ia332c14ef8d9d1fe2df128dbf36b5521802ccdf1
2015-08-25 14:39:06 +09:00
Dan Willemsen
7c9263bce0 Merge "Support smart-sync through persistent-http[s]" 2015-08-19 23:36:35 +00:00
Dan Willemsen
dab9e99f0f Merge "Fix formatting of message when retrying clone" 2015-08-19 17:54:50 +00:00
Dan Willemsen
c5f15bf7c0 Merge "Include project path in --force-sync error message" 2015-08-19 17:53:28 +00:00
Dan Willemsen
0745bb2657 Support smart-sync through persistent-http[s]
Use the same cookies and proxy that git traffic goes through for
persistent-http[s] to support authentication for smart-sync.

Change-Id: I20f4a281c259053a5a4fdbc48b1bca48e781c692
2015-08-19 10:22:11 -07:00
David Pursehouse
25857b8988 Fix formatting of message when retrying clone
Passing the force_sync variable into the string formatting results in
the message:

  "Retrying clone after deleting None"

or

  "Retrying clone after deleting True".

Pass the name of the git directory instead.

Also, move the print inside the if-block so it's only displayed
when the retry is actually going to be attempted.

Change-Id: I76d9ecc176cecee4ad512d13e9d1f6bd36aacbbb
2015-08-19 13:03:13 +00:00
Simon Ruggier
f9b7683a3b Include project path in --force-sync error message
For projects that have been cloned outside of the repo command (or
cloned a long time ago), commit abaa7f312f
introduced an error message to invite the user to use --force-sync.
However, due to the risk of data loss, it's useful to know which
project's git directory is being replaced before deciding whether or not
to provide --force-sync.

This change updates the exception's associated value to include the
project's relative path and explain to the user how they can resolve the
issue. A previous version of this commit used the project name. However,
for projects that have multiple work trees, the name can be ambiguous,
while the path clearly identifies which git directory will be replaced.

Change-Id: If717e66fda4d19accc0a8e889a91f4cd4ff14dff
2015-08-04 18:41:20 -04:00
Dan Willemsen
eeab6860f1 Fix shallow clone behavior
The existing code here makes sure that switching clone-depth from on to
off actually causes the history to be fully restored. Unfortunately, it
does this by fetching the full history every time the fetch spec
changes. Switching between two clone-depth="1" branches will fetch far
more than the top commit.

Instead, when not using clone-depth, pass --depth=2147483647 to git
fetch so that it ensures that we have the entire history. That is
slightly less efficient, so limit it to only when there are shallow
objects in the project by checking for the existance of the 'shallow'
file.

Change-Id: Iee0cfc9c6992c208344b1d9123769992412db67b
2015-08-03 16:54:16 -07:00
Dan Willemsen
2a3e15217a Fix _ReferenceGitDir symlinking
This fixes these errors:

  ...
  File ".repo/repo/project.py", line 2371, in _ReferenceGitDir
    os.symlink(os.path.relpath(src, os.path.dirname(dst)), dst)
  OSError: [Errno 17] File exists

Which was happening for checkouts that were created before v1.12.8, when
project-objects was created. Nothing had yet been forcing these
checkouts to use project-objects, until the recent verification changes.

In this OSError case, we already created the symlink, so src == dst, and
the directory did not exist. This caused us to run os.makedirs the
os.symlink on the same file.

dst really should be the file in gitdir, not the target of that symlink
if it exists. So just use realpath for the dotgit portion of the path.

Change-Id: Iff5396a2093de91029c42cf38aa57131fd22981c
2015-07-30 21:29:53 -07:00
Kevin Degi
abaa7f312f Add option to correct gitdir when syncing
In some cases, a user may wish to continue with a sync even though
it would require overwriting an existing git directory. This behavior
is not safe as a default because it could result in the loss of some
user data, but as an optional flag it allows the user more flexibility.

To support this, add a --force-sync flag to the sync command that will
attempt to overwrite the existing git dir if it is specified and the
existing git dir points to the wrong obj dir.

Change-Id: Ieddda8ad54e264a1eb4a9d54881dd6ebc8a03833
2015-07-29 14:44:46 -06:00
Conley Owens
7cccfb2cf0 Merge "InitGitDir: Clean up created directories" 2015-07-29 18:49:12 +00:00
Kevin Degi
b1a07b8276 InitGitDir: Clean up created directories
If _InitGitDir fails, it leaves any progress it had made on the file
system. This can cause subsequent calls to repo sync to behave
differently. This is especially evident when _CheckDirReference() fails,
since it will not be invoked when sync is retried because both the
source and destination directories already exist.

To address this, have _InitGitDir() clean up any directories it has created
if it catches an exception. Also behave the same way for _InitWorkTree().

Change-Id: Ic16bb3feea649e115b59bd44be294e89e3692aeb
2015-07-27 13:33:43 -06:00
David Pursehouse
4e16c24981 Revert "Add --prune option to fetch when syncing a mirror repo"
For some users it is not desirable to remove refs that don't exist
on the remote server when syncing a mirror repo.

This reverts commit b4d43b9f66.

Change-Id: Ie849b66682138ef88da6cd1a5fbb27e993197dd7
2015-07-20 22:31:04 +09:00
Conley Owens
b3d6e67196 Merge "Fail if gitdir does not point to objdir during sync" 2015-07-15 19:30:41 +00:00
Kevin Degi
679bac4bf3 project.RemoteFetch: Handle depth cases more robustly
The fetch logic for the case where depth is set and revision is a
SHA1 has several failure modes that are not handled well by the
current logic.

1) 'git fetch <SHA1>' requires git version >= 1.8.3
2) 'git fetch <SHA1>' can be prevented by a configuration option on the server.
3) 'git fetch --depth=<N> <refspec>' can fail to contain a SHA1 specified by
   the manifest.

Each of these cases cause infinite recursion when _RemoteFetch() tries to call
itself with current_branch_only=False because current_branch_only is set to
True when depth != None.

To try to prevent the infinite recursion, we set self.clone_depth to None
before the first retry of _RemoteFetch(). This will allow the Fetch to
eventually succeed in the case where clone-depth is specified in the manifest.
A user specified depth from the init command will still recurse infinitely.

In addition, never try to fetch a SHA1 directly if the git version being used
is not at least 1.8.3.

Change-Id: I802fc17878c0929cfd63fff611633c1d3b54ecd3
2015-07-15 15:53:14 +00:00
Wink Saville
4c426ef1d4 Teach _LinkFile._Link to handle globs.
This allows a project to use globs in the linkfile src attribute. When
a glob is used in the src the dest field must be a directory. Then
_LinkFile._Link(self) calls will create symbolic links in the dest
directory to all of the entries in the src as defined by the glob
specification.

Below all of the entries in master-configs/ will have symbolic links
in <root dir>/configs directory:

  <project name="helloworld.git" path="apps/helloworld">
      <linkfile src="master-configs/*" dest="configs"/>
  </project>

Change-Id: Idfed8fa47c83d2ca6e2b8e867731b8e2f9e2eb47
2015-06-03 08:05:17 -07:00
David Pursehouse
472ce9f5fa Merge changes I32da12c2,Ie4a65b3e
* changes:
  Skip sleep and retry if git remote update exits with a signal
  Catch exceptions in project list generator
2015-06-02 00:14:43 +00:00
Colin Cross
0184dcc510 Make linkfile symlinks relative
The source (target) of the symlink is specified relative to a project
within a tree, and the destination is specified relative to the top
of the tree, so it should always be possible to create a relative symlink
to the target file.  Relative symlinks will allow moving an entire tree
without breaking the symlink, and copying a tree (with -p) without leaving
a symlink to the old tree.

Change-Id: I16492a8b59a137d2abe43ca78e3b212e2c835599
2015-06-01 01:24:38 +00:00
Colin Cross
c4b301f988 Skip sleep and retry if git remote update exits with a signal
Pressing ctrl-c during repo sync often hangs for 30 to 45 seconds
due to the time.sleep and retry in _RemoteFetch.  If git exits with
a signal, for example -2 for SIGINT triggered by ctrl-c, skip the
sleep and retry.

Change-Id: I32da12c2dcc96d9cc0b12a066e824b12ebfb52a0
2015-05-13 18:11:34 +00:00
Kevin Degi
384b3c5948 Fail if gitdir does not point to objdir during sync
There are a set of cases that can cause the git directory in
.repo/projects to point to a directory in .repo/project-objects that
is not the one specified in the manifest. This results in a tree that
is not sane, and so should cause a failure.

In order to reproduce the failure case:
1) Sync to any manifest
2) Change the 'name' of a project to a different repository. Leave the
   'path' the same.
3) Resync the modified project. The project-objects directory will not
   be created, and the projects directory will remain pointed at the old
   project-objects.

Change-Id: Ie6711b1c773508850c5c9f748a27ff72d65e2bf2
2015-05-12 09:15:53 -06:00
David Pursehouse
35de228f33 Merge "Don't attempt to create "fully qualified names" for SHA1s" 2015-05-11 09:20:54 +00:00
David Pursehouse
ace097c36e Merge "Add option on sync to avoid fetching from remotes for existing sha1" 2015-05-01 07:51:52 +00:00
David Pursehouse
b155354034 Add option on sync to avoid fetching from remotes for existing sha1
In 2fb6466f79 an optimisation was
added to avoid fetching from remotes if the project is fixed to
a revision and the revision is already available locally.

This causes problems for users who expect all objects to be
fetched by default.

Change the logic so that the optimized behaviour is only enabled if
an option is explicitly given to repo sync.

Change-Id: I3b2794ddd8e0071b1787e166463cd8347ca9e24f
2015-04-30 14:29:02 +00:00
Alexandre Boeglin
382582728e Don't attempt to create "fully qualified names" for SHA1s
Doing so breaks "repo init -b <SHA1>".

Change-Id: Ic071a1b099a9125db22ea446d7e92e7854d69b37
2015-04-30 14:54:47 +02:00
David Pursehouse
b4d43b9f66 Add --prune option to fetch when syncing a mirror repo
When syncing a mirror repo, add the --prune option to the fetch
command to force removal of stale refs from the mirror.

Change-Id: I4b43b2a5c86b9915627887c16f6569066f3ab978
2015-04-30 10:32:37 +09:00
David Pursehouse
403b64edf4 Don't append branch to fetch spec when syncing to a mirror
Appending the branch to the fetch spec causes sync of a mirror to
fail for projects that don't have an explicit revision specified,
and don't have a branch of the same name as the default revision.

For example, a manifest defining a default revision:

 <default revision="master">

having a project without an explicit revision:

 <project name="path/to/project">

and not having a branch named "master", will cause repo sync to
fail for that project with the error:

 Couldn't find remote ref refs/heads/master

Modify the logic to not append the branch onto the fetch spec when
syncing to a mirror.

Change-Id: I5c4457bd125519abf27abe682dea62ad708978c9
2015-04-27 10:56:27 +09:00
Conley Owens
04f2f0e186 Maintain fully qualified tracking branches
When running repo branch, the git merge line (in many circumstances)
is set to the revision of the project specified in the manifest.  If
this is a branch name that is not fully-qualified, we will end up with
something like "merge = master" instead of "merge = refs/heads/master".
This change examines the revision if we are going to use that and
changes branch short names to fully qualified branch names.

Change-Id: Ie1be94fb8d45df8eeac44a47f729a3819a05fa81
2015-04-01 17:43:36 +00:00
Anthony King
23ff7df6a7 use the max depth instead of unshallow
This allows the use of older versions of git

Change-Id: I88ea685066603af19896a791829355ddbfa91ffe
2015-03-30 21:54:26 +00:00
John L. Villalovos
9c76f67f13 Always capture output for GitCommand
Switch the GitCommand program to always capture the output for stdout
and stderr.  And by default print the output while running.

The options capture_stdout and capture_stderr have effectively become
options to supress the printing of stdout and stderr.

Update the 'git fetch' to use '--progress' so that the progress messages
will be displayed.  git checks if the output location isatty() and if it
is not a TTY it will by default not print the progress messages.

Change-Id: Ifdae138e008f80a59195f9f43c911a1a5210ec60
2015-03-26 11:43:55 -07:00
Jonathan Nieder
9371979628 Revert "Implementation of manifest defined githooks"
This reverts commit 38e4387f8e.

A "repo init" followed by "repo sync" is meant to be as safe as
"git clone".  In particular it should not run arbitrary code provided
by the manifest owner.

It would still be nice to have support for manifest-defined git hooks
--- they'd just need a prompt like the upload RepoHook has.  Hopefully
a later change can bring them back.

Change-Id: I5ecd90fb5c2ed64f103d856d1ffcba38a47b062d
Signed-off-by: Jonathan Nieder <jrn@google.com>
2015-03-17 11:29:58 -07:00
Jakub Vrana
0402cd882a Add space between project path and branch in repo status.
Currently, paths longer than 39 chars have no space after them so it looks
like this:

project path/branch master

Change-Id: I4c1bb13648ac099ade8a8d4ebafa04131571f842
2015-03-11 07:42:17 +00:00
Conley Owens
85e8267031 Merge "Implementation of manifest defined githooks" 2015-03-05 20:52:30 +00:00
John L. Villalovos
e30f46b957 Print stderr output from git command for RemoteFetch
The stderr output generated by git during a RemoteFetch was not being
printed.  This information is useful so print it.

Change-Id: I6e6ce12c4a57e5ca2359f76ce14f2fcbbc37a5ef
2015-02-25 14:29:28 -08:00
Conley Owens
e4978cfbe3 Ensure the repo project is never fetched with partial depth
If the repo project is synced with partial depth, then the tags
won't be fetched and users will be told the newest sha1 in the
stable branch isn't signed.

Change-Id: I107df97b4836b928c76aa33a700fa35d1705ae09
2015-02-10 14:44:05 -08:00
John L. Villalovos
126e298214 Handle case where 'git remote prune' needs to be run
Handle the case when this error occurs:
    error: some local refs could not be updated; try running
     'git remote prune origin' to remove any old, conflicting branches

This is usually caused by a reference getting changed from a file to a
directory.

For example:
  Initially someone creates a branch 'foo' and it is stored as:
    .git/refs/remotes/origin/foo

  Then later on it is decided to change the layout structure where 'foo'
  is a directory with branches below it:
    .git/refs/remotes/origin/foo/master

  The problem occurs when someone still has
  '.git/refs/remotes/origin/foo' on their system and does a repo sync.
  When this occurs the error message for needing to do a
  'git remote prune origin' occurs.

Now when doing a 'git fetch' if the error message from git says that a
'git remote prune' is needed, it will do the prune and then retry the
fetch.

Change-Id: I4c6f5aa6bd932f0ef7a39134400bedd52e82f633
Signed-off-by: John L. Villalovos <john.l.villalovos@intel.com>
2015-02-03 13:49:51 -08:00
Jimmie Wester
38e4387f8e Implementation of manifest defined githooks
When working within a team or corporation it is often
useful/required to use predefined git templates. This
change teaches repo to use a per-remote git hook template
structure.

The implementation is done as a continuation of the
existing projecthook functionality. The terminology is
therefore defined as projecthooks.

The downloaded projecthooks are stored in the .repo
directory as a metaproject separating them from the users
project forest.

The projecthooks are downloaded and set up when doing a
repo init and updated for each new repo init.

When downloading a mirror the projecthooks gits are
not added to the bare forest since the intention is to
ensure that the latest are used (allows for company policy
enforcement).

The projecthooks are defined in the manifest file in the
remote element as a subnode, the name refers to the
project name on the server referred to in the remote.
<remote name="myremote ..>
   <projecthook name="myprojecthookgit" revision="myrevision"/>
</remote>

The hooks found in the projecthook revision supersede
the stock hooks found in repo. This removes the need for
updating the projecthook gits for repo stock hook changes.

Change-Id: I6796b7b0342c1f83c35f4b3e46782581b069a561
Signed-off-by: Patrik Ryd <patrik.ryd@stericsson.com>
Signed-off-by: Ian Kumlien <ian.kumlien@gmail.com>
2015-02-03 16:01:15 +09:00
Conley Owens
db6f1b0884 Merge "Use depth flag when fetching" 2015-01-30 19:36:06 +00:00
Conley Owens
ee69084421 Merge "Handle shallow checkout of SHA1 pinned repos" 2015-01-28 20:29:37 +00:00
Conley Owens
d37d43f036 Merge "Don't delete hooks in .git/hooks" 2015-01-28 20:29:05 +00:00
Anthony King
7bdac71087 pylint fixes for project.py
Fix all the formatting warnings and unused variables

Change-Id: I17d88a23572303879530077f3a80451de5417fbb
2015-01-22 04:20:21 +00:00
Conley Owens
f97e8383a3 Use depth flag when fetching
Currently, we only use the depth flag when cloning.  The result is that when
new project history has merges, the entire history of the merged branch is
brought in and the project becomes unshallow very quickly.  --depth and
clone-depth are often used to save on space, not just network load, so this
seems less than ideal.

This change uses --depth on every fetch (when the user has depth specified),
not just the initial clone.  The result is that the given project stays
consistently shallow as opposed to growing over time, especially when merges
are involved.

Change-Id: Iac706cfdad4a555c72f9d9f1119195d38d91df12
2015-01-22 01:20:22 +00:00
Bertrand SIMONNET
3000cdad22 Handle shallow checkout of SHA1 pinned repos
When doing a shallow checkout SHA1 pinned repos with repo init --depth=1 and
repo sync -c, repo would try to fetch only some reference and fail if the exact
SHA1 repo was missing.
Instead, when depth is set, fetch only the specific commit.

Change-Id: If3f799d0e78c03faea47f796380bb5e367b11998
2015-01-21 14:14:23 -08:00
Mitchel Humpherys
b9d9efd394 Don't delete hooks in .git/hooks
We currently delete all hooks in .git/hooks for each project before
symlink'ing in the standard project hooks.  This can be annoying for
users who have installed custom git hooks.

There's no reason to delete all existing hooks.  Just rip out the
deletion code.

Change-Id: I5062a6cd20af700f6d6a17b11ad6c94853987c57
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
2015-01-15 22:49:08 -08:00
Dave Borowitz
497bde4de5 Respect --quiet when looking up bundle cookie file
Change-Id: I02a244132c49e4bb50ecda978974d6d2b220f6d1
2015-01-02 13:58:05 -08:00
Dave Borowitz
4abf8e6ef8 Save cookies back to jar when fetching clone.bundle
Change-Id: I3ef71b5e7f8ee1cda66057e46ae234866c7258c4
2015-01-02 13:57:14 -08:00
Dave Borowitz
137d0131bf Hold persistent proxy connection open while fetching clone.bundle
The persistent proxy may choose to present a per-process cookie file
that gets cleaned up after the process exits, to help with the fact
that libcurl cannot save cookies atomically when a cookie file is
shared across processes. We were letting this cleanup happen
immediately by closing stdin as soon as we read the configuration
option, resulting in a nonexistent cookie file by the time we use the
config option.

Work around this by converting the cookie logic to a context manager
method, which closes the process only when we're done with the cookie
file.

Change-Id: I12a88b25cc19621ef8161337144c1b264264211a
2015-01-02 13:57:13 -08:00
Kris Giesing
c8d882ae2a Silence warnings about invalid clone.bundle files when quieted
The invalid clone.bundle file warning is not typically user actionable,
and can be confusing. So don't show it when -q flag is in effect.

Change-Id: If9fef4085391acf54b63c75029ec0e161c38eb86
2014-12-24 10:23:24 +09:00
Conley Owens
3eb87cec5c Revert "Check for existence of refs upon initial fetch"
This reverts commit 565480588d.

We are reverting this change for 2 reasons:

1) It introduced a bug for users using sync -c with a reference mirror.
2) The fetch specs have recently changed to cause git to properly fail
when we request a non-existent branch of a manifest, removing the need
for this change.

Change-Id: I0f63da9bfb40cf5ffafb7979f1b8c929a738fc7b
2014-11-10 23:49:32 +00:00
Conley Owens
7893b85509 Merge changes I1f71be22,I5b119f11
* changes:
  Always fetch the specific revision given
  Support specifying non-HEADS refs as upstream
2014-10-22 00:23:18 +00:00
Vadim Bendebury
14e134da02 upload: report names of uncommitted files
When there are uncommitted files in the tree, 'repo upload' stops to
ask if it is OK to continue, but does not report the actual names of
uncommitted files.

This patch adds plumbing to have the outstanding file names reported
if desired.

BUG=None
TEST=verified that 'repo upload' properly operates with the following
    conditions present in the tree:
    . file(s) modified locally
    . file(s) added to index, but not committed
    . files not known to git
    . no modified files (the upload proceeds as expected)

Change-Id: If65d5f8e8bcb3300c16d85dc5d7017758545f80d
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Signed-off-by: Vadim Bendebury <vbendeb@google.com>
2014-10-14 11:20:05 -07:00
Nasser Grainawi
04e52d6166 Always fetch the specific revision given
Don't assume the revision is in refs/heads/.

Change-Id: I1f71be222ed3ed940d2265aad43d1f2d601fc03a
2014-10-09 13:41:56 -06:00
Nasser Grainawi
909d58b2e2 Support specifying non-HEADS refs as upstream
While not typical, some users might have an upstream that isn't in
the usual refs/heads/* namespace. There's no reason not to use
those refs as the value for the upstream attribute, so support
doing so.

Change-Id: I5b119f1135c3268c20e7c4084682e860d3ee1fb1
2014-10-09 13:41:51 -06:00
Chirayu Desai
303a82f33a Don't open non-binary files as binary
* Don't pen the git config file, and the git ".lock" file as binary.

Change-Id: I7b3939658456f2fd0a0500443cdd8d1ee1a4459d
2014-08-19 23:05:44 +05:30
Conley Owens
80b87fe6c1 Use fetch --unshallow when appropriate.
If a user reinits to a different manifest or the manifest updates so
that a project no longer has a fixed depth, we need to use --unshallow
when we fetch.

Change-Id: I6d3f15e5464b5eaad9205654bc24354947a78aea
2014-05-09 18:47:35 -07:00
Conley Owens
e9f75b1782 Merge "Enable remotes to define their own revision" 2014-05-08 18:38:33 +00:00
Conley Owens
5196805fa2 Merge "Use exec() rather than execfile()" 2014-05-07 18:18:56 +00:00
Anthony King
36ea2fb6ee Enable remotes to define their own revision
Some projects use multiple remotes.
In some cases these remotes have different naming conventions.
Add an option to define a revision in the remote configuration.

The `project` revision takes precedence over `remote` and `default`.
The `remote` revision takes precedence over `default`.
The `default` revision acts as a fall back as it originally did.

Change-Id: I2b376160d45d48b0bab840c02a3eef1a1e32cf6d
2014-05-07 08:29:30 +00:00
Anthony King
2cd1f0452e Use next(iterator) rather than iterator.next()
iterator.next() was replaced with iterator.__next__() in Python 3.
Use next(iterator) instead which will select the correct method for
returning the next item.

Change-Id: I6d0c89c8b32e817e5897fe87332933dacf22027b
2014-05-07 08:44:20 +01:00
Anthony King
70f6890352 Use exec() rather than execfile()
execfile() is not in Python 3.

Change-Id: I5af222340f13c1e8edaa820e7675d3e4d62a1689
2014-05-05 23:41:07 +01:00
Conley Owens
666d534636 Ensure HEAD is correct when skipping remote fetch
A recent optimization (2fb6466f79) skips
performing a remote fetch if we already know we have the sha1 we want.
However, that optimization skipped initialization steps that ensure HEAD
points to the correct sha1.  This change makes sure not to skip those
steps.

Here is an example of how to test this change:

"""""""""
url=<manifest url>
branch1=<branch name>
branch2=<branch name>
project=<project with revision set to different sha1 in each branch>

repo init -u $url -b $branch1 --mirror
repo sync $project
first=$(cd $project.git; git rev-parse HEAD)

repo init -b $branch2
repo sync $project
second=$(cd platform/build.git; git rev-parse HEAD)

if [[ $first == $second ]]
then
    echo 'problem!'
else
    echo 'no problem!'
fi
"""""""""
2014-05-01 13:20:32 -07:00
Conley Owens
f2af756425 Add 'shallow' gitfile to symlinks
This fixes the bug that kept clients from doing things like `git log`
in projects using the clone-depth feature.

Change-Id: Ib4024a7b82ceaa7eb7b8935b007b3e8225e0aea8
2014-04-30 11:34:00 -07:00
Conley Owens
544e7b0a97 Merge "Ignore clone-depth attribute when fetching to a mirror" 2014-04-24 21:21:02 +00:00
Jeff Hamilton
e0df232da7 Add linkfile support.
It's just like copyfile and runs at the same time as copyfile but
instead of copying it creates a symlink instead.  This is needed
because copyfile copies the target of the link as opposed to the
symlink itself.

Change-Id: I7bff2aa23f0d80d9d51061045bd9c86a9b741ac5
2014-04-22 14:35:47 -05:00
David Pursehouse
9bc422f130 Ignore clone-depth attribute when fetching to a mirror
If a manifest includes projects with a clone-depth=1 attribute, and a
workspace is initialised from that manifest using the --mirror option,
any workspaces initialised and synced from the mirror will fail with:

  fatal: attempt to fetch/clone from a shallow repository

on the projects that had the clone-depth.

Ignore the clone-depth attribute when fetching from the remote to a
mirror workspace. Thus the mirror will be synched with a complete
clone of all the repositories.

Change-Id: I638b77e4894f5eda137d31fa6358eec53cf4654a
2014-04-16 11:00:40 +09:00
Mitchel Humpherys
26c45a7958 Make --no-tags work with -c
Currently, the --no-tags option is ignored if the user asks to only
fetch the current branch. There is no reason for this restriction. Fix
it.

Change-Id: Ibaaeae85ebe9955ed49325940461d630d794b990
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
2014-03-12 16:34:53 +09:00
David Pursehouse
68425f4da8 Fix indentation in project.py
Change-Id: I81c630536eaa54d5a25b9cb339a96c28619815ea
2014-03-11 14:55:52 +09:00
Dan Sandler
53e902a19b More verbose errors for NoManifestExceptions.
The old "manifest required for this command -- please run
init" is replaced by a more helpful message that lists the
command repo was trying to execute (with arguments) as well
as the str() of the NoManifestException. For example:

> error: in `sync`: [Errno 2] No such file or directory:
> 	'path/to/.repo/manifests/.git/HEAD'
> error: manifest missing or unreadable -- please run init

Other failure points in basic command parsing and dispatch
are more clearly explained in the same fashion.

Change-Id: I6212e5c648bc5d57e27145d55a5391ca565e4149
2014-03-11 05:33:43 +00:00
Chris AtLee
2fb6466f79 Don't fetch from remotes if commit id exists locally
In existing workspaces where the manifest specifies a commit id in the
manifest, we can avoid doing a fetch from the remote if we have the
commit locally. This substantially improves sync times for fully
specified manifests.

Change-Id: Ide216f28a545e00e0b493ce90ed0019513c61613
2014-03-03 10:17:03 +00:00
Julien Campergue
dd6542268a Add the "diffmanifests" command
This command allows a deeper diff between two manifest projects.
In addition to changed projects, it displays the logs of the
commits between both revisions for each project.

Change-Id: I86d30602cfbc654f8c84db2be5d8a30cb90f1398
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
2014-02-17 11:20:11 +00:00
Conley Owens
565480588d Check for existence of refs upon initial fetch
When we do an initial fetch and have not specified any branch etc,
the following fetch command will not error:
git fetch origin --tags +refs/heads/*:refs/remotes/origin/*

In this change we make sure something got fetched and if not we report
an error.

This fixes the bug that occurs when we init using a bad manifest url and
then are unable to init again (because a manifest project has been
inited with no manifest).

Change-Id: I6f8aaefc83a1837beb10b1ac90bea96dc8e61156
2014-02-12 09:11:00 -08:00
Conley Owens
c9877c7cf6 Merge "Only fetch current branch on shallow clients" 2014-01-29 21:12:34 +00:00
Shawn Pearce
69e04d8953 Only fetch current branch on shallow clients
Fetching a new branch on a shallow client may download the entire
project history, as the depth parameter is not passed to git
fetch. Force the fetch to only download the current branch.

Change-Id: Ie17ce8eb5e3487c24d90b2cae8227319dea482c8
2014-01-29 12:48:54 -08:00
Conley Owens
e695338e21 Merge "repo: Support multiple branches for the same project." 2014-01-10 01:20:13 +00:00
Conley Owens
bd80f7eedd Merge "Canonicalize project hooks path before use" 2014-01-09 02:11:10 +00:00
Julien Campergue
335f5ef4ad Add --archive option to init to sync using git archive
This significantly reduces sync time and used brandwidth as only
a tar of each project's revision is checked out, but git is not
accessible from projects anymore.

This is relevant when git is not needed in projects but sync
speed/brandwidth may be important like on CI servers when building
several versions from scratch regularly for example.

Archive is not supported over http/https.

Change-Id: I48c3c7de2cd5a1faec33e295fcdafbc7807d0e4d
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
2013-12-10 08:27:07 +00:00
Jesse Hall
672cc499b9 Canonicalize project hooks path before use
If the top-level .repo directory is moved somewhere else (e.g. a
different drive) and replaced with a symlink, _InitHooks() will create
broken symlinks. Resolving symlinks before computing the relative path
for the symlink keeps the path within the repo tree, so the tree can
be moved anywhere.

Change-Id: Ifa5c07869e3477186ddd2c255c6c607f547bc1fe
2013-12-03 09:02:16 -08:00
Conley Owens
cbc0798f67 Fix print of git-remote-persistent-https error
If git-remote-persistent-https fails, we use an iter() and then
subsequently a .read() on stderr.  Python doesn't like this and
gives the following error message:
ValueError: Mixing iteration and read methods would lose data

This change removes the use of iter() to avoid the issue.

Change-Id: I980659b83229e2a559c20dcc7b116f8d2476abd5
2013-11-21 10:38:03 -08:00
Chirayu Desai
0eb35cbe50 Fix some python3 encoding issues
* Add .decode('utf-8') where needed
* Add 'b' to `open` where needed, and remove where unnecessary

Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
2013-11-21 06:03:22 +00:00
David James
8d20116038 repo: Support multiple branches for the same project.
It is often useful to be able to include the same project more than
once, but with different branches and placed in different paths in the
workspace. Add this feature.

This CL adds the concept of an object directory. The object directory
stores objects that can be shared amongst several working trees. For
newly synced repositories, we set up the git repo now to share its
objects with an object repo.

Each worktree for a given repo shares objects, but has an independent
set of references and branches. This ensures that repo only has to
update the objects once; however the references for each worktree are
updated separately. Storing the references separately is needed to
ensure that commits to a branch on one worktree will not change the
HEAD commits of the others.

One nice side effect of sharing objects between different worktrees is
that you can easily cherry-pick changes between the two worktrees
without needing to fetch them.

Bug: Issue 141
Change-Id: I5e2f4e1a7abb56f9d3f310fa6fd0c17019330ecd
2013-10-14 15:34:32 -07:00
Conley Owens
edd0151a26 Accept all UTF-8 committer names
Change-Id: I7d9d49a8bacf2dc332614d26cdfcc905be7a5290
2013-09-27 00:35:35 +00:00
Dave Borowitz
0836a22d38 Wait for git-remote-persistent-https -print_config to exit
Change-Id: I5ab96e7c8575682217d440ddc52ecfdc8c35f179
2013-09-25 17:46:01 -07:00