The manifest usually doesn't sepecify the revision with the full name
(e.g. refs/heads/REV).
However, when checking if the name of the merge branch, full name is
used on the merge branch.
The CL use full name of revision when comparing it with the merge
branch.
Bug: b/370919047
Test: repo upload on a project with `dest-branch` set
Change-Id: Ib6fa2f7246beb5bae0a26a70048a7ac03b6c5a2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438401
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Joe Hsu <joehsu@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
The command _PostRepoFetch will try to self update
during repo sync. That is beneficial but adds
version uncertainty, fail potential and slow downs
in non-interactive scenarios.
Conditionally skip the update if env variable
REPO_SKIP_SELF_UPDATE is defined.
A call to selfupdate works as before, meaning even
with the variable set, it will run the update.
Change-Id: Iab0ef55dc3d3db3cbf1ba1f506c57fbb58a504c3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439967
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
This adds an optional, built-in checker for
guarding against patches hanging on wrong
base revisions, which is useful if a lower layer of
the manifest changes after a patch was done.
When adding a patch with a new revision using
extend-project or remove-project/project:
C---D---E patches in project bla
/
A---B project bla in manifest state 1
<extend-project name="bla" revision="E" base-rev="B">
If project bla gets updated, in a new snap ID
or by a supplier or similar, to a new state:
C---D---E patches in project bla
/
A---B---F---G project bla in manifest state 2
Parsing will fail because revision of bla is now G,
giving the choice to create a new patch branch
from G and updating base-rev, or keeping previous
branch for some reason and only updating base-rev.
Intended for use in a layered manifest with
hashed revisions. Named refs like branches and tags
also work fine when comparing, but will be misleading
if a branch is used as base-rev.
Change-Id: Ic6211550a7d3cc9656057f6a2087c505b40cad2b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436777
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Background:
- Manifest object is large (for projects like Android) in terms of
serialization cost and size (more than 1mb).
- Lots of Project objects usually share only a few manifest objects.
Before this CL, Project objects were passed to workers via function
parameters. Function parameters are pickled separately (in chunk). In
other words, manifests are serialized again and again. The major
serialization overhead of repo sync was
O(manifest_size * projects / chunksize)
This CL uses following tricks to reduce serialization overhead.
- All projects are pickled in one invocation. Because Project objects
share manifests, pickle library remembers which objects are already
seen and avoid the serialization cost.
- Pass the Project objects to workers at worker intialization time.
And pass project index as function parameters instead. The number of
workers is much smaller than the number of projects.
- Worker init state are shared on Linux (fork based). So it requires
zero serialization for Project objects.
On Linux (fork based), the serialization overhead is
O(projects) --- one int per project
On Windows (spawn based), the serialization overhead is
O(manifest_size * min(workers, projects))
Moreover, use chunksize=1 to avoid the chance that some workers are idle
while other workers still have more than one job in their chunk queue.
Using 2.7k projects as the baseline, originally "repo sync" no-op
sync takes 31s for fetch and 25s for checkout on my Linux workstation.
With this CL, it takes 12s for fetch and 1s for checkout.
Bug: b/371638995
Change-Id: Ifa22072ea54eacb4a5c525c050d84de371e87caa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439921
Tested-by: Kuang-che Wu <kcwu@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>
When performing an incremental sync (re-running repo init with an
updated manifest revision) with --fetch-submodules or sync-s=true,
there is an attempt to get a list of all projects (including
submodules) before projects are actually fetched. However, we can
only list submodules of a project if we have already fetched its
revision. Instead of throwing an error when we don't have the
revision, assume there are no submodules for that project. In the
sync cmd, we already update the list of projects to include
submodules after fetching superprojects.
Change-Id: I48bc68c48b5b10117356b18f5375d17f9a89ec05
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439761
Commit-Queue: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Tested-by: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org>
With 551285fa35, the comment about number
of workers no longer stands - dict is shared among multiprocesses and
real time information is available.
Using 2.7k projects as the baseline, using chunk size of 4 takes close
to 5 minutes. A chunk size of 32 takes this down to 40s - a reduction of
rougly 8 times which matches the increase.
R=gavinmak@google.com
Bug: b/371638995
Change-Id: Ida5fd8f7abc44b3b82c02aa0f7f7ae01dff5eb07
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438523
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
If a user is not authenticated, repo continues execution and it will
likely result in more of the same errors being printed. A user is also
likely to SIGTERM the process resulting in more errors.
This change stops repo sync if any of repositories can't be fetched to
Git authentcation using sso helper. We could extend this to all Git
authentication
Change-Id: I9e471e063450c0a51f25a5e7f12a83064dfb170c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438522
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
git fetch operation may prompt user to enter username and password.
This won't be visible to user when repo sync operation since stdout and
stderr are redirected. If that happens, user may think repo is doing
work and likely won't realize it's stuck on user's input.
This patch disables prompt for clone and fetch operations, and repo will
fail fast.
R=gavinmak@google.com
Bug: b/368644181
Change-Id: I2efa88ae66067587a00678eda155d861034b9127
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438001
Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
When a sha1 is provided to '--manifest-branch', the ref which
is expected to contain that sha1 can be provided using the new
'--manifest-upstream-branch' option. This is useful with
'--current-branch' to avoid having to sync all heads and tags,
or with a commit that comes from a non-head/tag ref (like a
Gerrit change ref).
Change-Id: I46a3e255ca69ed9e809039e58b0c163e02af94ef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436717
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Tested-by: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
In the winding maze that constitutes Sync_LocalHalf(), there are paths
in which we don't copy-and-link files. Examples include something like:
cd some/project/
repo start head .
# do some work, make some commit, upload that commit to Gerrit
[[ ... in the meantime, someone addes a <linkfile ...> for
some/project/ in the manifest ... ]]
cd some/project/
git pull --rebase
repo sync
In this case, we never hit a `repo rebase` case, which might have saved
us. Instead, the developer is left confused why some/project/ never had
its <linkfile>s created.
Notably, this opens up one more corner case in which <linkfile ... /> or
<copyfile ... /> could potentially clobber existing work in the
destination directory, but there are existing cases where that's true,
and frankly, those seem like bigger holes than this new one.
Change-Id: I394b0e4529023a8ee319dc25d03d513a19251a4a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/437421
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Brian Norris <briannorris@google.com>
Commit-Queue: Brian Norris <briannorris@google.com>
When using the smart sync option, we try to construct the target that
was "lunched" from the TARGET_PRODUCT and TARGET_BUILD_VARIANT envvars.
However, an android target is now made of three parts,
{TARGET_PRODUCT}-{TARGET_RELEASE}-{TARGET_BUILD_VARIANT}.
I am leaving the option of creating a target if a TARGET_RELEASE is not
specified in case there are other consumers who depend on that option.
BUG=b:358101714
TEST=./run_tests
TEST=smart sync on android repo and manually inspecting
smart_sync_override.xml
Change-Id: I556137e33558783a86a0631f29756910b4a93d92
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436977
Tested-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
In method _parse the value of this variable 'have_fg ' is always
False, Maybe reassign it to True is lost.
I guess the author’s original intention was:
if set some value in gitconfig file(for ex: text = black red ul),
the first is bg color, the second is fg color, and the last one is attr.
Change-Id: I372698fe625db4c1fdaa94ea7f193a80a850ecb9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/425997
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Bright Ma <mmh1989@foxmail.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Current clone bundle fetches from Google storage results HTTP/404
and curl exiting 56. This is basically WAI, so stop emitting
verbose error output whenever that happens. Also add a few more
curl exit statuses based on chromite history, and document them.
Change-Id: I3109f8a8a19109ba9bbd62780b40bbcd4fce9b76
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/432197
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The log message is already formatted before being passed to the colorer.
To avoid the exception "TypeError: not enough arguments for format
string", we should use the `nofmt_colorer` instead.
This bug occurs only when the formatted string still contains '%'
character. The following snippet can reproduce the bug:
```
from repo_logging import RepoLogger
RepoLogger(__name__).error("%s", "100% failed")
```
Change-Id: I4e3977b3d21aec4e0deb95fc1c6dd1e59272d695
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/432017
Tested-by: Shik Chen <shik@google.com>
Commit-Queue: Shik Chen <shik@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Let people specify the exact topic when uploading CLs. The existing
-t option only supports setting the topic to the current local branch.
Add a --topic-branch long option to the existing -t to align it a bit
better with --hashtag & --hashtag-branch.
Change-Id: I010abc4a7f3c685021cae776dd1e597c22b79135
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/431997
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
According to https://git-scm.com/docs/protocol-v2#_ssh_and_file_transport,
when using SSH, the environment variable GIT_PROTOCOL must be set
when establishing the connection to the git server.
Normally git does this by itself. But in repo-tool where the SSH
connection is managed by the repo-tool, it must be passed in
explicitly instead.
Under some circumstances of environment configuration, this
caused all repo sync commands over ssh to always use
git protocol version 1. Even when git was configured to use
version 2.
Using git protocol v2 can significantly improve fetch speeds,
since it uses server side filtering of refs, reducing the
amount of unneccessary objects to send.
Change-Id: I6d4c3b7300a6090d707480b1a638ed03622fa71a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/411362
Tested-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Erik Elmeke <erik@haleytek.corp-partner.google.com>
git push, like most git commands, does not warn or otherwise prompt
users when there are local uncommitted changes. To simplify the
upload logic, and to harmonize repo upload with git push as a more
git-esque flow, stop checking/warning/prompting the user here too.
Change-Id: Iee18132f0faad0881f1a796cb58821328e04b694
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/425337
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Debian 7 Wheezy went EOL in May 2018. We don't need to carry support
for that anymore as there have been 5 major releases since. Ubuntu
Precise went EOL in Apr 2019 (including the extended support phase).
That means we can bump the required git version from 1.7.9 to 1.9.1.
git-1.7.9 was released in 2012 while git-1.9.1 was released in 2014.
So that shouldn't be a problem either. And we've been warning people
using git versions older than 1.9.1 for 3 years now that they need to
upgrade.
Change-Id: Ifbbf72f51010b0a944c2785895d1b605333f9146
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415637
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
These hooks are maintained in other projects. Add a script to automate
their import so people don't send us changes directly, and we can try to
steer them to the correct place.
Change-Id: Iac0bdb3aae84dda43a1600e73107555b513ce82b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/422177
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
On my Windows machine, I mount drive D: to the directory C:\src.
The old implementation returns the incorrect 'C:\\??\\Volume{ad2eb15e-f293-4d48-a448-54757d95a97c}' result, which breaks the repo init command.
With the use of os.path.realpath, it can return 'D:\\' correctly.
Change-Id: Ia5f53989055125cb282d4123cf55d060718aa1ff
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/416580
Reviewed-by: Greg Edelston <gredelston@google.com>
Tested-by: Kaiyi Li <kaiyili@google.com>
Commit-Queue: Kaiyi Li <kaiyili@google.com>
os.path.exists returns false on a broken symlink. This is not what repo
needs when checking if a project is setup properly.
For example, if src/foo/.git can't be resolved, repo tries to create
symlink and that results in FileExistsError.
Use lexists which returns True even if symlink is broken. That will
force path where repo checks where symlink is pointing to and will fix
it to the correct location.
Bug: b/281746795
Change-Id: Id3f7dc3a3cb6499d02ce7335eca992ddc7deb645
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415197
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: George Engelbrecht <engeg@google.com>
Reviewed-by: Greg Edelston <gredelston@google.com>
git-1.9.1 was released in 2014 while git-2.7.4 was released in 2016.
Debian Stretch was released in 2017 and Ubuntu Xenial was released in
2016 which are plenty old at this point. Both of those include at
least git-2.7.4.
We will start warning users of Debian Jessie (released in 2015 & EOL
in 2020) and Ubuntu Trusty (released in 2014 & EOL Apr 2024) that
they will need to upgrade.
Change-Id: I6be3809bc45968fdcb02cad3f7daf75ded1bb5b1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415137
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Debian 6 Squeeze went EOL in Feb 2016. We don't need to carry support
for that anymore as there have been 6 major releases since. That means
we can bump the required git version from 1.7.2 to 1.7.9. Ubuntu Precise
shipped with the latter.
git-1.7.2 was released in 2010 while git-1.7.9 was released in 2012.
So that shouldn't be a problem either. And we've been warning people
using git versions older than 1.9.1 for 3 years now that they need to
upgrade.
Change-Id: I7712f110ea158297b489b8379b112c6700b21a46
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415097
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
If a project can't be removed from checkout due to uncommitted changes
present, error.project is type of Project and not a string (as it is in
some cases). Project is not JSON serializable, resulting in exception
within exception handler:
TypeError: Object of type Project is not JSON serializable
This change casts project to string as a defensive mechanism. It also
passes project name instead of project object.
Change-Id: Ie7b782d73dc3647975755d5a3774d16ea6cd5348
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/413877
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
The current logic to create checkout layers doesn't work in all cases.
For example, let's assume there are three projects: "foo", "foo/bar" and
"foo-bar". Sorting lexicographical order is incorrect as foo-bar would
be placed between foo and foo/bar, breaking layering logic.
Instead, we split filepaths based using path delimiter (always /) and
then use lexicographical sort.
BUG=b:325119758
TEST=./run_tests, manual sync on chromiumos repository
Change-Id: I76924c3cc6ba2bb860d7a3e48406a6bba8f58c10
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/412338
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: George Engelbrecht <engeg@google.com>
If a repo manifest is updated so that project B is placed within a
project A, and if project A had content in new B's location in the old
checkout, then repo sync could break depending on checkout order, since
B can't be checked out before A.
This change introduces checkout levels which enforces right sequence of
checkouts while still allowing for parallel checkout. In an example
above, A will always be checked out first before B.
BUG=b:325119758
TEST=./run_tests, manual sync on ChromeOS repository
Change-Id: Ib3b5e4d2639ca56620a1e4c6bf76d7b1ab805250
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/410421
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Greg Edelston <gredelston@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
git-trace2 events contain additional information what git is doing under
the hood, and repo doesn't have visibility into.
Instead of relying on timestamp information to match REPO_TRACE with
git-trace2 events, add SID information into REPO_TRACE.
Change-Id: I37672a3face81858072c7a3ce34ca3379199dab5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/410280
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Some (most?) Linux repos don't have /usr/bin/python, unless
python-is-python3 is installed. While package owners can adjust shebang,
we have seen an increase in number of bugs filed as extra steps are
required.
Per PEP-0394, python3 is acceptable and should be available if python3
is supported. We no longer support python2, and repo no longer works
with python2, so this change makes that explicit.
Bug: 40014585
Change-Id: I9aed90fd470ef601bd33bd596af3df69da69ef5d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/407497
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Максим Паймушкин <maxim.paymushkin@gmail.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Prior to this change RepoChangedException would be caught and re-rasied
as a different exception. This would prevent RepoChangedException
handler from running in main.py
Bug: b/323232806
Change-Id: I9055ff95d439d6ff225206c5bf1755cc718bcfcc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/407144
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
If a project contains files not owned by the current user, remove will
fail. In order to ensure repo sync continues to work, rename the
affected project instead, and let user know about it.
Bug: 321273512
Change-Id: I0779d61fc67042308a0226adea7d98167252a5d3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/404372
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Symbolic references need to be checked each time sync is called, not
only for newly created repositories. For example, it is possible to
change a project name to the already existing name, and that will result
in a broken git setup without this patch: refs/ will still point to the
old repository, whereas all objects will point to the new repository.
Bug: 40013418
Change-Id: I596d29d182986804989f0562fb45090224549b0f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/395798
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Before the change, repo normalizes the urls
with a following format only:
git@github.com:foo/bar
It doesn't cover the following case:
<remote name="org" fetch="git@github.com:org/" />
<project name="somerepo" remote="org" />
Results to:
error: Cannot fetch somerepo
from ssh://git@github.com/org/git@github.com:org/somerepo
Current change fixes it by normalizing this format:
git@github.com:foo
Test: ./run_tests tests/test_manifest_xml.py
Change-Id: I1ad0f5df0d52c0b7229ba4c9a4db4eecb5c1a003
Signed-off-by: Vitalii Dmitriev <vitalii.dmitriev@unikie.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/398337
Commit-Queue: Vitalii Dmitriev <dmit.vitalii@gmail.com>
Tested-by: Vitalii Dmitriev <dmit.vitalii@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
repo blocks indefinitely until trace collector receives trace events,
which is not desired. This change adds a fixed timeout to connect and
send operations. It is possible that some events will be lost. repo logs
any failed trace operation.
Bug: b/316227772
Change-Id: I017636421b8e22ae3fcbab9e4eb2bee1d4fbbff4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/398717
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Jason Chang <jasonnc@google.com>
Most times a repo sync after some time (week+) results in a bunch of
messages, which are not very useful for average user:
- discarding 1 commits
- Deleting obsolete checkout.
Bug: N/A
Test: repo sync
Change-Id: I881eab61f9f261e98f3656c09e73ddd159ce288c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/397038
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Tomasz Wasilczyk <twasilczyk@google.com>
In the case of a project being removed from the manifest, and in the
path in which the project used to exist, and symlink is place to another
project repo will start to warn about partial syncs when a partial sync
did not occur.
Repro steps:
1) Create a manifest with two projects. Project a -> a/ and project b -> b/
2) Run `repo sync`
3) Remove project b from the manifest.
4) Use `link` in the manifest to link all of Project a to b/
Bug: 314161804
Change-Id: I4a4ac4f70a7038bc7e0c4e0e51ae9fc942411a34
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/395640
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Matt Schulte <matsch@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Python 3.11 introduces PYTHONSAFEPATH and the -P flag which, if enabled,
does not prepend the script's directory to sys.path by default.
This breaks repo because main.py expects its own directory to be part of
Python's import path.
This causes problems with tools that add PYTHONSAFEPATH to python
programs, most notably Bazel.
We will simply prepend main.py's path to PYTHONPATH instead.
Bug: 307767740
Change-Id: I94f3fda50213e450df0d1e2df6a0b8b597416973
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/391236
Tested-by: Duy Truong <duytruong@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This updates RepoLogger.log_aggregated_errors to print out the error
message the RepoExitError when there is not a list of aggregated
errors.
Previously it would log out:
=======================================================================
Repo command failed: ManifestParseError
This told us what class of error occurred but missed the helpful error
message that developers put in the error. After this change it will now
print out the error message:
=======================================================================
Repo command failed: ManifestParseError
error parsing manifest /path/to/manifest.xml: no element found:
line 197, column 0
Change-Id: I4805540fddb5fa9171dbc8912becfa7fdfb1ba67
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/392614
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Joshua Bartel <josh.bartel@garmin.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Avoids treating the operation as if it were acting on a bare repository, thereby triggering failures when the Git client is configured with `safe.bareRepository=explicit`. Repo doesn't actually use a bare repository, but pointing at the gitdir acts as if it had.
Bug: 307559774
Change-Id: I2c142275b2726a59526729c0b2c54faf728f125d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/391554
Commit-Queue: Jason R. Coombs <jaraco@google.com>
Tested-by: Jason R. Coombs <jaraco@google.com>
Tested-by: Emily Shaffer <emilyshaffer@google.com>
Reviewed-by: Emily Shaffer <emilyshaffer@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
When a new shared project is added to manifest, there's a short window
where objects can be deleted that are used by other projects.
To close that window, set preciousObjects during git init. For
non-shared projects, repo should correct the state in the same execution
instance.
Bug: 288102993
Change-Id: I366f524535ac58c820d51a88599ae2108df9ab48
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/390234
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
When parsing the manifest XML, the code looks for a top
level DOM node named "manifest". However, it doesn't check
that it's an element type node so if there is also an XML
document type declaration node present (which has the same
name as the root element) then it selects the wrong node
and hence you end up with no projects defined at all.
Change-Id: I8d101caffbbc2a06e56136ff21302e3f09cfc96b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/390357
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Chris Allen <chris.allen@arm.com>
Commit-Queue: Chris Allen <chris.allen@arm.com>
When running the tests in my environment, tests that derived from `test_wrapper.GitCheckoutTestCase` would fail on commit or tag due to incomplete or incorrect gpg config. Ideally, the tests should not be dependent on the user's git config. This change ensures $HOME (or Windows equivalent) is replaced for the session.
Bug: 302797407
Change-Id: Ib42b712dd7b6602fee6e18329a8c6d52fb9458b9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/388235
Tested-by: Jason R. Coombs <jaraco@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Jason R. Coombs <jaraco@google.com>
This doc was written back in 2019 when we were planning on the Python 3
migration. It isn't relevant anymore, and people are reading it thinking
we still support Python 2. Rewrite it to match current requirements and
to make it clear there is no support for older versions.
Bug: 302871152
Change-Id: I2acf3aee1816a03ee0a70774db8bf4a23713a03f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/389455
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
tox uses virtualenv under its hood for managing virtual environments.
Virtualenv 20.22.0 dropped support for Python <= 3.6.
Since we want to test against Python 3.6 we need to make sure we use
a version of virtualenv earlier than 20.22.0.
This error was not stopping any tests from passing but was printed
multiple times to stderr when executing the py36 target:
Error processing line 1 of [...]/.tox/py36/[...]/_virtualenv.pth:
Traceback (most recent call last):
File "/usr/lib/python3.6/site.py", line 168, in addpackage
exec(line)
File "<string>", line 1, in <module>
File "[...]/.tox/py36/[...]/_virtualenv.py", line 3
from __future__ import annotations
^
SyntaxError: future feature annotations is not defined
Source: https://tox.wiki/en/latest/faq.html#testing-end-of-life-python-versions
Change-Id: I27bd8200987ecf745108ee8c7561a365f542102a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/387694
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Commit-Queue: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
DeprecationWarning: datetime.datetime.utcnow() is deprecated and
scheduled for removal in a future version. Use timezone-aware objects to
represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
Change-Id: Ia2c46fb87c544d98cc2dd68a829f67d4770b479c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/386615
Tested-by: Łukasz Patron <priv.luk@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Łukasz Patron <priv.luk@gmail.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
tests/test_subcmds_sync.py::LocalSyncState::test_prune_removed_projects
was failing in Python 3.6 and 3.7 due to topdir not being set with the
following error message:
TypeError: expected str, bytes or os.PathLike object, not MagicMock
topdir is accessed from within PruneRemovedProjects().
Test: tox with Python 3.6 to 3.11
Change-Id: I7ba5144df0a0126c01776384e2178136c3510091
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/382816
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Daniel Kutik <daniel.kutik@lavawerk.com>
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
tests/test_git_superproject.py::SuperprojectTestCase::test_Fetch was
failing in Python 3.6 and 3.7 due to attribute args only being
introduced in Python 3.8. Falling back on old way of accessing
the arguments.
Test: tox with Python 3.6 to 3.11
Change-Id: Iae1934a7bce8cbd6b4519e4dbc92d94e21b43435
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/382818
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Daniel Kutik <daniel.kutik@lavawerk.com>
When comparing 2 manifests, most of the time is
spent getting the relevant commit id as it relies
on _allrefs which ends up loading all git references.
However, the value from `revisionIs` (when it is valid)
could be used directly leading to a huge performance improvement
(from 180+ seconds to less than 0.01 sec which is more
than 25000 times faster for manifests with 700+ projects).
Bug: 295282548
Change-Id: I5881aa4b2326cc17bbb4ee91d23293111f76ad7e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/385834
Tested-by: Sylvain Desodt <sylvain.desodt@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Sylvain Desodt <sylvain.desodt@gmail.com>
The logic in projectsDiff performs various operations which
suggest that a set is more appropriate than a list:
- membership lookup ("in")
- removal
Also, sorting can be performed on the the remaining elements at the
end (which will usually involve a much smaller number of elements).
(The performance gain is invisible in comparison to the time being
spent performing git operations).
Cosmetic chance:
- the definition of 'fromProj' is moved to be used in more places
- the values in diff["added"] are added with a single call to extend
Change-Id: I5ed22ba73b50650ca2d3a49a1ae81f02be3b3055
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/383434
Tested-by: Sylvain Desodt <sylvain.desodt@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Sylvain Desodt <sylvain.desodt@gmail.com>
If a KeyboardInterrupt is encountered before an error is aggregated then
the context surrounding the interrupt is lost. This change aggregates
errors as soon as possible for the sync command
Bug: b/293344017
Change-Id: Iac14f9d59723cc9dedbb960f14fdc1fa5b348ea3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/384974
Tested-by: Jason Chang <jasonnc@google.com>
Commit-Queue: Jason Chang <jasonnc@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Black will only check .py files when given a dir and --check, so list
our few standalone programs explicitly. This causes the repo launcher
to be reformatted since it was missed in the previous mass reformat.
Bug: b/267675342
Change-Id: Ic90a7f5d84fc02e9fccb05945310fd067e2ed764
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/385034
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Per discussion in go/repo-error-update updated aggregated and exit
errors for sync command.
Aggregated errors are errors that result in eventual command failure.
Exit errors are errors that result in immediate command failure.
Also updated main.py to log aggregated and exit errors to git sessions
log
Bug: b/293344017
Change-Id: I77a21f14da32fe2e68c16841feb22de72e86a251
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/379614
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Commit-Queue: Jason Chang <jasonnc@google.com>
In order to better analyze and track repo errors, repo command failures
need to be tied to specific errors in repo source code.
Additionally a new GitCommandError was added to differentiate between
general git related errors to failed git commands. Git commands that opt
into verification will raise a GitCommandError if the command failed.
The first step in this process is a general error refactoring
Bug: b/293344017
Change-Id: I46944b1825ce892757c8dd3f7e2fab7e460760c0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/380994
Commit-Queue: Jason Chang <jasonnc@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Fixed a couple of bugs in ExitEvent logging:
- log exitcode 130 on KeyboardInterrupt
- log exitcode 1 on unhandled Exception
- log errorevent with specific reason for exit
Before this CL an exitcode of 0 would be logged, and it would be
difficult to determine the cause of non-zero exit codes
Bug: b/287105597
Change-Id: I2d34f180581f9fbd77a1c78c966ebed065223af6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/377834
Tested-by: Jason Chang <jasonnc@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
A something.xml that gets included by two different
files, that both remove and add same shared project
to two different locations, would not work
prior to this change.
Reason is that remove killed all name keys, even
though reuse of same repo in different locations
is allowed.
Solve by adding optional attrib path to
<remove-project name="foo" path="only_this_path" />
and tweak remove-project.
Behaves as before without path, and deletes
more selectively when remove path is supplied.
As secondary feature, a project can now also be removed
by only using path, assuming a matching project name
can be found.
Change-Id: I502d9f949f5d858ddc1503846b170473f76dc8e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/375694
Tested-by: Fredrik de Groot <fredrik.de.groot@aptiv.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
An investigation go/git-repo-shallow shows a number of problems
when doing a shallow git fetch/clone. This change introduces an
environment variable REPO_ALLOW_SHALLOW. When this environment variable
is set to 1 during a repo init or repo sync all shallow git fetch
commands are replaced with partial fetch commands. Any shallow
repository needing update is unshallowed. This behavior continues until
a subsequent repo sync command is run with REPO_ALLOW_SHALLOW set to 1.
Bug: b/274340522
Change-Id: I1c3188270629359e52449788897d9d4988ebf280
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374754
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Jason Chang <jasonnc@google.com>
If the original fetch attempt did not want tags, we should continue to
honor that when doing a retry fetch with depth set to None. This seems
to match the intent of the retry based on the inline comment and results
in a significant performance improvement when the original fetch-by-sha1
fails due to the server not allowing requests for unadvertised objects.
Change-Id: Ia26bb31ea9aecc4ba2d3e87fc0c5412472cd98c4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/374918
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Tested-by: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Attribute groups can now be added to manifest include, thus
all projects in an included manifest file can easily modify
default branch without modifying all projects in that manifest file.
For example,
the main manifest.xml has an include node contain revision attribute,
```
<include name="include.xml" revision="r1" />
```
and the include.xml has some projects,
```
<project path="project1_path" name="project1_name" revision="r2" />
<project path="project2_path" name="project2_name" />
```
With this change, the final manifest will have revision="r1" for project2.
```
<project name="project1_name" path="project1_path" revision="r2" />
<project name="project2_name" path="project2_path" revision="r1" />
```
Test: added unit tests to cover the inheritance
Change-Id: I4b8547a7198610ec3a3c6aeb2136e0c0f3557df0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/369714
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Shuchuan Zeng <zengshuchuan@allwinnertech.com>
Tested-by: Shuchuan Zeng <zengshuchuan@allwinnertech.com>
By default, datetime.isoformat() uses different format depending on
microseconds - if is equal to 0, microseconds are omitted, but otherwise
not.
Setting timespec = 'microseconds' ensures the format is the same
regardless of current time.
Change-Id: Icb1be31eb681247c7e46923cdeabb8f5469c20f0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/371694
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Remove logic to handle importing the right version of pytest.
'./run_tests' still works but this allows presubmit builders to test
using 'vpython3 ./run_tests'.
Google-Bug-Id: b/266734831
Change-Id: I6a543c1f4b5b4449e723095b4a70e5228b1ccd34
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/356717
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If interrupt signal is sent to repo process while sync is running, repo
prints stack trace for each concurrent job that is currently running
with no useful information.
Instead, this change captures KeyboardInterrupt in each process and
prints one line about current project that is being processed.
Change-Id: Ieca760ed862341939396b8186ae04128d769cd56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357135
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
repo sync progress bar is misleading. Many bug reports mentioned that
repo is stuck at the repo that is currently displayed in the progress
bar. Repo sync actually shows what repository is the last processed.
This change makes that obvious.
Change-Id: I962bf0bc65af7ac0ed98db86e9144f07d9e1f96f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357134
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
This can potentially show up when sync'ing projects with submodules
that are not declared in the manifest as well as the internal
'.repo/repo' project, which is likely not desirable from a user
standpoint.
Change-Id: I93d7fcd6e3fd1818357ea4537882a864dea9942c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/355920
Reviewed-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
If syncing in _FetchOne fails with GitError, sync_result does not get
set. There's already a separate local variable for success; do the same
for remote_fetched instead of referring to the conditionally defined
named tuple.
This bug is originally caused by a combination of ad8aa697 "sync: only
print error.GitError, don't raise that exception." and 1eddca84 "sync:
use namedtuples for internal return values".
Change-Id: I0f9dbafb97f8268044e5a56a6f92cf29bc23ca6a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354176
Tested-by: Karsten Tausche <karsten@fairphone.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
Do not use a progress bar when not doing GC, and restrict activity in
that case to only repairing preciousObject state.
This also includes additional cleanup based on review comments from
previous changes.
Change-Id: I48581c9d25da358bc7ae15f40e98d55bec142331
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353514
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
REPO_AUTO_GC was introduced as a way for users to restore the previous
default behavior, since the default changed at the same time as the
option was added. As such, it should be marked as deprecated, and
removed entirely in a future release.
Change-Id: Ib73d98fbea693e7057cc4587928c225a9e4beab2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353734
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Fixed "DeprecationWarning: the load_module() method is deprecated and
slated for removal in Python 3.12; use exec_module() instead." in
wrapper.py. Additionally removed Python 2 code (imp.load_source()).
Test: tox
Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: Ib7cc19b1c545f6449e034c4b01b582cf6cf4b581
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353237
Reviewed-by: Mike Frysinger <vapier@google.com>
On Linux/macOS we allow \n in the end of the line.
On Windows we allow both \r\n and \n. Here we also allow Unix line
seperators as tests might be excuted in for example git-shell.
Change-Id: I3975b563cf95407da92e5479980e670eb748b30e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353181
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Allows us to pass on arguments to run_tests and pytest after -- when
executing tox.
E.g.: To run all tests verbose in a test class:
tox -- -v tests/test_project.py::ReviewableBranchTests
Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: Ibd78856c6d4053c769f3d0b6130ebc8145275f78
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353176
Reviewed-by: Mike Frysinger <vapier@google.com>
Use pytest-timeout to make sure tests don't get stuck for more than
5 minutes. In future individual tests can exceed this timeout by
being decorated with @pytest.mark.timeout(600).
Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: I8f5b61a20230c22a86fd5636297c78f41369449a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353124
Reviewed-by: Mike Frysinger <vapier@google.com>
In Python 3.10 onwards we see a DeprecationWarning:
currentThread() is deprecated, use current_thread() instead.
Same goes for getName(), replaced by name attribute.
Test: tox (python 3.6 - 3.10)
Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: I80ec819752a5276cff3b2dadba0ec10cc92d09a4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353018
Reviewed-by: Mike Frysinger <vapier@google.com>
Since the same project can be checked out in multiple paths, we need to
track the "to be uploaded" projects by path, rather than project name.
Bug: crbug.com/gerrit/16260
Test: manual
Change-Id: Ic3dc81bb8acb34886baa6299e90a49c7ba372957
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/351054
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
This will allow the next repo release to be a fast-forward on stable.
* origin/stable:
v2.29.7: Revert back to v2.29.5
Change-Id: I3e52f76766807c58f56d3e246fa142ed55ede59b
This change reverts stable to v2.29.5, to fix clients that received
v2.29.6, and keep future updates simpler.
Change-Id: I2f5c52c466b7321665c9699ccdbf98f928483fee
If this is a project that is not using object sharing (there is only one
copy of the remote project) then clear preciousObjects.
To override this for a project, run:
git config --replace-all repo.preservePreciousObjects true
Change-Id: If3ea061c631c5ecd44ead84f68576012e2c7405c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350235
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Before Python 3.8, xml.dom.minidom sorted the attributes of an element
when writing it to a file, while later versions output the attributes
in the order they were created. Avoid these differences by sorting the
attributes for each element before comparing the generated manifests
with the expected ones.
This corresponds to commit 5d58c18, but for new tests introduced since
it was integrated.
Change-Id: I5c360656a0968e6e8d57eb068c8e87da7dfa61c1
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349917
Reviewed-by: LaMont Jones <lamontjones@google.com>
People rarely care about the history of the manifest repo. Add a
parameter to specify depth for the manifest.
For now, make the default behavior the same as the current behavior. At
a future date, the default will be changed to 1. People who need the
full history should begin passing --manifest-depth=0 to preserve the
behavior when the default changes.
We can't reuse the existing --depth option because that applies to
all projects we clone, not just the manifest repo.
Bug: https://crbug.com/gerrit/16193, https://crbug.com/gerrit/16358
Change-Id: I9130fed3eaed656435c778a85cfe9d04e3a4a6a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349814
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Use pre-3.9 syntax for NamedTuple, so that users do not need to have
python 3.9 or later installed.
Bug: b/255632143, crbug.com/gerrit/16355
Test: manually verified with python 3.8
Change-Id: I488d2d5267ed98d5c55c233cc789e629f1911c9d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349395
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
If a file that is copied using a <copyfile> tag is modified and not
committed or if it is committed to a detached head, then running `repo
sync` would update the target file as expected. However, if the
modified file is committed to a local branch, then running `repo sync'
would not update the target file as expected.
Change-Id: Ic98e37d1c2e51fd1bf15abf149c7d06190cfd6d2
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344475
Reviewed-by: Mike Frysinger <vapier@google.com>
Replace tuple returns with namedtuples, to simplify adding new fields.
Extend the Sync_NetworkHalf return value to:
- success: True if successful (the former return value)
- remote_fetched: True if we called `git fetch`
Change-Id: If63c24c2f849523f77fa19c05bbf23a5e9a20ba9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344534
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
By default there are 4 categories in the diffmanifests
api puts the diffs in to - added, removed, changed and unreachable
Example of command - repo diffmanifests 1.xml 2.xml
added - list down the projects present in second manifest but not in
first
removed - list down the projects present in first but not in
second
changed - list down the changes and the differences for each project
unreachable - when it encounters revision value in a project is incorrect
But, when there are projects present in both manifests and could not
find in local workspace where we have cloned the repo(because of
different/subset manifest xml) - this will create unhandled exception
Now we have added a 5th category called 'missing' - where in such
cases it will handle the scenario and print the log for user
Example:
added projects :
project_2 at revision e6c8a59832c05dc4b6a68cee6bc0feb832181725
removed projects :
project_1 at revision e6c8a59832c05dc4b6a68cee6bc0feb832181725
changed projects :
project_3 changed from 3bb890e1286f04e84d505e5db48e0ada89892331 to e434b3736f11537c67590fefadfe4495895e9785
missing projects :
project_4
Change-Id: I244e8389bff7e95664c29d3dcb61e22308e3a573
Signed-off-by: Shashank Devaraj <shashankkarthik@gmail.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344774
Reviewed-by: Mike Frysinger <vapier@google.com>
This is ENABLED BY DEFAULT due to data corruption potential. To disable
it, set REPO_BACKUP_OBJECTS=0 in the environment.
While the workspace will grow over time, this provides a recovery path
for an issue where objects are erroneously deleted from the workspace,
resulting in lost work. Once the root cause is determined, we will be
able to stop saving backups again.
Backups are kept in .git/objects/.repo/pack.bak
Bug: https://crbug.com/gerrit/16247
Change-Id: Ib8b5c9b4bf0dfa9e29606e0f5c881d65996b2a40
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/345114
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
After reworking this function to use subprocess for output capturing
in commit c87c1863b1 ("git_command:
switch process capturing over to subprocess"), passing input via
stdin write no longer works. We have to pass it via communicate(),
and we have to pass it a string instead of bytes (since we always
use encoding='utf-8' now). This is fine since the only user of the
input= setting today is already passing in a string.
Bug: https://crbug.com/gerrit/16151
Change-Id: Ic58db1e568b8f8aa840a6d62c5a157c14aa6d9bc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343515
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
If initializing the manifest fails for any reason, don't leave it in
a half complete state. This can cause problems if/when the user tries
to reinit because different codepaths will be taken. For example, if
we initialize manifests.git and don't finish probing the remote to see
what default branch it uses, we end up always using "master" even if
that isn't what the remote uses.
To avoid all of this, use .tmp dirs when initializing, and rename to
the final path only after we complete all the right steps.
We should roll this out to all projects we clone, but start with the
manifest project for now.
Bug: https://crbug.com/gerrit/13526
Bug: https://crbug.com/gerrit/15805
Change-Id: I0214338de69ee11e090285c6b0b211052804af06
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343539
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
People rarely care about the history of the manifest repo. Change
the default to 1 to speed up initial setup. If people really want
the full history, they can pass --manifest-depth=0.
We can't reuse the existing --depth option because that applies to
all projects we clone, not just the manifest repo.
Bug: https://crbug.com/gerrit/16193
Change-Id: Ideab1712e9ffc743b9bae97903e074d7d429b134
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343435
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
People rarely care about having all manifest branches locally. Change
the default to only pull down the selected branch. If people want other
branches, the -b option will fetch it automatically, or people can use
--no-current-branch.
This only applies to the manifest project syncing, not the rest of the
projects that are in the checkout.
Bug: https://crbug.com/gerrit/16194
Change-Id: Ia9e7e2f23b8028d82772db72dcc7d6c32260be79
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343434
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
The `--wip` allow to bulk push changed as work-in-progress. This CL
intend to allow the opposite opperation by removing the wip mark on the
CL and set it to be ready to review
Change-Id: If0743c5b14829f77be2def5a8547060d06a5648c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/342214
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: William Escande <wescande@google.com>
For --jobs-network, the logic is now:
* If the user specifies --jobs-network, use that.
* Else, if the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to 1.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.
For --jobs-checkout, the logic is now:
* If the user specifies --jobs-checkout, use that.
* Else, if the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to DEFAULT_LOCAL_JOBS which is based on user's ncpus.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.
For garbage collecting, the logic is now:
* If the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to the user's ncpus.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.
Having to factor in the manifest settings makes this more complicated
which is why we delay processing of defaults until after we've synced
the manifest projects.
Bug: http://b/239712300
Change-Id: Id27cda63c76c156f1d63f6a20cb2c4ceeb3d547c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/341394
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
When we create superproject_override.xml, do not include projects that
are present from local_manifests/*. Such projects are fully under the
control of the local_manifests/ file.
Bug: b/238934278
Test: manual, ./run_tests
Change-Id: I40382ceb82d9cf7b8dc7b5f2abed3f6d4d80017e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340877
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Sam Saccone 🐐 <samccone@google.com>
Display the project path relative to the outermost manifest by default,
and relative to the sub manifest only when --this-manifest-only is
specified.
For project-related diagnostic messages, use the outermost manifest for
messages.
Change-Id: I4537d7dd412a2c182e77d6720e95c1b0ef70eb0e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340754
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
This option will suppress the
Uncommitted changes in ... (did you forget to amend?)
prompt when there are untracked (unknown) files in the working copy.
The prompt is still shown if tracked files are modified.
Change-Id: Ia3fcc82989b7fad09b69214eda31e2d0dfc14600
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340456
Tested-by: Martin Geisler <mgeisler@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
With this change, partial syncs (sync with a project list) are again
supported.
If the updated manifest includes new sub manifests, download them
inheriting options from the parent manifestProject.
Change-Id: Id952f85df2e26d34e38b251973be26434443ff56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334819
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
For now, this is opt-in via environment variables:
- export REPO_USE_ALTERNATES=1
The shared project logic that shares the internal .git/objects/ dir
directly between multiple projects via the project-objects/ tree has
a lot of KI with random corruption. It all boils down to projects
sharing objects/ but not refs/. Git operations that use refs to see
what objects are reachable and discard the rest can easily discard
objects that are used by other projects.
Consider this project layout:
<show fs layout>
There are unique refs in each of these trees that are not visible in
the others. This means it's not safe to run basic operations like
git prune or git gc.
Since we can't share refs (each project needs to have unique refs
like HEAD in order to function), let's change how we share objects.
The old way involved symlinking .git/objects/ to the project-objects
tree. The new way shares objects using git's info/alternates.
This means project-objects/ will only contain objects that exist in
the remote project. Local per-project objects (like when creating
branches and making changes) will never be shared. When running a
prune or gc operation in the per-project state, it will only ever
repack or discard those per-project objects. The common shared
objects would only be cleaned up when running a common operation
(i.e. by repo itself).
One downside to this for users is if they try blending unrelated
upstream projects. For example, in CrOS we have multiple kernel
projects (for diff versions) checked out. If a dev fetched the
upstream Linus tree into one of them, the objects & tags would
not be shared with the others, so they would have to fetch the
upstream state for each project. Annoying, but better than the
current corruption situation we're in now.
Also if the dev runs a manual `git fetch` in the per-project to
sync it up to newer state than the last `repo sync` they ran,
the objects would get duplicated. However, git operations later
on should eventually dedupe this.
Bug: https://crbug.com/gerrit/15553
Change-Id: I313a9b8962f9d439ef98ac0ed37ecfb9e0b3864e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328101
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
If init was run with --use-superproject, init failed.
If init was run without --{no,}use-superproject option then manifests
with <superproject/> elements were mishandled.
Bug: b/233226285
Test: manual
Change-Id: I737e71c89d2d7c324114f58bf2dc82b40e5beba7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337534
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
When a manifest file is overridden, remember that and keep using the
override for the remainder of the process. If we need to revert it,
make the override name evaluate False.
Change-Id: I1eee05fec6988c1ee4a3c751c4b540d5b5d11797
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335136
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
We need to iterate over multiple manifests, and generally use the
outer_client.manifest for multi-manifest support. This refactors the
use of self.manifest into a chosen manifest.
Change-Id: I992f21d610c929675e99555ece9c38df4b635839
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334699
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Make git_superproject independent of the command line by passing
the specific value instead of requiring the caller to have an
optparse.Values object to pass in.
Flag --use-superproject and --archive as incompatible in subcmds/init.py
Change-Id: Ied7c874b312e151038df903c8af4328f070f387c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335135
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
- REPO_PATH is relative to the root of the client. REPO_OUTERPATH is not
needed.
- REPO_INNERPATH is relative to the sub manifest root.
- REPO_OUTERPATH is the path for the sub manifest root relative to the
root of the client.
Change-Id: I031692891cfef2634d1358584d27a6a4df735c20
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334899
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Always create Superproject when there is a <superproject> tag, and have
it hold the XML content, similar to how other manifest elements are
handled.
This also adds SetQuiet and SetPrintMessages to Superproject
consistent with manifest.SetUseLocalManifests.
Change-Id: I522bf3da542006575799f0640c67f7052704f266
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334641
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
The documentation states that a `sync-c` attribute in the manifest file
can set a default for whether only the current branch should be fetched
or all branches. This seems to have been broken for some time.
Commit 7356114 introduced the `--no-current-branch` CLI option and
relied on getting `None` via `optparse` if neither `--current-branch`
nor `--no-current-branch` was set to distinguish it from a boolean
value. If `None` was received, it would read the value from the manifest
option `sync-c`. The parsing went through the utility function
`_GetCurrentBranchOnly` which returned `True` if `--current-branch` had
been given on the command-line, or fell back on the "superproject"
setting, which would either return `True` or `None`. This would
incorrectly make `repo` fall back to the manifest setting even if the
user had given `--no-current-branch` if no superproject was requested --
the manifest became "too powerful":
Command-line Using superproject → `current_branch_only`
------------ ------------------ -----------------------
No From manifest
Yes True
--current-branch No True
--current-branch Yes True
--no-current-branch No From manifest ← wrong
--no-current-branch Yes True
In commit 0cb6e92 the superproject configuration value reading changed
from something that could return `None` to something that always
returned a boolean. If it returned `False`, this would then incorrectly
make `repo` ignore the manifest option even if neither
`--current-branch` nor `--no-current-branch` had been given. The
manifest default became useless:
Command-line Using superproject → `current_branch_only`
------------ ------------------ -----------------------
No False ← wrong
Yes True
--current-branch No True
--current-branch Yes True
--no-current-branch No False
--no-current-branch Yes True
By swapping the order in which the command-line option target and the
superproject setting is evaluated, things should work as documented:
Command-line Using superproject → `current_branch_only`
------------ ------------------ -----------------------
No From manifest
Yes True
--current-branch No True
--current-branch Yes True
--no-current-branch No False
--no-current-branch Yes True
Change-Id: I933c232d2fbecc6b9bdc364ebac181798bce9175
Tested-by: Daniel Andersson <daniel.r.andersson@volvocars.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334270
Reviewed-by: Mike Frysinger <vapier@google.com>
Correctly pass the full path of the manifest file for the submanifest.
The manifest-name in the <submanifest/> element was being passed in
as given, which caused it to not be found since the current directory
never set. (b/226333721: fails when manifest-name is given.)
Also verify that the manifest_file passed to XmlManifest() is an
absolute path.
Bug: https://b.corp.google.com/issues/226333721
Change-Id: I23461078233e34562bc2eafeb732cfe8bd38ddc1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/333861
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Git can write trace2 events to a Unix domain socket [1]. This can be
specified via Git's `trace2.eventTarget` config option, which we read to
determine where to log our own trace2 events. Currently, if the Git
config specifies a socket as the trace2 target, we fail to log any
traces.
Fix this by adding support for writing to a Unix domain socket,
following the same specification that Git supports.
[1]: https://git-scm.com/docs/api-trace2#_enabling_a_target
Change-Id: I928bc22ba04fba603a9132eb055141845fa48ab2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/332339
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Josh Steadmon <steadmon@google.com>
It was reported that git-lfs did not work with git-repo. Specifically,
`git read-tree -u` run by `repo sync` would fail git-lfs's smudge
filter. See https://github.com/github/git-lfs/issues/1422.
In fact, by the time `git read-tree -u` is run, the repository is not
bare. It is just that, the working directory is not the same as the
.git directory. git-lfs's filter should work. No one seems to have
delved into that issue.
Today, with newer versions of git-repo and git-lfs, that issue will
not reproduce. Tested with
- git 2.33, git-lfs 2.13 on macOS
- git 2.17, git-lfs 2.3 on ubuntu
So, it seems fine to add an option --enable-git-lfs-filter, default to
false, and stat that it may not work with older versions of git and
git-lfs in the help doc.
Bug: https://crbug.com/gerrit/14516
Change-Id: I8d21854eeeea541e072f63d6b10ad1253b1a9826
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328359
Tested-by: XD Trol <milestonejxd@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If the user passes a bad --repo-rev setting in a new checkout, add a
tip to the error message that their option is probably bad instead of
just saying "unable to resolve".
If the user has already initialized a checkout, we'd display a raw
traceback which would confuse them. Swallow that and also include
the --repo-rev tip.
Bug: https://crbug.com/gerrit/15610
Change-Id: I5d72513c7b37bf9bb5d19862fcdfaf0d1f44e886
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328820
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the branch is current, or present in less than half of the projects,
list which projects it is *in*.
Otherwise, correctly detect which projects (by relpath) it is not in.
Previously, the "not in" path would incorrectly list all projects.
Change-Id: Ia153856f577035a51f538b7bf5d3135b70c69d52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328199
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
In order to stop sharing objects/ directly between shared projects,
we have to fetch the remote objects into project-objects/ manually.
So instead of running git operations in the individual project dirs
and relying on .git/objects being symlinked to project-objects/,
tell git to store any objects it fetches in project-objects/.
We do this by leveraging the GIT_OBJECT_DIRECTORY override. This
has been in git forever, or at least since v1.7.2 which is what we
already hard require. This tells git to save new objects to the
specified path no matter where it's being run otherwise.
We still otherwise run git in the project-specific dir so that it
can find the right set of refs that it wants to compare against,
including local refs. For that reason, we also have to leverage
GIT_ALTERNATE_OBJECT_DIRECTORIES to tell git where to find objects
that are not in the upstream remote. This way git doesn't blow up
when it can't find objects only associated with local commits.
As it stands right now, the practical result is the same: since we
symlink the project objects/ dir to the project-objects/ tree, the
default objects dir, the one we set $GIT_OBJECT_DIRECTORY to, and
the one we set $GIT_ALTERNATE_OBJECT_DIRECTORIES to are actually
all the same. So this commit by itself should be safe. But in a
follow up commit, we can replace the symlink with a separate dir
and git will keep working.
Bug: https://crbug.com/gerrit/15553
Change-Id: Ie4e654aec3e1ee307eee925a54908a2db6a5869f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328100
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When using --reference, the path is written to objects/info/alternates.
The path is accessed inconsistently -- sometimes through projects/ (via
self.gitdir) and sometimes through project-objects/ (via self.objdir).
This works because projects/.../objects is a symlink to the objects dir
under project-objects/. Change all accesses to go through self.objdir.
This will allow us to stop symlinking projects/.../objects without the
reference dir logic breaking. The projects/ path is going to use its
alternates file for its own needs.
Bug: https://crbug.com/gerrit/15553
Change-Id: I6b452ad1aaffec74ecb7ac1bb9baa3a3a52e076c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328099
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jack Neus <jackneus@google.com>
Path.readlink is only available on Python 3.9, breaking compatibility
with all python versions below. os.readlink is already used in other
places of this file, so use it here as well.
Change-Id: I5acf8f5334a3e7c8de9cea1939d7e2b9af5f30ae
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327844
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sebastian Wagner <sebix@sebix.at>
Verify all the .git/ paths will be handled by the migration logic before
starting the migration. This way we still abort & log an error, but the
user gets to see it before we put the tree into a state that they have to
manually recover. Also add a few more known-safe-to-clobber paths.
Bug: https://crbug.com/gerrit/15273
Change-Id: If49d69b341bc960ddcafa30da333fb5ec7145b51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327557
Reviewed-by: Colin Cross <ccross@android.com>
Tested-by: Mike Frysinger <vapier@google.com>
This reverts commit d53cb9549a. As long as
repo's reference docs treat this feature as a work in progress and don't
cover it well enough to allow all repo maintainers to easily support it,
it is inconsistent to report to users that it is no longer in beta.
Thanks for vapier@google.com for noticing.
https://crbug.com/gerrit/15527 tracks the required documentation changes
before we'd be ready to roll forward again.
Change-Id: Ic9bd951cfb3c1abf6e1bfa30dfe4afa1c9b7bec6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327337
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jonathan Nieder <jrn@google.com>
Unsharing this directory shouldn't be a problem. The current repo code
treated it as a file, and while that's actually incorrect, files & dirs
are basically treated the same, so it's practically the same.
Let's enumerate each subpath since there aren't that many.
info/refs:
Only used when the project is exported over git dumb transports (i.e.
a http:// server). Repo never does this, and it's extremely unlikely
any user has ever done this. Plus, this proposal talks about unsharing
project refs, so this file should get unshared too.
info/grafts:
A user-configurable file that repo never touches. Might be useful to
share across projects, but probably rarely (if ever) used by developers,
and forcing them to configure it for each project isn't that big of a
deal.
info/exclude:
info/attributes:
User-configurable files that repo never touches. Doesn't seem like
most users ever touch these, and if they do, having them do it for
each shared project isn't a big deal.
info/sparse-checkout:
Repo doesn't use sparse checkouts, and it's extremely unlikely to even
work if a user tried doing something themselves.
Bug: https://crbug.com/gerrit/15460
Change-Id: I53e44d73a6d7a92da615b46600d8ea51cb46e3ac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327519
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Nothing uses this path. It’s only for exporting git dirs e.g. for
online gitweb use which probably no one does. It is not the same
description file as exists on servers we cloned from. Leaving it
as the default plain text file will simplify code.
We don't undo any existing symlinks if they exist since repo does
not care about them, and their existence doesn't hurt.
Bug: https://crbug.com/gerrit/15460
Change-Id: Ic34fe7c3cfb8f6da844de5be30158f59382b1cc8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327518
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This path only matters to users of `git svn` who manually run it in
local projects after they get a full repo client checkout. With svn
usage falling in general, and with the fact that the source checkout
now symlinks its .git/ state to the internal projects/ path, we don't
need to manage this anymore.
It means the path won't be shared among multiple local projects that
have the same remote, but so it goes. It was an optimization only,
not functionality required for correctness. We want to simplify the
internals to stop messing with git state, and this particular path
doesn't seem worth the effort to maintain.
We don't undo any existing svn symlinks if they exist since repo does
not care about them, and their existence doesn't hurt anything.
Bug: https://crbug.com/gerrit/15460
Change-Id: Ie8496b275bcc589771aa9f4ee874ed2ee6d5241d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327517
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
+ Display 'Defaulting to no-use-superproject because there is no working tree.'
message if --use-superproject option is used and we are not using
superproject because manifest is either a mirror or is an archive.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo init -u https://android.googlesource.com/mirror/manifest --mirror
$ repo_dev sync
Receiving objects: 100% (3/3), done.eiving objects: 33% (1/3)
$ repo_dev sync --use-superproject
Defaulting to no-use-superproject because there is no working tree.
Fetching: 0% (0/2158) warming up
Bug: https://crbug.com/gerrit/15368
Change-Id: I16b87ee9623315dbc3100b612b1decdaab7ac1dc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/325797
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Historically we created a .git/ subdir in each source checkout and
symlinked individual files to the .repo/projects/ paths. This layer
of indirection isn't actually needed: the .repo/projects/ paths are
guaranteed to only ever have a 1-to-1 mapping with the actual git
checkout. So we don't need to worry about having files in .git/ be
isolated.
To that end, change how we manage the actual project checkouts from
a dir full of symlinks (and a few files) to a symlink to the internal
.repo/projects/ dir. This makes the code simpler & faster.
The directory structure we have today is:
.repo/
project-objects/chromiumos/third_party/kernel.git/
<paths omitted as not relevant to this change>
projects/src/third_party/kernel/
v3.8.git/
config
description -> …/project-objects/…/config
FETCH_HEAD
HEAD
hooks/ -> …/project-objects/…/hooks/
info/ -> …/project-objects/…/info/
logs/
objects/ -> …/project-objects/…/objects/
packed-refs
refs/
rr-cache/ -> …/project-objects/…/rr-cache/
src/third_party/kernel/
v3.8/
.git/
config -> …/projects/…/v3.8.git/config
description -> …/project-objects/…/v3.8.git/description
HEAD
hooks/ -> …/project-objects/…/v3.8.git/hooks/
index
info/ -> …/project-objects/…/v3.8.git/info/
logs/ -> …/projects/…/v3.8.git/logs/
objects/ -> …/project-objects/…/v3.8.git/objects/
packed-refs -> …/projects/…/v3.8.git/packed-refs
refs/ -> …/projects/…/v3.8.git/refs/
rr-cache/ -> …/project-objects/…/v3.8.git/rr-cache/
The directory structure we have after this commit:
.repo/
<nothing changes>
src/third_party/kernel/
v3.8/
.git -> …/projects/…/v3.8.git
Bug: https://crbug.com/gerrit/15273
Change-Id: I9dd8def23fbfb2f4cb209a93f8b1b2b24002a444
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323695
Reviewed-by: Mike Nichols <mikenichols@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
init.py
+ Similar to opt.archive, gave an error if --mirror option is
used with --use-superproject.
sync.py
+ Defaulted to --no-use-superproject if manifest is a mirror or
archive (similar to error at line# 1067).
Tested:
+ run_tests
+ flake8 (will fix known errors in another CL).
$ repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest --use-superproject --mirror
Usage: repo init [options] [manifest url]
main.py: error: --mirror and --use-superproject cannot be used together.
+ repo init and repo sync with --mirror and without --mirror
options.
$ repo_dev init -u https://android.googlesource.com/platform/manifest
$ repo_dev sync
...superproject.git: Initial setup for superproject completed.
+ With --mirror option, verfied there are no exceptions in git_superproject.py
Bug: [google internal] b/206537893
Change-Id: I059f20e76f0ab36f0587f29779bb53ede4663bd4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323955
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
repo sync only handles a git tag properly when it is in the "revision"
field. However, "revision locked manifests" (`repo manifest
--revision-as-HEAD`) specifies the tag in the "upstream" field. The
issue is that this tag is not fetched. Only the commit that the tag
points to is fetched. This cases issues as
self._CheckForImmutableRevision() runs and comes to the conclusion that
the tag was changed while in fact, it was just not fetched. This causes
a full sync.
File docs/manifest-format.md, section Element-project:
> Attribute upstream: Name of the Git ref in which a sha1 can be found.
Used when syncing a revision locked manifest in -c mode to avoid having
to sync the entire ref space. Project elements not setting their own
upstream will inherit this value.
Change-Id: I0507d3a5f30aee8920a9f820bafedb48dd5db554
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323620
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Robin Schneider <ypid@riseup.net>
objdir is the .repo/project-objects/ dir based on the remote path.
gitdir is the .repo/projects/ dir based on the local source checkout
path. When we setup the gitdir, we symlink "hooks" to the one in the
objdir. But when we go to initialize the hooks, we do it via gitdir.
There is a 1-to-many mapping from project-objects to projects, so
initializing via gitdir can be repetitive. Collapse the hook init
logic to the objdir init path.
Change-Id: I828fca60ce6e125d6706c709cdb2797faa40aa50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323815
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If a remote deletes a ref, and it points to an object that doesn't
exist locally, we can get into a bad state, and the only way for the
user to recover is to run `repo sync --prune` (and to know that is
the option they need). The error message is not helpful:
fatal: bad object refs/remotes/cros/firmware-zork-13421.B-master
error: https://chromium.googlesource.com/chromiumos/platform/ec did not send all necessary objects
This situation can also come up when the remote renames refs in a
UNIX FS incompatible way. For example, replacing refs/heads/foo
with refs/heads/foo/bar.
Also add a --no-prune option for users to disable the behavior.
Bug: https://issuetracker.google.com/203366450
Change-Id: Icf45d838a10938feb091d29800f7e49240830ec3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322956
Reviewed-by: Andrew Lamb <andrewlamb@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Previously gsutil stderr was getting piped into stdout, which
yields bad results if there are non-fatal warnings in stderr.
Additionally, we should fail outright if gsutil fails (by adding
`check = True`) rather than fail later on when we try to sync to
a manifest that is in fact just a stderr dump.
BUG=none
TEST=manual runs with bad gs urls
Change-Id: Id71791d0c3f180bd0601ef2c783a8e8e4afa8f59
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/321935
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
sync should not attempt to sync the manifest project if it was
created from a standalone manifest. The current work around is to
run sync with --nmu.
BUG=none
TEST=manual runs
Change-Id: I2e121af0badf9642143e77c7af89d1c2d993b0f3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/321195
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
`repo init` doesn't do anything on re-init when the checkout has
been initialized using --standalone manifest. Rather than let the
tool run through its existing flows (which happen to noop), check
the args and explicitly quit if a bare `repo init` is run on a
standalone checkout.
BUG=none
TEST=manual tests
Change-Id: Ie4346ef6df1282ec3e3f8045a08138c93653fece
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/320735
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Rather than display "3.0" all the time and confuse people, extract
the version from the launcher we're signing and display that.
Also reformat the text to follow our current practice: upload the
versioned launcher by itself first, and then later copy that over
the default.
And while we're here, add tips for rollbacks.
Change-Id: I1654425c88e5c67d78879f2f33ad685c59be14dc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319637
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
gc.autoDetach is enabled by default which makes 'git gc --auto' return
immediately and run in background. This can lead to a pile up of
operations all using large amounts of memory at the same time. To avoid
this set gc.autoDetach to false so that the garbage collect task waits
for instances to finish before spawning more.
Bug: https://crbug.com/gerrit/15113
Test: repo sync -j # and check the number of 'git gc' processes
Change-Id: Ic0815156ba3db03972968f33f6f9f51e4928f23b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319835
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
The v2.14.[345] releases were cut on a branch based on v2.14.2. We
had some regression fixes we wanted in v2.14, but too many risky
changes landed in main since to cut another v2.14.x directly, and
we didn't want to destabilize even more by pushing a v2.15 right
away. So we branched to keep things healthy.
But people with old checkouts trying to upgrade from those versions
run into an old repo bug where it only selfupdates with fast-forwards,
and repo can't fast-forward from those divergent histories. So let's
do a merge commit to stitch the history back together.
There's no actual changes in here.
Change-Id: I05a96048e3846321e57c5f5224fb8dcf3c191d35
For older versions of repo, this would make it easier for it to perform
a self update by making it a fast-forward from the following tags:
v1.13.9.2, v1.13.9.3, v1.13.9.4, v1.13.10, v1.13.11
Change-Id: Ia75776312eaf802a150db8bd7c0a6dce57914580
removeprefix is available i python 3.9. Mac and Linux are running in
a version below 3.9. Thus tests are failing with the following error:
"AttributeError: 'str' object has no attribute 'removeprefix' "
Replaced the removeprefix with custom function which we will delete
once Linux and macOS versions are updated.
Tested:
$ ./run_tests
Bug: [google internal] b/201453085
Change-Id: I9b4d564ff1176e1b4471805ef05472c1914cd9f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319375
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Now that _do throws an exception when `git` fails, update the logic
that tries to read config files but the file doesn't exist.
Bug: b/192664812
Change-Id: I6417ecd70891b8f2d5f2bdb819f91df69ac4b70c
Test: `repo upload` no longer crashes when .repo/config doesn't exist
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319295
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When a hard revision ID is provided in a regular project tag then the
revisionId is updated as well if it is a commit hash. The difference
is that if the revisionExpr is a commit, git-repo needs to update
refs/remotes/m/master with update-ref not symbolic-ref, as the latter
must refer to another ref, not to a specific commit.
Change-Id: I215a62dabb30225e480ad2c731416d775fc0c750
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310963
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
The current code would check for enrollment status when the user did not
explicitly specify --[no-]use-superproject and do not have a remembered
value in their repo client. However, because superproject only makes
sense for manifests that have one specified, we should skip the
enrollment logic in that case.
Address this by checking manifest.superproject prior to proceeding. This
would avoid showing the greeting message of superproject enrollment
which can be confusing for developers.
Tested:
For manifest without superproject:
- repo sync --use-superproject will still show message for
superproject;
- repo sync will not show message regardless of enrollment state
For manifest with superproject:
- repo sync will show message and perform enrollment if not
previously enrolled
Bug: https://crbug.com/gerrit/15039
Change-Id: Ic2be9f9d037f0e7cf3446da474a5a0d0e4bd88da
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319255
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
All the values of syncstate are strings, check the first byte and last
byte to see if it is an array. For syncstate data, there were no false
positives.
Tested:
$ repo_dev sync
Verified event logged for argv is "data-json".
$./run_tests
Bug: [google internal] b/201102002
Change-Id: Id56adb532b80267f08d09147ac663cdd5987ce87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319075
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Some of the file removal calls are subject to race conditions (if
something else deletes the file), so extend our remove API to have
an option to ignore ENOENT errors. Then update a bunch of random
call sites to use this new functionality.
Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195
Reviewed-by: Sean McAllister <smcallis@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Added --standalone_manifest to repo tool. If set, the
manifest is downloaded directly from the appropriate source
(currently, we only support GS) and used instead of creating
a manifest git checkout. The manifests.git repo is still created to
keep track of various config but is marked as being for a standalone
manifest so that the repo tool doesn't try to run networked git
commands in it.
BUG=b:192664812
TEST=existing tests (no coverage), manual runs
Change-Id: I84378cbc7f8e515eabeccdde9665efc8cd2a9d21
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312942
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Fixed: "we need to make a special case for logging the argv; it
should probably be a "data-json" event so that we log this directly as
an array rather than an encoded string.
Tested:
$ repo_dev sync
Verified event logged for argv is "data-json".
$./run_tests
Bug: [google internal] b/201102002
Change-Id: I18ccec79c73c8dc931cb8afc472b2361db8aea4c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319055
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Tested:
$ ./run_tests
Verified that a manifest that specified superproject revision would use
the specified revision, and superproject will use the default revision.
Note that this is a slight behavior change from earlier repo versions,
which would always use the branch name of the manifest itself. However,
the new behavior would be more consisitent with regular "project"
element and would allow superproject be used even if it is not enabled
for the particular manifest branch, so we have decided to make the
change as it would provide more flexibility and better matches what
other elements would do.
Bug: [google internal] b/187868160
Change-Id: I35255ee347aff6e65179f7879d52931f168b477e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317643
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Stop tracking Ubuntu Trusty & Xenial and Debian Jessie & Stretch
as they only had Python 3.5 available which we've dropped.
Backfill OpenSSH versions since we've started testing for it.
Change-Id: I03183ed97f6e43dce8a00e36cce2956544a26afc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318835
Reviewed-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Symptom: repo sync exception
Root Cause: os.rename only works when source and destination are on the same file system
Solution: using shutil.move
to save disk usage, I create links for projects and project-objects, link to folder on another disk
lrwxrwxrwx 1 owenwen owenwen 47 Jun 9 16:40 project-objects -> /disk3/AndroidLocalRepos/.repo/project-objects/
lrwxrwxrwx 1 owenwen owenwen 40 Jun 9 16:40 projects -> /disk3/AndroidLocalRepos/.repo/projects/
below are exception I met:
"""
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/disk2/Android11/.repo/repo/subcmds/sync.py", line 550, in _CheckoutOne
project.Sync_LocalHalf(syncbuf, force_sync=force_sync)
File "/disk2/Android11/.repo/repo/project.py", line 1251, in Sync_LocalHalf
self._InitWorkTree(force_sync=force_sync, submodules=submodules)
File "/disk2/Android11/.repo/repo/project.py", line 2801, in _InitWorkTree
self._CheckDirReference(self.gitdir, dotgit, share_refs=True)
File "/disk2/Android11/.repo/repo/project.py", line 2674, in _CheckDirReference
platform_utils.rename(dst_path, src_path)
File "/disk2/Android11/.repo/repo/platform_utils.py", line 127, in rename
os.rename(src, dst)
OSError: [Errno 18] Invalid cross-device link: '/disk2/Android11/system/libhidl/.git/packed-refs' -> '/disk2/Android11/.repo/projects/system/libhidl.git/packed-refs'
"""
Change-Id: Ifda2f16530cc5a8f280169f482ee858f9e5241d3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/316002
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Currently, repo will fail to sync to a manifest if the definition
of the repo-hooks project comes after the repo-hooks element.
BUG=none
TEST=new test, run_tests
Change-Id: I0bf85625173492af6c6404d4b67543e96e670562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318520
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
Our internal experiments was a success so far and we are enrolling 100%
users now. Instead of asking every two weeks, simply consider a lack of
unexpired choice as accepting the system default.
With this change the user would still be able to override the system
default with --no-use-superproject, or to permanently set the choice in
user's profile with git config --global repo.superprojectchoice.
Bug: [google internal] b/190688390
Change-Id: Idc77a9cbf88a169d90304169e91f0d722dc4ac8b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317975
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Saved superproject's remote URL in _remote_url data and used it
in the _Fecth function.
Tested:
$ ./run_tests
$ flake8 git_superproject.py
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
$ repo_dev sync
Verified the all log messages have the following format.
repo superproject branch: <branch> url: <url> warning: <message>
Bug: [google internal] b/200072098
Change-Id: Iac6af7c99225479fd50bc6909396b22e0ce5f76b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318177
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
git_trace2_event_log.py:
+ Added LogDataConfigEvents method to log 'data' events.
Sync's current_sync_state and previous_sync_state are logged
as 'data' events in the current log.
It logs are key/value in the |config| argument. Each key is
prefixed with |prefix| argument.
The following are sample events that are logged during repo sync.
{"event":"data",
"sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
"thread":"MainThread",
"time":"2021-09-14T18:16:19.935846Z",
"key":"previous_sync_state/repo.syncstate.main.synctime",
"value":"2021-09-14T17:27:11.573717Z"}
{"event":"data",
"sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
"thread":"MainThread",
"time":"2021-09-14T18:16:19.955546Z",
"key":"current_sync_state/repo.syncstate.main.synctime",
"value":"2021-09-14T18:16:19.935979Z"}
tests/test_git_trace2_event_log.py:
+ Added unit tests
sync.py:
+ Changed logging calls to LogDataConfigEvents.
Tested:
$ ./run_tests
Tested it by running the following command multiple times.
$ repo_dev sync -j 20
repo sync has finished successfully
Verified config data is looged in trace2 event logs.
Bug: [google internal] b/199758376
Change-Id: I75fd830e90c1811ec28510538c99a2632b104e85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317823
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Currently the code would give a message that would appear like the user
have enrolled the experiment regardless of the actual choice. For users
who choose to not enroll in the experiment, we should give them
instructions to override (enable) superproject once instead of how to
disable it, which is what the code already behave.
Bug: [google internal] b/199167992
Change-Id: Iba3314cb510aedf024375a26baa8bc1d5e2846cf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317382
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
* Add footer to the version table, so easier to read and maintain.
* Add version entry for Ubuntu 21.04 Hirsute (non-LTS).
* Add version entry for Debian 11 Bullseye (LTS).
Change-Id: Ic72f911e616b1a13901e56074004f05cdc2c7633
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313322
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Xin Li <delphij@google.com>
1) If the manifest has superproject tag (git_master, etc), then
display error/warning messages (as it is doing today)
2) If the manifest doesn't have superproject tag (nest, chromeos
manifests), then don't display any error/warning messages about
superrproject (behave as though user has specified
--no-use-superproject).
3) Print error/warning messages if --use-superproject passed as
argument to repo sync.
4) No change in behavior for the repo init command.
git_superproject.py:
+ Fixed typo in _WriteManifestFile method name
+ Superproject accepts print_message as an argument and it defaults
to True. All messages that are printed to stderr are controlled by
this flag. If it is True, then messages get printed.
+ Added PrintMessages function which return true if either
--use-superproject is specified on the command line or if the
manifest has a superproject tag.
sync.py:
+ Displays the warning message if PrintMessgages are enabled and
passes that as argument to superproject object.
+ Added 'hassuperprojecttag' trace2 log entry for analysis. We can
find users/branches that are using superproject, but the manifest is
missing the superproject tag.
Tested:
$ ./run_tests
+ Verified printing of messages with and without superproject tag, with
with --use-superproject option.
+ aosp-master
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
$ repo_dev sync
+ A manifest without superproject tag.
$ repo_dev init -m $(pwd)/manifest_7482982.xml
$ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml
Bug: [google internal] b/196411099
Change-Id: I92166dcad15a4129fab82edcf869e7c8db3efd4b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314982
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The current help output might change based on the number of CPU cores
available (since it reflects the dynamic --jobs logic). This is good
for users running repo locally, but not good for shipping static man
pages. Hook the help output to have it generate the same output all
the time.
Change-Id: I3098ceddc0ad914b0b8e3b25d660b5a264cb41ee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312882
Reviewed-by: Roger Shimizu <rosh@debian.org>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
GitConfig doesn't save keys if the keys contain "_" characters. Some
of the options like mp_update, use_superproject have underscores.
This fixes issue with previous_sync_state missing some of the options.
Tested:
$ ./run_tests
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
Tested it by running the sync command multiple times and verifing
previous_sync_state and current_sync_state have the same keys.
$ repo_dev sync -j 20
repo sync has finished successfully
Verified config file has [syncstate ...] data saved.
Bug: [google internal] b/188573450
Change-Id: I16b52a164f9dd1633d7dad1d8cf6b151c629fcb1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313242
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
git_config.py:
+ Added SyncAnalysisState class, which saves the following data
into the config object.
++ sys.argv, options, superproject's logging data.
++ repo.*, branch.* and remote.* parameters from config object.
++ current time as synctime.
++ Version number of the object.
+ All the keys for the above data are prepended with 'repo.syncstate.'
+ Added GetSyncAnalysisStateData and UpdateSyncAnalysisState methods
to GitConfig object to save/get the above data.
git_trace2_event_log.py:
+ Added LogConfigEvents method with code from DefParamRepoEvents
to log events.
sync.py:
+ superproject_logging_data is a dictionary that collects all the
superproject data that is to be logged as trace2 event.
+ Sync at the end logs the previously saved syncstate.* parameters
as previous_sync_state. Then it calls config's UpdateSyncAnalysisState
to save and log all the current options, superproject logged data.
docs/internal-fs-layout.md:
+ Added doc string explaining [repo.syncstate ...] sections of
.repo/manifests.git/config file.
test_git_config.py:
+ Added unit test for the new methods of GitConfig object.
Tested:
$ ./run_tests
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
Tested it by running the following command multiple times.
$ repo_dev sync -j 20
repo sync has finished successfully
Verified config file has [syncstate ...] data saved.
Bug: [google internal] b/188573450
Change-Id: I1f914ce50f3382111b72940ca56de7c41b53d460
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313123
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Skip updating the superproject when -l is present and use the existing
superproject, if available (this would make sync -l work as it's
intended to do), and fall back to sync without superproject when not
(this would catch the case when superproject is enabled by automatic
rollout).
Tested:
$ repo sync -j 20 -n
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/android/src/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Fetching: 100% (1032/1032), done in 41.184s
...
$ repo_dev sync -j 20 -l
prebuilts/asuite/: discarding 1 commits
prebuilts/runtime/: discarding 1 commits
...
repo sync has finished successfully.
+ With superproject-override.xml and test it.
$ ls -l .repo/exp-superproject/
total 176
drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
-rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 superproject_override.xml
rtenneti@rtenneti:~/work/android/src/aosp$ repo_dev sync -j 20 -l
...
repo sync has finished successfully.
+ Rename the file superproject-override.xml and test it.
$ ls -l .repo/exp-superproject/
total 176
drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
-rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 temp.xml
$ repo_dev sync -j 20 -l
Checking out: 1% (12/1031) platform/external/rust/crates/fallible-streaming-iteexternal/linux-kselftest/: discarding 1 commits
prebuilts/remoteexecution-client/: discarding 1 commits
Checking out: 51% (536/1031) platform/prebuilts/gcc/darwin-x86/aarch64/....
....
Checking out: 100% (1031/1031), done in 5.478s
repo sync has finished successfully.
Bug: [google internal] b/184368268
Change-Id: I3aba5872e4f7c299977b92c2a39847ef28698c5a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312962
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Currently we have the behavior:
* `repo`: Equivalent to `repo help` -- only shows common subcommands
(with short description), and then exits 0.
* `repo --help`: Shows repo's core options, lists all commands (no
specific info), and then exits 0.
The first case is not behaving well:
* If you run `repo` without a specific subcommand, that's an error,
so we should be exiting 1 instead.
* Showing only subcommands and no actual option summary makes it seem
like repo itself doesn't take any options. This confuses users.
Let's rework things a bit. Now we have the behavior:
* `repo`: Shows repo's core options, lists all commands (no specific
info), and then exits 1.
* `repo --help`: Shows repo's core options, shows common subcommands
(with short description), and then exits 0.
* `repo --help-all`: Shows repo's core options, shows all subcommands
(with short description), and then exits 0.
Basically we swap the behavior of `repo` and `repo --help`, and fix
the exit status when the subcommand is missing.
The addition of --help-all is mostly for the man pages. We were
relying on `repo help --all` to generate the repo(1) man page, but
that too omitted the core repo options. Now the man page includes
all the core repo options and provides a summary of all commands.
Change-Id: I1f99b99d5b8af2591f96a078d0647a3d76d6b0fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312908
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We don't really care what the subcommand is set to when --version
output is requested, so stop enforcing it. This fixes some weird
behavior like `repo --version version` fails, but `repo --version
help` works.
The new logic skips subcommand validation, so `repo --version asdf`
will still display the version output. This matches git behavior,
and makes a bit of sense when we consider that the user really wants
to see the tool version, and probably doesn't care about anything
else on the command line.
Change-Id: I87454d473c2c8869344b3888a7affaa2e03f5b0f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312907
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The _ParseArgs method parses the arguments and processes some of
the options, with the rest left to the _Run method. Simplify the
_ParseArgs method to only parse arguments and have _Run handle all
actual processing.
This will make it easier to add more terminal options (ones that
exit immediately without a subcommand), and makes it easier to
understand the overall code flow.
Change-Id: I47f7274c3f2b59378fd479e403e70fb24b681536
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312906
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The current list output only shows project paths relative to the
root of the repo client checkout. It can be helpful to also get
a listing of paths based on other paths (e.g. the current working
directory), so add an option to repo list to support that. We'll
leverage this in bash completion to support completing projects by
their local paths and not just remote names.
Bug: https://crbug.com/gerrit/14797
Change-Id: Ia2b35d18c890217768448118b003874a1016efd4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312904
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
For older git-repo versions, we might have only fetched the SHA1
revision that was provided by the project, but have remote branch left
intact as long as they exist. When the remote branch become stale,
some repo operations like rebase would fail, and repo sync would not
correct this situation.
Fix this by tightening the requirement to also require the superproject
provided SHA1 be an ancestor or equal to the tip-of-tree of the remote
branch.
Bug: [google internal] b/193798453
Change-Id: Ie34c5d860dabb1cbd9f822da929088ec69c79cf6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312642
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Currently users need to look up the baseline manifest by loading the
specific manifest file. This exposes them to the internals of how the
manifest is stored which may potentially be fragile (eg: It was
switched from a symlink pointing at the file in the report to an
actual file with an 'include' tag).
Instead of doing this, we can provide an option to the 'repo manifest'
command which will emit the baseline manifest and decouple users from
the internal manifest details.
Change-Id: I12ee9160feaa591484ae71f404bc529be500ae4e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311202
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
superproject is going to be default for some users. This change
doesn't fail for repo init or repo sync if source couldn't be synced
because of errors in superproject and superproject=true in the config
file. The commands will fail if --use-superproject is specified on
the command line explicitly.
The error messages are logged with trace2 event logs and will be
monitored.
+ sync - When there are errors with superproject and git_superproject
says it is fatal failure, sync will exit only when --use-superproject
option is specified on the command line.
+ init - command doesn't fail *if there are any superproject errors),
but it will print a warning message and logs message via trace2 event
logs. For fatal errors, init will exit only when --use-superproject
option is specified on the command line.
+ All git commands log the command that is being executed so trace2
event logs will know the manifest, remote url and the branch name.
There is no functional change other than fatal errors are honored with
--use-supeproject option with init/sync commands.
Tested the code with the following commands.
$ ./run_tests -v
Test 1 - sync'ing without errors
--------------------------------
Added the following lines to '~/.repoconfig/config
[repo]
superproject = true
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in /usr/local/google/home/rtenneti/work/drive2/android/test
$ repo_dev sync
remote: Total 4 (delta 1), reused 4 (delta 1)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/drive2/android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
...
Test 2 - init and sync fail when --use-superproject option is passed
--------------------------------------------------------------------
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 --use-superproject
remote: Total 57 (delta 16), reused 56 (delta 16)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning
rtenneti@rtenneti2:~/work/drive2/android/test$ repo_dev sync --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: Cannot get project commit ids from manifest
warning: Update of revisionId from superproject has failed, repo sync will not use superproject to fetch the source. Please resync with the --no-use-superproject option to avoid this repo warning.
Test 3 - git fetch command fails and git command is printed
-----------------------------------------------------------
With config change
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2
...
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/test/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Performing initial setup for superproject; this might take several minutes.
repo: error: git fetch call failed,command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in ....
Test 4 - no superproject tag
-----------------------------
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b pie-dev
...
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: .../android/pie_dev/.repo/manifest.xml
warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in ...
$ repo_dev sync
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: /usr/local/google/home/rtenneti/work/drive2/android/pie_dev/.repo/manifest.xml
warning: Cannot get project commit ids from manifest
warning: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option to avoid the repo warning.
Bug: [google internal] b/192614798
Bug: [google internal] b/Bug: [google internal] b/192614798
Change-Id: I9a97a0e7d9e609fad151bd7dd9cfc523eaa887cd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311502
Reviewed-by: Amith Dsouza <amithds@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
When loading of superproject failed, we were resetting the manifest to
None, and later code would reload the manifest to see if there are
submodules, which would load the non-local manifest, causing sync with
superproject to fail.
Address this by setting the manifest_name to opt.manifest_name instead.
Bug: [google internal] b/189139268
Change-Id: I3616512e1c4b73e7eca0d83fd1fc474b825adbbf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311102
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
Sometimes, we don't care if the remove project is referring to a
non-existing project and we can just ignore it. This change allows us
to ignore remove-project entries if the project that they refer to
doesn't exist, making them effectively a no-op.
Because this change breaks existing configuration, we allow this to be
configuration controlled using the `optional` attribute in the
remove-project tag.
Change-Id: I6313a02983e81344eadcb4e47d7d6b037ee7420e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310964
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Repo will remember a choice and an expiration time of the choice, per
user, about whether to use superproject by default. When not specified
from command line and the choice is not expired, repo would use the
user default value.
When a user default value is not present and when the system wide
enable default is provided in git's system configuration, repo would
ask the user for a confirmation which will be valid for two weeks.
git_config.py: Add support for system config. When reading system
config, we would use --system to avoid hardcoding a path as the
value may be different on some other distributions.
git_superproject.py: Add a new subroutine, _UseSuperproject(), which
returns whether superproject should be used and whether it
is from a user configuration.
The value is determined in the following order:
1. If the user specifies either --use-superproject or
--no-use-superproject, then that choice is being used.
2. If neither is specified, we would then check the saved value
(upon repo init) and use that choice when there was a choice.
3. We then check if there is a saved and unexpired value for
user's choice in their ~/.gitconfig, and use the unexpired
choice, if available.
4. Finally, if all the above didn't give us a decision, and if
the git system configuration is providing a rollout hint, present
a prompt to user for their decision and save it in ~/.gitconfig.
subcmds/sync.py: Make use of the new UseSuperproject() provided by
git_superproject.py.
While there also silent stderr from git describe when determining the
version of repo.
Bug: [google internal] b/190688390
Change-Id: Iad3ee03026342ee500e5d65e2f0fa600d7637613
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309762
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
It is possible that a clone bundle contained the object referenced by
the branch in the manifest and in the superproject, but not the branch
itself (for example, the branch may be newly created from an existing
branch, or is not vislble to the user downloading the clone bundle).
When --use-superproject is enabled, because we are overriding
revisionExpr with the SHA1 revision provided by the superproject, the
verification step would succeed, but because the expected branch do not
exist, it would confuse git-repo at a later time, as it is expecting the
remote branch to exist in the local clone.
In project.py, fix this by making SetRevisionId() to always remember
the actual branch name and verify it in _CheckForImmutableRevision()
so that we only skip the fetch step when both objects exists locally.
Bug: [google internal] b/191974277
Change-Id: I49d3ca0667f524c8c45f416492faf95b1dd822fb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
Pinned manifests and release manifest have revisionId set for all
projects. For such projects don't update the commit ids.
Tested the code with the following commands.
$ ./run_tests -v
$ repo_dev sync --use-superproject -j8
$ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml
Bug: [google internal] b/191995372
Change-Id: I4681135b1d15f4a63527b6f0356d76ec842485d6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310582
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Don't exit if there are missing commit ids in superproject.
This change implements the following suggestion from delphij@:
"we should note the event (so we know that --use-superproject but there
were some errors, e.g. manifest didn't specify commit id for some
reason, or if there is no superproject but --use-superproject is
used), print out a message telling the use that this is not support,
but continue as if --no-use-superproject was specified?"
Changes:
superproject:
+ Added git_trace2_event_log as an argument to the constructor.
+ Sync method returns SyncResult a NamedTuple of
++ success - True if sync of superproject is successful, or False.
++ fatal - True if caller should exit, Or False.
+ UpdateProjectsRevisionId returns UpdateProjectsResult a NamedTuple of
++ manifest_path - path name of the overriding manifest file instead
of None
++ fatal - True if caller should exit, Or False
+ _GetAllProjectsCommitIds returns CommitIdsResult a NamedTuple of
++ commit_ids - a dictionary with the projects/commit ids on success,
otherwise None
++ fatal - True if caller should exit, Or False
+ Added _SkipUpdatingProjectRevisionId a helper function to see if a
project's revision id needs to be updated or not. This function is
used to exclude projects from local manifest file.
+ Added the following error events into git_trace2_event_log
++ If superproject is missing in a manifest
++ If there are missing commit ids for projects.
command.py:
+ Deleted unused import - platform
+ Added git_trace2_event_log as a member so all subcmds can log error
events.
main.py:
+ Initialized git_trace2_event_log as a member of command object.
init.py:
+ Deleted unused import - optparse
init.py:
+ Called sys.exit only if Sync returns exit=True
sync.py:
+ Called sys.exit only if Superproject's UpdateProjectsRevisionId returns
exit=True
+ Reloaded the manifest only if manifest path is returned by
UpdateProjectsRevisionId. If not, fall back to the old way of doing
repo sync.
test_git_superproject:
+ Added code to verify error events are being logged.
+ Added a test for no superproject tag
+ Added test for UpdateProjectsRevisionId not updating the revision id
with the commit ids.
Tested the code with the following commands.
+ Positive test case with aosp-master.
$ repo_dev init -u persistent-https://android.git.corp.google.com/platform/manifest -b master --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Your identity is: Raman Tenneti <rtenneti@google.com>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in .../android/aosp
$ repo_dev sync -j40 --use-superproject
remote: Total 12 (delta 4), reused 12 (delta 4)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
...
repo sync has finished successfully.
+ Negative test case without superproject tag.
$ repo_dev sync -j40 --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
repo error: superproject tag is not defined in manifest: .../android/aosp/.repo/manifest.xml
error: Cannot get project commit ids from manifest
error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
...
Checking out: 100% (1022/1022), done in 3.589s
repo sync has finished successfully.
+ Test for missing commit_id for a project.
$ repo_dev sync -j40 --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
error: please file a bug using go/repo-bug to report missing commit_ids for: ['build/blueprint']
error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
...
Checking out: 100% (1022/1022), done in 3.364s
repo sync has finished successfully.
$ ./run_tests -v
...
...== 164 passed in 2.87s ==...
Bug: [google internal] b/189371541
Change-Id: I5ea49f87e8fa41be590fc0c914573e16c8cdfcfa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309162
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
With repo sync --use-superproject, don't update the commit ids of every project
that comes from local manifest.
Tested the code with the following commands.
$ ./run_tests -v
+ Test with local.xml
1. repo init --use-superproject -u persistent-https://googleplex-android.git.corp.google.com/a/platform/manifest
2. cd .repo
cp -r /google/src/head/depot/google3/wireless/android/build_tools/translations/pipeline/local_manifests local_manifests
cd ..
local$ time repo_dev sync --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../local/.repo/exp-superproject/feb2c2847da5e274f3d530d5ab438af8-superproject.git: Initial setup for superproject completed.
...
Bug: [google internal] b/189360443
Bug: [google internal] b/189139268
Bug: https://crbug.com/gerrit/14499
Change-Id: Ideaf268c294e9b500b2b9726ffbd733dd8d63004
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308822
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
superproject supports multiple remotes. Get all commit ids
from superproject for all projects that are in the manifest.
$ ./run_tests -v
Bug: [google internal] b/186395810
Change-Id: I6edce3918853a7a3a65aec5528e6a43a544eff53
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308862
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Finding the "right" pytest is challenging. In Debian, `pytest` is
Python 2 while `pytest-3` is the Python 3 version ... but only when
outside of a virtualenv. Inside of a virtualenv (e.g. the ones that
tox creates), we always want `pytest`.
Change-Id: Ic1fe84c10f06227bceeb9baad6a3c4598bbe9860
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303802
Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Running repo with Python 3.5 fails due to the use of the encoding
parameter to subprocess.run(). There are also f-strings being used in
some of the tests.
This drops support for these systems:
* Ubuntu Xenial: released Apr 2016, EOS Apr 2021, EOL Apr 2024
* Debian Stretch: released Jun 2017, EOL Jun 2022
So the minimum required distros now are:
* Ubuntu Bionic: released Apr 2018 w/Python 3.6
* Debian Buster: released Jul 2019 w/Python 3.7
Change-Id: I1144f7ab6f882b10cac0131982df081fe4ac44f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303363
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Before Python 3.8, xml.dom.minidom sorted the attributes of an element
when writing it to a file, while later versions output the attributes
in the order they were created. Avoid these differences by sorting the
attributes for each element before comparing the generated manifests
with the expected ones.
Bug: https://crbug.com/gerrit/14382
Change-Id: Ie2597727afcc48f9063a7261ad970e8a549f0587
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303326
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit b4429439 (sync: refactor main fetch loop) broke syncing of
submodules with sync-s="true". The first `repo sync` would just fetch
the superrepo, while the second `repo sync` would fetch the submodules.
This was due to the new _FetchMain() function not passing back the
all_projects variable that it had modified.
Change-Id: Ie8187cde7bb894e4e9a6b76c2aed83873d9f69a4
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/307065
Reviewed-by: Mike Frysinger <vapier@google.com>
We always pass in output_redir when syncing, but that's the common case:
there are a few situations (like `repo init`) where we don't pass in a
buffer, and if any errors show up in that case, we'd crash. Rely on the
print function to handle this logic for us.
Bug: https://crbug.com/gerrit/14568
Change-Id: I8cd47e82329797ffc42534418a3dfbd8429205be
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/307222
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
+ In XmlManifest._Unload set 'bugurl' to Wrapper().BUG_URL.
+ contactinfo returns a namedtuple.
+ bug_url can be accessed as self._manifest.contactinfo.bugurl.
Tested the code with the following commands.
$ ./run_tests -v
Added contactinfo tag to default.xml and verified that bugurl is used.
Bug: [google internal] b/186220520.
Change-Id: Iaafd6465e072b2e47a0a0b548bf6cb608a0b0a04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/306342
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
We changed sync to use multiprocessing for parallel work. This broke
the ssh proxy code as it's all based on threads. Rewrite the logic to
be multiprocessing safe.
Now instead of the module acting as a stateful object, callers have to
instantiate a new ProxyManager class that holds all the state, an pass
that down to any users.
Bug: https://crbug.com/gerrit/12389
Change-Id: I4b1af116f7306b91e825d3c56fb4274c9b033562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305486
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
The only time we really need ssh proxies is when we want to run many
connections and reuse them. That only happens when running sync.
Every other command makes at most two connections, and even then it's
only one or none. So the effort of setting up & tearing down ssh
proxies isn't worth it most of the time.
The big reason we want to move this logic to sync is that it's now
using multiprocessing for parallel work. The current ssh proxy code
is all based on threads, which means none of the logic is working
correctly. The current ssh design makes it hard to fix when all of
the state lives in the global/module scope.
So the first step to fixing this is top move the setup & teardown to
the one place that really needs it: sync. No other commands will use
proxies anymore, just direct connections.
Bug: https://crbug.com/gerrit/12389
Change-Id: Ibd351acdec39a87562b3013637c5df4ea34e03c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305485
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Gitlab returns a 400 error when trying to fetch clone.bundle
from a repository containing the git-repo tool. The repo
launcher doesn't then fall back to not using a clone.bundle
file and the repo init fails.
Change-Id: Ia3390d0638ef9a39fb2fab84625b269d28caf1cc
Signed-off-by: Craig Northway <cnorthway@codeaurora.org>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305382
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v
$ repo_dev sync -c -j8 --no-use-superproject
Fetching: 100% (1041/1041), done in 1m22.743s
$ repo_dev sync -c -j8 --use-superproject
WARNING: --use-superproject is experimental and not for general use
..
Bug: [google internal] b/187459275
Change-Id: I3f4269df38cd24a21723e8b2be5a1f013e7b5a91
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305682
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If superproject was not available for a branch, then the next
repo sync would also fail because --use-superproject is
remembered across repo init. In such cases, hoping the hint to
to use --no-use-superproject will help.
Tested the code with the following commands and by forcing
a failure.
$ ./run_tests -v
Bug: [google internal] b/187459275
Change-Id: Ie250812b7ba83afc230b5b1d154ba11f245f8b8a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305622
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The ssh master logic has never worked under Windows which is why this
code always returned False when running there (including cygwin). But
the OS check was still done while holding the threading lock. While
it might be a little slower than necessary, it still worked.
The switch from the threading module to the multiprocessing module
changed global behavior subtly under Windows and broke things: the
globals previously would stay valid, but now they get cleared. So
the lock is reset to None in children workers.
We could tweak the logic to pass the lock through, but there isn't
much point when the rest of the code is still disabled in Windows.
So perform the platform check before we grab the lock. This fixes
the crash, and probably speeds things up a few nanoseconds.
This shouldn't be a problem on Linux systems as the platform fork
will duplicate the existing process memory (including globals).
Bug: https://crbug.com/gerrit/14480
Change-Id: I1d1da82c6d7bd6b8cdc1f03f640a520ecd047063
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305149
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The ssh master logic has never worked under Windows which is why this
code always returned False when running there (including cygwin). But
the OS check was still done while holding the threading lock. While
it might be a little slower than necessary, it still worked.
The switch from the threading module to the multiprocessing module
changed global behavior subtly under Windows and broke things: the
globals previously would stay valid, but now they get cleared. So
the lock is reset to None in children workers.
We could tweak the logic to pass the lock through, but there isn't
much point when the rest of the code is still disabled in Windows.
So perform the platform check before we grab the lock. This fixes
the crash, and probably speeds things up a few nanoseconds.
This shouldn't be a problem on Linux systems as the platform fork
will duplicate the existing process memory (including globals).
Bug: https://crbug.com/gerrit/14480
Change-Id: I1d1da82c6d7bd6b8cdc1f03f640a520ecd047063
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305149
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
It will be used to let manifest authors self-register contact info.
This element can be repeated, and any later entries will clobber
earlier ones. This would allow manifest authors who extend
manifests to specify their own contact info.
It would have 1 required attribute: bugurl.
"bugurl" specifies the URL to file a bug against the manifest owner.
<contactinfo bugurl="bug-url"/>
TODO: This CL only implements the parsing logic and further work
will be in followup CLs.
Tested the code with the following commands.
$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v
Bug: [google internal] b/186220520.
Change-Id: I47e765ba2dab5cdf850191129f4d4cd6b803f451
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305203
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit fb527e3f52 ("sync: create dedicated
manifest project update func") refactored code from the main body into a
dedicated method. The manifest_name was passed as an argument, but never
used it, and instead reaches back out to the command line options. This
ignores the logic in the main loop where manifest_name might have changed
(like when using smart sync).
Change-Id: I4b84638fbb10c2b6f8f4b555e1475b0669c2daf4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305148
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We've switched most of this command over to multiprocessing and off
of _threading, so do the Event object too. The APIs are the same
between the modules, so we shouldn't need to update anything else.
Change-Id: I52d31f1c6ef2bcbe7bbc1dd1add79a8d5d08784a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305147
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We're inconsistent with help text as to whether it uses title case and
whether it ends in a period. Add a test to enforce a standard, and use
the style that Python optparse & argparse use themselves (e.g. with the
--help option): always lowercase, and never trailing period.
Change-Id: Ic1defae23daeac0ac9116aaf487427f50b34050d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305144
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
While this provides a way to undo earlier command line options (e.g.
`repo sync --tags --no-tags`) which can be helpful for scripting &
automation, this more importantly allows the user to override the
manifest settings for syncing tags from a project.
Bug: https://crbug.com/gerrit/12401
Change-Id: Id4c36cd82e6ca7cb073b5d63a09f6c7ccdebba83
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304904
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
For most commands, this is more about providing a way to undo earlier
command line options (e.g. `repo info -c --no-current-branch`) which
can be helpful for scripting & automation. But for the sync command,
this is helpful to undo the setting that exists in the manifest itself.
With this in place, tweak the sync current_branch_only logic to only
apply the manifest settings when the user hasn't specified a command
line option.
Bug: https://crbug.com/gerrit/12401
Change-Id: I21e2384624680cc740d1b5d1e49c50589d2fe6a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304903
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We're inconsistent with the short option for this flag:
* gitc-init: <none as -c is already used>
* info: -b
* init: -c
* overview: -b
* sync: -c
* upload: --cbr
Since info & overview are not as heavily used as the others, switch
them from -b to -c. We leave -b in as a hidden alias for now.
Similarly, switch upload from --cbr to just -c. A lot of people
use --cbr, so we leave this as a hidden alias for now too.
Ideally gitc-init wouldn't use -c, but that ship has sailed, and
we're more likely to deprecate gitc entirely at this point.
This provides a consistent set of options across subcommands.
Bug: https://crbug.com/gerrit/12401
Change-Id: Iec249729223866fe1ea0ebabed12ca851cc38b35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304902
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Search for project branches to upload in parallel. This can cut the
lookup time in half for large projects. We still run the actual hooks
in serial once we have the list of projects to process, but we would
need to rethink things quite a bit before we could handle running them
in parallel too.
Change-Id: I8da0cbc5010566aa860e1a158f3dc07f0709dcff
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304842
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We were updating the per-checkout m/ pseudo ref when syncing, but we
only created the common m/ redirect when initializing a project for
the first time. This is fine unless the user switches the manifest
branch in an existing project, then we never create that redirect.
Bug: https://crbug.com/gerrit/14468
Change-Id: I5325e7e602dcb4ce150bef258901ba5e9fdea461
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304822
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When running upload across multiple projects, include the project in
any error messages that come up. This lets users figure out where
the problem might be.
Change-Id: I09470c9a1b512baf910d6d97b747816d1a6f3a87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304783
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Currently, copyfiles and linkfiles which marked by
"<copyfile/>" and "<linkfile/>" in manifest will
be created by first exec 'repo sync'.
But if some "<copyfile/>" or "<linkfile/>" are removed
in manifest, then 'repo sync', these removed item
dest can not be removed in the sourcecode workspace.
This patch is intent to fix this issue, by save a
'copy-link-files.json' in .repo and then compared with
new dest path when next sync. If any "<copyfile/>" or
"<linkfile/>" were removed, the dest path will be
removed in sourcecode at the same time.
Bug: https://crbug.com/gerrit/11008
Change-Id: I6b7b41e94df0f9e6e52801ec755951a4c572d05d
Signed-off-by: jiajia tang <tangjiajia@xiaomi.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304202
Reviewed-by: Mike Frysinger <vapier@google.com>
When depth is used, we would fetch only SHA1 when superproject is
used, as the result, only the manifest branch is being recorded,
and commands like repo start would fail.
Fix this by saving the upstream branch value in the overlay
manifest and add the upstream branch to fetch list.
Bug: [google internal] b/185951360
Change-Id: Ib36f56067723f2572ed817785b31cc928ddfec0a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304562
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Xin Li <delphij@google.com>
Default the partial_clone_exclude argument to an empty set.
Fixes the following report by Emil Medve.
With this change (up to v2.14.1), on an existing "normal" clone (without partial-clone options) I'm seeing this traceback during `repo selfupdate`:
Traceback (most recent call last):
File ".../.repo/repo/main.py", line 630, in <module>
_Main(sys.argv[1:])
File ".../.repo/repo/main.py", line 604, in _Main
result = run()
File ".../.repo/repo/main.py", line 597, in <lambda>
run = lambda: repo._Run(name, gopts, argv) or 0
File ".../.repo/repo/main.py", line 261, in _Run
result = cmd.Execute(copts, cargs)
File ".../.repo/repo/subcmds/selfupdate.py", line 54, in Execute
if not rp.Sync_NetworkHalf():
File ".../.repo/repo/project.py", line 1091, in Sync_NetworkHalf
if self.name in partial_clone_exclude:
TypeError: argument of type 'NoneType' is not iterable
$ ./run_tests -v
Change-Id: I71e744e4ef2a37b13aa9ba42eba3935e78c4e40a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304082
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The merge of the repo & init parser missed this default.
When running `repo init ...` in an existing checkout but w/out the -m
option, then repo would error out complaining that -m is required when
it didn't do this before.
Change-Id: I58035d48cc413b5d373702b9dc3b9ecd3fd1e900
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303945
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Make sure we print a message whenever we retry so it's clear to the
user why repo is pausing for a long time, and why repo might have
passed even though it displayed some errors earlier.
Also unify the sleep logic so we don't have two independent methods.
This makes it easier to reason about.
Also don't sleep if we're in the last iteration of the for loop. It
doesn't make sense to and needlessly slows things down when there are
real errors.
Bug: https://crbug.com/gerrit/12494
Change-Id: Ifceace5b2dde75c2dac39ea5388527dd37376336
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303402
Reviewed-by: Sam Saccone 🐐 <samccone@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
In _FetchOne & _CheckOne, only print error.GitError exception,
but other exceptions are still thrown
Fixes the GitError exceptions from /usr/lib/python3.8/multiprocessing/pool.py
exiting the repo sync.
Tested the code with the following commands and verified repo sync
continues after fetch error because of an invalid SHA1.
$ ./run_tests -v
$ python3 ~/work/repo/git-repo/repo sync -m manifest_P21623846.xml -j32
...
error.GitError: Cannot fetch platform/vendor/google_devices/redbull/proprietary update-ref: fatal: d5a99e518f09d6abb0c0dfa899594e1ea6232459^0: not a valid SHA1
....
An error like the following when jobs=1
error.GitError: Cannot checkout platform/vendor/qcom/sdm845/proprietary/qcrilOemHook: Cannot initialize work tree for platform/vendor/qcom/sdm845/proprietary/qcrilOemHook
Bug: https://crbug.com/gerrit/14392
Change-Id: I8922ad6c07c733125419f5698b0f7e32d70c7905
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303544
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Now that we have a bunch of subcommands doing parallel execution, a
common pattern arises that we can factor out for most of them. We
leave forall alone as it's a bit too complicated atm to cut over.
Change-Id: I3617a4f7c66142bcd1ab030cb4cca698a65010ac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301942
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Since Python has deprecated the formatter module, switch to the textwrap
module instead for reflowing text. We weren't really using any other
feature anyways.
Verified by diffing the output before & after the change and making sure
it was the same.
Then made a few tweaks to tighten up the output.
Change-Id: I0be1bc2a6661a311b1a4693c80d0f8366320ba55
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303282
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Currently, list only shows projects that exist in the checkout, and
doesn't offer any way to list all projects in the manifest (based on
the current settings, or on the options passed to list). This seems
to be the opposite of what (at least some) users expect, so let's
add an option to show all of them regardless of checkout state.
Change-Id: I94bbdc5bd0ff2a411704fa215e7fc2b60fa3360e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301263
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We want progress bars in the default output mode, but not when the
user specifies --quiet. Add a setting to the Progress bar class so
it takes care of not displaying anything itself rather than having
to update every subcommand to conditionally setup & call the object.
Change-Id: I1134993bffc5437bc22e26be11a512125f10597f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303225
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Add new CommonOptions entry points to move the existing --jobs to,
and relocate all --verbose/--quiet options to that. This provides
both a consistent interface for users as well as for code.
Change-Id: Ifaf83b88872421f4749b073c472b4a67ca6c0437
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303224
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
partial-clone-exclude option excludes projects during
partial clone. This is a comma-delimited project names
(from manifest.xml). This option is persisted and it
is used by the sync command.
A project that has been unparital'ed will remain unpartial if
that project's name is specified in the --partial-clone-exclude
option. The project name should match exactly.
Added
$ ./run_tests -v
Bug: [google internal] b/175712967
"I can't "unpartial" my androidx-main checkout"
$ rm -rf androidx-main/
$ mkdir androidx-main/
$ cd androidx-main/
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml
$ repo_dev sync -c -j8
+ Verify a project is partial
$ cd frameworks/support/
$ git config -l | grep 'partial'
+ Unpartial a project.
$ /google/bin/releases/android/git_repack/git_unpartial
+ Verify project is unpartial
$ git config -l | grep 'partial'
$ cd ../..
+ Exclude the project from being unparial'ed after init and sync.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml
+ Verify project is unpartial
$ cd frameworks/support/
$ git config -l | grep 'partial'
$ cd ../..
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep 'partial'
$ cd ../..
+ Remove the project from exclude list and verify that project is partially cloned.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep 'partial'
Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The number of jobs one wants to run against the network tends to
factor differently from the number of jobs one wants to run when
checking out local projects. The former is constrained by your
internet connection & server limits while the later is constrained
by your local computer's CPU & storage I/O. People with beefier
computers probably want to keep the network/server jobs bounded a
bit lower than the local/checkout jobs.
Change-Id: Ia27ab682c62c09d244a8a1427b1c65acf0116c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302804
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The current logic has a downside in that it doesn't sync to the latest
signed version available if the latest commit itself is unsigned. This
can come up when using the "main" branch as it is sometimes signed, but
often not as it's holding the latest merged commits. When people use
the main branch, it's to get early testing on versions tagged but not
yet released, and we don't want them to get stuck indefinitely on that
old version of repo.
For example, this series of events:
* "stable" is at v2.12.
* "main" is tagged with v2.13.
* early testers use --repo-rev main to get v2.13.
* new commits are merged to "main".
* "main" is tagged with v2.14.
* new commits are merged to "main".
* devs who had synced in the past to test v2.13 are stuck on v2.13.
repo sees "main" is unsigned and so doesn't try to upgrade at all.
The only way to get unwedged is to re-run `repo init --repo-rev main`,
or to manually sync once with repo verification disabled, or for us to
leave "main" signed for a while and hope devs will sync in that window.
The new logic is that whenever changes are available, we switch to the
latest signed tag. We also replace some of the duplicated verification
code in the sync command with the newer wrapper logic. This handles a
couple of important scenarios inaddition to above:
* rollback (e.g. v2.13.8 -> v2.13.7)
* do not trash uncommitted changes (in case of ad-hoc testing)
* switch tag histories (e.g. v2.13.8 -> v2.13.8-cr1)
Change-Id: I5b45ba1dd26a7c582700ee3711f303dc7538579b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300122
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We need to pass back an int, not a CompletedProcess object. Switch to
check=False so we don't throw an exception on failure -- we're already
showing pytest's stderr, and will return the non-zero status.
Change-Id: Ib0d3862a09a3963f25025f39a8e34419cf2a54df
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299624
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This avoids GIL limitations with using threads for parallel processing.
This reworks the fetch logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.
We have to tweak the chunking logic a little here because multiprocessing
favors batching over returning immediate results when using a larger value
for chunksize. When a single job can be quite slow, this tradeoff is not
good UX.
Bug: https://crbug.com/gerrit/12389
Change-Id: I0f0512d15ad7332d1eb28aff52c29d378acc9e1d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298642
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~40sec to ~16sec with the default -j8.
The output processing does not appear to be a significant bottle
neck -- it accounts for <1sec out of the ~16sec runtime. Thus we
leave it in the main thread to simplify the code.
Change-Id: I750b72c7711b0c5d26e65d480738fbaac3a69971
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297984
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~10sec to ~4sec with the default -j8.
This only does a simple conversion over to get an easy speedup. It
is currently written to collect all results before displaying them.
If we refactored this module more, we could have it display results
as they came in.
Change-Id: I5caf4ca51df0b7f078f0db104ae5232268482c1c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298643
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v
Bug: [google internal] b/183232698
Bug: https://crbug.com/gerrit/13707
$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync --use-superproject
$ repo_dev sync
real 0m8.046s
user 0m2.866s
sys 0m2.457s
Second time repo sync took only 8 seconds and verified by printing that
urrent_branch_only is True in project.py's Sync_NetworkHalf function.
Change-Id: Ic48efb23ea427dfa36e12a5c49973d6ae776d818
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301182
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v
Without --depth
$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject
real 6m48.086s
user 3m27.281s
sys 1m1.386s
With --depth=1
$ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject
real 2m49.637s
user 2m51.458s
sys 0m39.108s
From dwillemsen@:
"For me it's the difference between 9m28s using the current code and
16s using --depth=1 while fetching the superproject."
Bug: [google internal] b/180451672
Bug: https://crbug.com/gerrit/13707
Change-Id: I1c3e4aef4414c4e9dd259fb6e4619da0421896b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300922
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Add a new "event": "command", which is emitted at when all command
arguments have been processed.
Additional fields:
"name": Name of the primary command (ex: repo, git)
"subcommands"': List of the sub-commands once command-line arguments
are processed
Examples:
Command: repo --version
Event: {"event": "command", <common fields>,
"name": "repo",
"subcommands": ["version"]
}
Bug: [google internal] b/178507266
Testing:
- Unit tests
- Verified repo git trace2 logs had expected data
Change-Id: I825bd0ecedee45135382461a4ba10f987f09aef3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300343
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
The refactor to multiprocessing broke status reporting slightly when
checking out projects. Make sure we mark the step as failed if any
of the projects failed, not just when --fail-fast is set.
Change-Id: I0efb56ce83b068b2c334046df3fef23d797599c9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299882
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We've had a limited version of this in CrOS for a long time. There's
nothing CrOS specific about it, so lets move it to the repo project so
everyone can utilize it.
Change-Id: I04cd94610c1100f3afcd2baf8c8e7ab13e589490
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299202
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the current project doesn't have any local branches, then there's
nothing to prune, so return right away. This avoids running a few
git commands when we aren't actually going to use the results, and
it avoids checking repository validity. Since we aren't going to do
anything in here, no need to check it.
Change-Id: Ie9d5c75a954e42807477299f3e5a63a92fac138b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299742
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Added the following methods to XmlManifest class.
+ GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower()
+ GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func.
+ Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr.
+ Used the above methods to get groups in command.py::GetProjects
and part of init.py.
TODO: clean up these funcs to take structured group data more instead
of passing strings around everywhere that need parsing.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL
and verified prebuilts/fullsdk-linux directory has all the folders.
Tested repo init and repo sync with --use-superproject and without
--use-superproject argument.
$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
$ repo_dev sync -c -j32
Bug: [google internal] b/181804931
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Superproject objects accept the optional argument “quiet”.
The following progress messages are displayed if quiet is false.
Displayed the following message whenever we find we have to make a new
folder (aka new remote), because if you started with repo init android
and later do googleplex-android that is when it will be slow.
"<location>: Performing initial setup for superproject; this might take
several minutes.".
After fetch completion, added the following notification:
"<location>: Initial setup for superproject completed."
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
Bug: [google internal] b/181178282
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Allow the user to specify relative or absolute or any other funky
path that they want when using `repo init` or `repo sync`. Our
goal is to restrict the paths in the remote manifest git repo we
cloned from the network, not protect the user from themselves.
Bug: https://crbug.com/gerrit/14156
Change-Id: I1ccfb2a6bd1dce2bd765e261bef0bbf0f8a9beb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298823
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
With the recent commit 0501b29e7a
("status: Use multiprocessing for `repo status -j<num>` instead of
threading"), the limitation with project serialization no longer
applies. It turns out that ad-hoc logic is expensive. In the CrOS
checkout (~1000 projects w/8 jobs by default), it adds about ~7sec
overhead to all invocations. With a fast nop run:
time repo forall -j8 -c true
This goes from ~11sec to ~4sec -- more than 50% speedup.
Change-Id: Ie6bcccd21eef20440692751b7ebd36c890d5bbcc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298724
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The ProjectArgs function can be inlined which simplifies it quite a
bit. We shouldn't need the custom exception handling here either.
This also makes the next commit easier to review.
Change-Id: If3be04f58c302c36a0f20b99de0f67e78beac141
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298723
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This is in preparation for simplifying the jobs support. The nested
function is referenced in the options object which can't be pickled,
so pull it out into a static method instead.
Change-Id: I01d3c4eaabcb8b8775ddf22312a6e142c84cb77d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298722
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If a manifest checksout a project multiple times, repo download isn't
able to accurately pick the right project. We were just picking the
first result which could be a bit random for the user. If we hit that
situation, check if the cwd is one of the projects, and if it isn't,
we emit an error and tell the user it's an ambiguous request.
Bug: https://crbug.com/gerrit/13070
Change-Id: Id1059b81330229126b48c7312569b37504808383
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298702
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The clone bundle logic assumes there is a one-to-one mapping between the
projects/ and project-objects/ trees. When using shared projects (where
we checkout different branches from the same project), this would lead us
to fetching the same clone bundle multiple times. Automatically skip the
clone bundle logic if the project-objects/ dir already exists.
Bug: https://crbug.com/gerrit/10993
Change-Id: I82c6fa1faf8605fd56c104fcea2a43dd4eecbce4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
On macOS, the Finder app likes to poop .DS_Store files in every path
that the user browses. If the user pokes around the .git/ tree, it
could generate a .DS_Store file in there too. When repo goes to read
all the local refs, it tries to decode this binary file as UTF-8 and
subsequently crashes.
Since paths that begin with . are not valid refs, ignore them like we
already do with paths that end in .lock. Also bump the check up to
ignore dirs that match since that follows the git rules: they apply
to any component in its path, not just the final path (name).
We don't implement the full valid ref algorithm that git employs as
it's a bit complicated, and we only really need to focus on what will
practically show up locally.
Bug: https://crbug.com/gerrit/14162
Change-Id: I6519f990e33cc58a72fcb00c0f983ad3285ace3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298662
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
When using Git under Windows, it seems that Git doesn't always parse
GIT_DIR correctly when it uses the Windows \ form, but does when it
uses / only.
For example, when using worktrees:
$ GIT_DIR='C:\Users\vapier\Desktop\repo\breakpad\tools\test\.git' git worktree list
fatal: not a git repository: ..\..\.repo\worktrees\linux-syscall-support.git\worktrees\test
$ GIT_DIR='C:/Users/vapier/Desktop/repo/breakpad/tools/test/.git' git worktree list
C:/Users/vapier/Desktop/repo/breakpad/.repo/worktrees/linux-syscall-support.git fd00dbbd0c06 (detached HEAD)
..\..\..\..\..\src\src\third_party\lss\.git fd00dbbd0c06 (detached HEAD)
..\..\..\..\..\tools\test\.git fd00dbbd0c06 (detached HEAD)
Change-Id: I666c03ae845ecb55d7f9800731ea6987d3e7f401
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298622
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This function is currently written with copyfile & linkfile in mind.
Generalize the logic & function arguments slightly so we can reuse
in more places that make sense.
This changes the validation logic slightly too in that we no longer
allow "." for the dest attribute with copyfile & linkfile, nor for
the src attribute with copyfile. We already rejected those later on
when checking against the active filesystem, but now we reject them
a little sooner when parsing.
The empty path check isn't a new requirement exactly -- repo used to
crash on it, so it was effectively blocked, but now we diagnosis it.
Bug: https://crbug.com/gerrit/14156
Change-Id: I0fdb42a3da60ed149ff1997c5dd4b85da70eec3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can speed things up. Across 1000 repos, it goes from
~9sec to ~5sec with the default -j8.
Change-Id: Ida6dd565db78ff7bac0ecb25d2805e8a1bf78048
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297982
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This avoids GIL limitations with using threads for parallel processing.
In a CrOS checkout with ~1000 repos, the nop case goes from ~6 sec down
to ~4 sec with -j8. Not a big deal, but shows that this actually works
to speed things up unlike the threading model.
This reworks the checkout logic to return results for processing in the
main thread instead of leaving every thread to do its own processing.
Bug: https://crbug.com/gerrit/12389
Change-Id: I143e5e3f7158e83ea67e2d14e5552153a874248a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298063
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run in parallel. When operating on multiple
projects, this can greatly speed things up. Across 1000 repos, it
goes from ~30sec to ~3sec with the default -j8.
Change-Id: I0dc62d704c022dd02cac0bd67fe79224f4e34095
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297484
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
+ superproject will be fetched into a directory with the name
“<remote name>-superproject.git” instead of the current
“superproject.git” folder.
+ Deleted _Clone method and added _Init method.
+ _Init method will do “git init --bare <remote>-superproject.git”.
It will create the folder and set up a bare repository in
<remote>-superproject.git folder.
+ _Fetch method, will pass <remote url>, <branch> arguments.
Moved the --filter argument from “git clone” to “git fetch”.
_Fetch method will execute the following command to fetch
superproject. Added --no-tags argument.
master: git fetch <remote url> --force --no-tags --filter blob:none
branch: git fetch <remote url> --force --no-tags --filter blob:none \
<branch>:<branch>
+ Performance improvements for aosp-master
++ repo init performance improved from 35 seconds to 17 seconds.
++ repo init --use-superproject is around 5 to 7 secsonds slower.
++ repo sync --use-superproject is around 3 to 4 minutes faster.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real 0m20.648s
user 0m8.046s
sys 0m3.271s
+ Without superproject
$ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main
real 0m13.078s
user 0m9.783s
sys 0m2.528s
$ time repo_dev sync -c -j32 --use-superproject
...
real 15m7.072s
user 110m7.216s
sys 20m17.559s
+ Without superproject
$ time repo sync -c -j32
...
real 19m25.644s
user 91m56.331s
sys 20m59.170s
Bug: [google internal] b/180492484
Bug: [google internal] b/179470886
Bug: [google internal] b/180124069
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
This is in preparation for adding jobs support. The nested function
is referenced in the options object which can't be pickled, so pull
it out into a static method instead.
Change-Id: I280ed2bf26390a0203925517a0d17c13053becaa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297983
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The idea for skipping some progress updates was to avoid spending
too much time on the progress bar itself. Unfortunately, for large
projects (100s if not 1000s) of repos, we get into the situation
with large/slow checkouts that we skip showing updates when a repo
finishes, but not enough repos finished to increase the percent.
Since the progress bar should be relatively fast compared to the
actual network & local dick operations, have it show an update
whenever the caller requests it. A test with ~1000 repos shows
that the progress bar in total adds <100ms.
Bug: https://crbug.com/gerrit/11293
Change-Id: I708a0c4bd923c59c7691a5b48ae33eb6fca4cd14
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297903
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When people switch to non-default branches, they sometimes want to
switch back to the default, but don't know the exact name for that
branch. Add a -b HEAD shortcut for that.
Change-Id: I090230da25f9f5a169608115d483f660f555624f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297843
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The XmlManifestTests class is getting to be large and we're only
adding more to it. Factor out the core logic into a new TestCase
so we can reuse it to better group more tests.
Change-Id: I5113444a4649a70ecfa8d83d3305959a953693f7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298222
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Now that we've converted the few users of this over to subprocess APIs,
we don't need this anymore. It's been a bit hairy to maintain across
different operating systems, so there's no desire to bring it back.
Using multiprocessing Pool to batch things has been working better in
general anyways.
Change-Id: I10769e96f60ecf27a80d8cc2aa0d1b199085252e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Now that these code paths are all synchronous, there's no need to run
our own poll loop to read & pass thru/save output. Delete all of that
and just let the subprocess module take care of it all.
Change-Id: Ic27fe71b6f964905cf280ce2b183bb7ee46f4a0d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297422
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This fixes intermingling of parallel jobs and simplifies the code
by switching to subprocess.run. This also provides stable output
in the order of projects by returning the output as a string that
the main loop outputs.
This drops support for interactive commands, but it's unclear if
anyone was relying on that, and the default behavior (-j2) made
that unreliable. If it turns out someone still wants this, we can
look at readding it.
Change-Id: I7555b4e7a15aad336667292614f730fb7a90bd26
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297482
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The status command runs a bunch of jobs in parallel, and each one
is responsible for writing to stdout directly. When running many
noisy jobs in parallel, output can get intermingled. Pass down a
StringIO buffer for writing to so we can return the entire output
as a string so the main job can handle displaying it. This fixes
interleaved output as well as making the output stable: we always
display results in the same project order now. By switching from
map to imap, this ends up not really adding any overhead.
Bug: https://crbug.com/gerrit/12231
Change-Id: Ic18b07c8074c046ff36e306eb8d392fb34fb6eca
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297242
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Extend the Command class to support adding the --jobs option to the
parser if the command declares it supports running in parallel. Also
pull the default value used for the number of local jobs into the
command module so local commands can share it.
Change-Id: I22b0f8d2cf69875013cec657b8e6c4385549ccac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297024
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Every use of GitCommand in the tree just calls Wait as soon as it's
instantiated. Move the bulk of the logic into the init path to make
the call synchronous to simplify. We'll cleanup the users of the
Wait API to follup commits -- having this split makes it easier to
track down regressions.
Change-Id: I1e8c519efa912da723749ff7663558c04c1f491c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297244
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Since the --manifest-url flag is always required when creating a new
checkout, allow the url to be specified via a positional argument.
This brings it a little closer to the `git clone` UI.
Change-Id: Iaf18e794ae2fa38b20579243d067205cae5fae2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297322
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
This is the only code in the tree that uses GitCommand asynchronously.
Rewrite it to use multiprocessing.Pool as it makes the code a little
bit easier to understand and simpler.
Change-Id: I3ed3b037f24aa1e9dfe8eec9ec21815cdda7678a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297143
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
These are the only users in the tree that process the output as it's
produced. All others capture all the output first and then process
the results. However, these functions still don't fully return until
it's finished processing, and these funcs are in turn used in other
synchronous code paths. So it's unclear whether anyone will notice
that it's slightly slower or less interactive. Let's try it out and
see if users report issues.
This will allow us to simplify our custom GitCommand code and move it
over to Python's subprocess.run, and will help fix interleaved output
when running multiple commands in parallel (e.g. `repo diff -j8`).
Change-Id: Ida16fafc47119d30a629a8783babeba890515de0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297144
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We only provide input to GitCommand in one place, so inline the logic
to be more synchronous and similar to subprocess.run. This makes the
code simpler and easier to understand.
Change-Id: Ibe498fedf608774bae1f807fc301eb67841c468b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297142
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Added --no-use-superproject to repo and init.py to disable use of
manifest superprojects.
Replaced the term "sha" with "commit id".
Added _GetBranch method to Superproject object.
Moved shared code between init and sync into SyncSuperproject function.
This function either does git clone or git fetch. If git fetch fails
it does git clone.
Changed Superproject constructor to accept manifest, repodir and branch
to avoid passing them to multiple functions as argument.
Changed functions that were raising exceptions to return either True
or False.
Saved the --use-superproject option in config as repo.superproject.
Updated internal-fs-layout.md document.
Updated the tests to work with the new API changes in Superproject.
Performance for the first time sync has improved from 20 minutes to
around 15 minutes.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo init took around 20 seconds longer because of cloning of superproject.
$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real 0m35.919s
user 0m21.947s
sys 0m8.977s
First run
$ time repo sync --use-superproject
...
real 16m41.982s
user 100m6.916s
sys 19m18.753s
No difference in repo sync time after the first run.
Bug: [google internal] b/179090734
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Changed "git pull" to "git fetch" as we are using --bare option. Used the
following command to fetch:
git fetch origin +refs/heads/*:refs/heads/* --prune
Pass --branch argument to Superproject's UpdateProjectsRevisionId function.
Returned False/None when directories don't exist instead of raise
GitError exception from _Fetch and _LsTree functions. The caller of Fetch
does Clone if Fetch fails.
Tested the code with the following commands.
$ ./run_tests -v
Tested the init and sync code by copying all the repo changes into my Android
AOSP checkout and running repo sync with --use-superproject option.
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222
Reviewed-by: Mike Frysinger <vapier@google.com>
After updating all project’s revsionIds with the SHAs from superproject,
write the updated manifest into superproject_override.xml file. Reload
that file for future Reloads. This file is created in exp-superproject
directory.
Moved most of the code that is superproject specific into
git_superproject.py and wrote test code.
If git pull fails, did a git clone of the superproject.
We saw performance gains for consecutive repo sync's. The time to sync
went down from around 120 secs to 40 secs when repo sync is executed
consecutively.
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
First run
$ time repo sync --use-superproject
...
real 21m3.745s
user 97m59.380s
sys 19m11.286s
After two consecutive sync runs
$ time repo sync -c -j8 --use-superproject
real 0m39.626s
user 0m29.937s
sys 0m38.155s
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
_CheckForImmutableRevision is used to see if repo can
skip fetching a project, but 'git rev-parse' with partial
clone does a data fetch to accomplish this.
Changed to use: 'git rev-list -1 --missing=allow-any <SHA>^0' which
checks the local ref without fetching from the server first.
Bug: [google internal] b/179477822
Testing:
- Unit tests
- Verified init/sync working on aosp-master
- Verified wwith a pinned manifest that local ref check works (no fetch)
Change-Id: If327b893c6658421f41df1f58c337f53b4c60ce6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296142
Reviewed-by: Dan Willemsen <dwillemsen@google.com>
Tested-by: Ian Kasprzak <iankaz@google.com>
+ This is without --depth option. This is done for reachability.
Server doesn't know what you know about in the history so it always
sends you the whole thing Which is very slow.
If we have the full history it can send you incremental update history
which is very small and fast.
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
.../WORKING_DIRECTORY$ repo sync --use-superproject
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I239de6d8f1c2ed6b4c69e7a78b8aa95338fa838c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/295362
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
.../WORKING_DIRECTORY$ repo sync --use-superproject
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I7e4b1e51ca1d18b836a5fa8d139a0765262ba500
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
.../WORKING_DIRECTORY$ repo sync --use-superproject
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Ieea31445ca89ba1d217e779ec7a7a2ebe81ac518
repo diffmanifests saves git commit messages in buf and uses default
utf-8 decoding, in some scenarios git commit message can itself contain
a non UTF-8 character due to a typo or incorrect i18n.commitEncoding.
e.g.
d354d9afe923 [PATCH] fbcon: don\xb4t call set_par() in fbcon_init() if vc_mode == KD_GRAPHICS
Convert the buf containing git commits to string if decoding to utf-8
encounters an error.
Signed-off-by: Gaurav Pathak <gaurav.pathak@pantacor.com>
Change-Id: If818562f0faaa5062c765fbea11dc0e1c86a24d7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294742
Reviewed-by: Mike Frysinger <vapier@google.com>
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.
Tested the code with the following commands.
$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,
.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []
Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
Currently we don't have a way for the checked out repo version to
declare the version of tools it needs before we start running it.
For somethings, like git, it's not a big deal as it can handle all
the asserts itself. But for things like Python, it's impossible
to reliably check before executing.
We're in this state now:
- we've been allowing Python 3.4, so the launcher accepts it
- the repo codebase starts using Python 3.6 features
- launcher tries to import us but hits syntax errors
- user is left confused and assuming new repo is broken because
they're seeing syntax errors
This scenario is playing out with old launchers that still accept
Python 2, and will continue to play out as time goes on and we want
to require newer versions of Python 3.
Lets create a JSON file to declare all these system requirements.
That file format is extremely stable, so loading & parsing from
even ancient versions of Python shouldn't be a problem. Then the
launcher can read these settings and check the system state before
attempting to execute any code. If the tools are too old, it can
clearly diagnose & display information to the user as to the real
problem (and not emit tracebacks or syntax errors).
We have a couple of different tool version checks already (git,
python, ssh) and can harmonize them in a single place.
This also allows us to assert a reverse dependency if the need
ever comes up: force the user to upgrade their `repo` launcher
before we'll let them run us. Even though the launcher warns
whenever a newer release is available, some users seem to ignore
that, or they don't use repo that often (on the scale of years),
and their upgrade jump is so dramatic that they fall back into
the syntax error pit.
Hopefully by the end of the year we can assume enough people
have upgraded their launcher such that we can delete all of the
duplicate version checks in the codebase. But until then, we'll
keep them to maintain coverage.
Change-Id: I5c12bbffdfd0a8ce978f39aa7f4674026fe9f4f8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293003
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
At most one superproject may be specified. It will be used
to specify the URL of superproject.
It would have 3 attributes: remote, name, and default.
Only "name" is required while the others have reasonable defaults.
<remote name="superproject-url" review="<url>" />
<superproject remote="superproject-url" name="platform/superproject"/>
TODO: This CL only implements the parsing logic and further work
will be in followup CLs.
Tested the code with the following commands.
$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v
Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I5b4bba02c8b59601c754cf6b5e4d07a1e16ce167
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292982
Reviewed-by: Mike Frysinger <vapier@google.com>
We've been warning about Python 3.4 for almost a year. This drops
support for these systems:
* Ubuntu Trusty: released Apr 2014, EOL Apr 2022
* Debian Jessie: released Apr 2015, EOL Jun 2020
So the min required distros would now be:
* Ubuntu Xenial: released Sep 2015 w/Python 3.5
* Debian Stretch: released Jun 2017 w/Python 3.6
I don't think we're quite ready to drop Python 3.5 which would affect
Ubuntu Xenial -- we'd have to update to Ubuntu Bionic from Apr 2018.
Let's see how much the community reacts to loss of Python 3.4 first.
Bug: https://crbug.com/gerrit/10418
Change-Id: Ib24a57818fdca49e23db53e1bdd1f4c76b4963f7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291502
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The codebase still supports Python 3.5, so allow use of that instead
of requiring Python 3.6+. Supporting this mode well is a bit tricky
as we want to first scan for newer versions before falling back to
older ones. And we have to avoid infinite loops in the process.
Bug: https://crbug.com/gerrit/13795
Change-Id: I47949a173899bfa9ab20d3fefa1a97bf002659f6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When checkout is done with Git worktrees then the HEAD in the
bare-git repositories point to the initialized default (e.g.
'refs/heads/master'). This default branch does not exist
locally and is not automatically created.
When a user now creates a branch in any git repository named
'master' then it is no longer possible to get rid of this branch,
neither is it possible to switch to another branch and switch
back to this master branch. Git concludes the 'master' branch is
already checked out (in the bare Git) and that results in a
lockdown of this master branch.
To repoduce this issue, run these commands in a repo tree
checked out with --worktree:
- git checkout master # assuming the remote repo has a master branch,
# a local tracking branch master is created here
- git checkout -b temp
- git checkout master # This one now fails
- git branch -d master # fails too
The failure is caused by Git assuming the master branch is checked out
by the bare git repository since HEAD is pointing towards it.
To workaround this, we always detach HEAD in the bare-git when
syncing. We don't need it to point to a ref in general, but we
would like it to be valid so git tools "just work" if they're run
in here.
Signed-off-by: Remy Bohmer <oss@bohmer.net>
Change-Id: I15c96604363c41f0d01c42f533174393097daeb5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290985
Reviewed-by: Mike Frysinger <vapier@google.com>
Spread the operation of querying which local branches exist across a
pool of processes and build the name map of projects -> branches as
these tasks finish rather than blocking on the entire query. The search
operations are submitted in batches to reduce the overhead of interprocess
communication. The `chunksize` argument used to control this batch size
was selected by incrementing through powers of two until it stopped being
faster.
Change-Id: Ie3d7f799ee8e83e5058536caf53e2979175408b7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291342
Tested-by: Chris Mcdonald <cjmcdonald@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Rather than pull the client dir out to construct the manifest
filename which the manifest itself already has, pull the filename
out and use that.
Change-Id: I33991084dcb3205f819bb841084e3c48d6ccb284
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291264
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When updating the tracking ref to whatever the user requested,
make sure we reset state completely rather than trying to update
the ref to it. This avoids confusing git as to the current state
of the tree, and is more inline with user intentions: if they made
a local change to the checkout, but ran repo init with a specific
rev, we shouldn't stay wedged forever until they manually clean it
all up.
Bug: https://crbug.com/gerrit/12801
Change-Id: Ieba8d9c15781b4d0649bf01c7460694da63387b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290923
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We allow project.groups to be whitespace or comma delimited, but
repo-hooks.enabled-list is only whitespace delimited. This hasn't
been a big deal as it's only ever had one valid value, but if we
want to add more, we should harmonize these a bit.
Refactor the groups method to be more generic, and run the enabled-
list attribute through it. Then add missing docs for it.
Change-Id: Iaa96a0faa9c4a68b313b49336751831b73bf855d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290743
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Tests worked fine if init.defaultBranch main was used,
but failed due to git branch reasons if master was still used.
Since we can only use init.defaultBranch if git version >= 2.28,
I also went with a template dir HEAD main tweak if lower so tests
now pass regardless of client git default branch and version.
Test: Ran tests with ~/.gitconfig:init.defaultBranch=master
Test: Ran tests with ~/.gitconfig:init.defaultBranch=main
Test: Ran tests for both code branches of git require
Change-Id: I49fa1e4ae45b8aec16a093132ee9fa466cbc11ec
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290404
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Attrib groups can now be added to manifest include, thus
all projects in an included manifest file can easily be tagged
with a group without modifying all projects in that manifest file.
Include groups will add and recurse, meaning included manifest
projects will carry all parent includes. Intentionally, no support
added for group remove, to keep complexity down.
Group handling for projects is untouched, meaning a group set on
a project will still append to whatever was or was not inherited
in parent manifest includes, resulting in union of groups inherited
and set for the project itself.
Test: manual multi-level manifest include structure, in serial and parallel,
with different groups set on init
Test: added unit tests to cover the inheritance
Change-Id: Id2229aa6fd78d355ba598cc15c701b2ee71e5c6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/283587
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
The knowledge about running hooks and all its exception handling
is scattered over multiple files. This makes the code harder
to read, but also it requires duplication of logic in case
other RepoHooks are added to different commands.
This refactoring also creates uniform behavior of the hooks
across multiple commands and it guarantees the re-use of the same
arguments on all of them.
Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: Ia4d90eab429e4af00943306e89faec8db35ba29d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/277562
Tested-by: Remy Bohmer <oss@bohmer.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
When intializing a new repo with the --reference option on Windows 10
the objects/info/alternates in each git repository is created with
Windows line endings (\r\n), leading to the following error:
error: object directory C:/<PATH_TO_MIRROR>/<REPO_NAME>.git/objects?
does not exist; check .git/objects/info/alternates
This can be fixed by simply using unix line endings on both
Windows and unix platforms.
Reported-by: Francisco Javier Alvarez Garcia <javier.alvarez.garcia.17@gmail.com>
Follow-up-from: I268fe029ede68802c21037b0f2ae8a95afb85e48
Bug: https://crbug.com/gerrit/13208
Change-Id: I6da60c4ca957778b3c42ab6b9ad85c40483f0042
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289431
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <oss@bohmer.net>
Worktree .git and gitdir reference files are written by Git with
Unix line ending, even on Windows & macOS. The conversion to
relative paths makes these files end with DOS line endings in
Windows. The Git integration in Visual Studio 2019 cannot deal
with these DOS line endings and considers these worktrees invalid.
Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: I088cfd994f3cc31db4e0ca7791fa0a4ee3ac222f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289310
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <linux@bohmer.net>
We conflate the manifest & parsing logic with the management of the
repo client checkout in a single class. This makes testing just one
part (the manifest parsing) hard as it requires a full checkout too.
Start splitting the two apart into separate classes to make it easy
to reason about & test.
Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Using sha1 manifest, project's revisionId is initialized
first by the manifest.
An update of a projet revision by extend-project node does
not apply to the revisionId which is therefore kept to the
initial value.
Resets revisionId value when revision is updated by an
extend-project node.
Change-Id: I873af283890cebaeaabde966f04b125642af929f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/275715
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Miguel Gaio <miguel.gaio@renault.com>
In the .repo discovery loop
while curdir != '/' and curdir != olddir:
... break if we found .repo ...
olddir = curdir
curdir = os.path.dirname(curdir)
the "while" condition is meant to avoid searching forever if we do not
find .repo before reaching the top-level directory of the filesystem.
For that purpose, the first half of the condition is redundant; once
we reach "/", the parent directory will be "/" again and the curdir !=
olddir check would suffice to terminate the search. Simplify by
removing the redundant first half of the check.
Noticed by code inspection. The first half of the check was retained
when introducing the second half in df14a70c ("Make path references OS
independent", 2011-01-09), in an excess of caution.
This also improves consistency a little: if I start with curdir =
'/home/me', then with the redundant check in place we search
/home/me
/home
before hitting / and giving up. On Windows, if I start with
'c:/users/me', then we search
c:/users/me
c:/users
c:/
before hitting a repetition and giving up. Fortunately it is not
common for people to set up repo clients at the top level of
filesystems, but consistently following the latter behavior should
make debugging a little easier in case it comes up.
Link: https://gerrit-review.googlesource.com/id/Ib9e830e3b9adfb1c4e56f3bcfba4746c401fb84f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/286002
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jonathan Nieder <jrn@google.com>
Instead of hardcoding "master" as our default, use the remote server's
default branch instead. For most people, this should be the same as
"master" already. For projects moving to "main", it means we'll use
the new name automatically rather than forcing people to use -b main.
For repositories that never set up a default HEAD, we should still use
the historical "master" default.
Bug: https://crbug.com/gerrit/13339
Change-Id: I4117c81a760c9495f98dbb1111a3e6c127f45eba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280799
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This change increases the speed of the command with parallelization with
processes. The parallelization with threads doesn't work well, and
increasing the number of jobs to many (8 threads ~) didn't increase the speed.
Possibly, the global interpreter lock of Python affects.
Bug: https://crbug.com/gerrit/12389
Change-Id: Icbe5df8ba037dd91422b96f4e43708068d7be924
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279936
Tested-by: Kimiyuki Onaka <kimiyuki@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If you pass args to `repo init` when first creating a checkout, the
repo launcher throws an error. But the init subcommand that runs in
an existing checkout silently ignores them. Throw a proper error.
Change-Id: I433bfcc73902d25f6b6a2974e77f6a977a75ed16
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279696
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the manifest uses a trailing slash on the name attribute, repo
will construct bad internal filesystem paths which confuses tools
later on.
For example, this manifest entry:
<project name="aosp/platform/system/libhidl/" ...
will cause repo to use paths like:
.repo/project-objects/aosp/platform/system/libhidl/.git/
when it really should be using:
.repo/project-objects/aosp/platform/system/libhidl.git
Apply the normalization when we construct the local filesystem paths
as we cannot guarantee that the remote URL constructed from these
will behave the same. A server might really want:
https://example.com/aosp/platform/system/libhidl/
and would throw an error if we instead tried to fetch:
https://example.com/aosp/platform/system/libhidl
Unfortunately, any existing repo client checkouts that use such a
manifest will hit a one-time sync error as the internal git location
has changed. I'm not sure there's a way to cleanly migrate that.
Bug: https://crbug.com/1086043
Change-Id: I30bea0ffd23e478de89a035f408055e48a102658
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268742
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
For large projects, clone bundle is useful because it provided a way to
efficiently transfer a large portion of git objects through CDN, without
needing to interact with git server. However, with partial clones, the
intention is to not download most of the objects, so the use of clone
bundles would defeat the space savings normally seen with partial
clones, as they are downloaded before the first fetch.
A new option, --clone-bundle is added to override this behavior.
Add a new repo.clonebundle variable which remembers the choice if
explicitly given from command line at repo init.
Change-Id: I03638474af303a82af34579e16cd4700690b5f43
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268452
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
If the dest-branch attribute is set in the project manifest, then
we need to push to that branch. Previously, we would unconditionally
pre-pend the refs/heads prefix to it. The dest-branch attribute is
allowed to be a ref expression though, so it may already have it.
Simple fix is to check if it already has the prefix before adding it.
Bug: crbug.com/gerrit/12770
Change-Id: I45d6107ed6cf305cf223023b0ddad4278f7f4146
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268152
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
When running repo info -d an error would be thrown saying:
fatal: bad revision 'refs/remotes/m/refs/heads/master..'
Using the short branch name here instead, like 'refs/remotes/m/master..'
resolves this issue.
Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: I50ea92c45c011b2c3e3a63803decb88e7837a380
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/266578
Reviewed-by: Mike Frysinger <vapier@google.com>
The intention of the check is to verify whether the target
file name contains a wild card. The code, however, assumes
that if the file is non-existent - it contains a wild card.
This has the side effect that a target file that does not
exist at the moment of the check is considered to contain a
wild card, this leads itself to softlink not being created.
Change-Id: I4e4cd7b5e1b8ce2e4b2edc9abf5a1147cd86242f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/265736
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Angel Petkov <apetkov86@gmail.com>
Git config files may have an include tag pointing to another file.
The included file is not parsed unless “git config --list” is
explicitly told to follow includes by adding the argument ”--includes”.
This change add the "--includes" when parsing the global gitconfig file.
Change-Id: I892c9a3a748754c1eb8c9e220578305ca5850dd5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/264759
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Ulrik Laurén <ulrik.lauren@gmail.com>
exit if no repo_main can be found right before executing the command.
This happens for instance when 'repo init' is run on root path
(for example in a container). Without this counter measure the tool
will crash at exec_command with
TypeError: sequence item 1: expected str instance, NoneType found
Change-Id: Ia8480cfe2151c3b35c9572789ad8cb619288cce1
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263457
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
The launcher already raised itself up to use Python 3 on the fly, and
the main.py script uses a plain `python` shebang. So make sure we use
the active interpreter when re-execing ourselves to avoid falling back
down to Python 2 (which then triggers warnings).
Change-Id: Ic53c07dead3bc9233e4089a0a422f83bb5ac2f91
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263272
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
When generating a revision locked manifest, we need to know what
ref to push changes to when doing 'repo upload'. This information
is lost when we lock the revision attribute to a particular commit
hash, so we need to expose it through the dest-branch attribute.
Bug: https://crbug.com/1005103
Test: manual execution
Change-Id: Ib31fd77ad8c9379759c4181dac1ea97de43eec35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263572
Tested-by: Sean McAllister <smcallis@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Recent changes in ChromeOS Infra to ensure we're reading from
snapshot manifests properly have exposed several bugs in our
assumptions about manifest files. Mainly that the revision field
for a project does _not_ have to refer to a ref, it can just be
a commit hash.
Several places assume that the revision field can be parsed as a
ref to get the branch the project is on, which isn't true. To fix
this we need to be able to look at the upstream and dest-branch
attributes of the repo, so we expose them through the environment
variables set in `repo forall`.
Test: manual 'repo forall' run
Bug: https://crbug.com/1032441
Change-Id: I2c039e0f4b2e0f430602932e91b782edb6f9b1ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263132
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
The generated socket path can be too long, if your FQDN is very long...
Typical error message from ssh client:
unix_listener: path "/tmp/ssh-fqduawon/master-USER@HOST:PORT.qfCZ51OAZgTzVLbg" too long for Unix domain socket
Use a hashed version instead, to keep within the socket file path limit.
This requires OpenSSH_6.7p1, or later.
Change-Id: Ia4bb9ae8aac6c4ee31d5a458f917f3753f40001b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255632
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Anders Björklund <anders.bjorklund.2@volvocars.com>
The SetupGnuPG test tries to test the full setup, including the
creation of the directories. In order to do that, it create a
temporary directory, and redefines the home_dot_repo to point there.
When a home_dot_repo directory does not exist, it should be created.
The gpg_dir, which should exist inside home_dot_repo, also needs to be
created if it does not exist. However, since the gpg_dir path is
relative to home_dot_repo, once we redefine one, we need to redifine
the other.
The failure of this test might have gone unnoticed so far, since in
only fails if you do not have a ~/.repoconfig/gnupg/ on the
environment you are running the tests on.
Change-Id: Ic69d59e56137eea43349a61b5cf81f215c6a7f9a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/262573
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Marcos Marado <mindboosternoori@gmail.com>
In older versions of Gerrit the Change-Id field was inserted at the
start of the trailers. Commit 68296f71804feab2e0ae18ae33f834a8a41621e4
simplified the trailers code by using git trailers instead of custom
code but now inserts Change-Id at the end of the trailers section.
A consequence of this is that folks who sign-off their commits using
`git commit -s` now has the sign-off appear first followed by
Change-Id. If the user then runs `git commit -s --amend` to update
the change because the Sign-off-by line is not last, git inserts
a 2nd duplicate Signed-off-by line.
This patch simply restores the previous behaviour of the Gerrit
commit-msg hook where Change-Id would be inserted before the
Sign-off-by line to avoid this issue.
Backported from [1] by Thanh Ha.
[1] https://gerrit-review.googlesource.com/c/gerrit/+/262072
Bug: https://crbug.com/12546
Change-Id: I1406c763a3935761247f6771f55e02367f698e6e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/262352
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
When developing repo itself, it helps to run repo directly out of it
and to run bisection tools. The current _SetDefaultsTo logic fails
in that situation though as it wants a branch, but the source isn't
checked out to one. Now that we support tracking commits via the
--repo-rev setting, fall back to using the current HEAD commit.
Change-Id: I37d79fd9f7bea87d212421ebed6c8267ec95145f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260192
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We respect this option when running the first `repo init`, but then
silently ignore it once the initial sync is done. Make sure users
are able to change things on the fly.
We refactor the wrapper API to allow reuse between the two init's.
Bug: https://crbug.com/gerrit/11045
Change-Id: Icb89a8cddca32f39a760a6283152457810b2392d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260032
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
While the help/usage suggested that revisions would work, they never
actually did, and just throw confusing errors. Now that we warn if
the checkout isn't tracking a branch, allow people to specify commits
or tags explicitly. Hopefully our nags will be sufficient to keep
most people on the right path.
Bug: https://crbug.com/gerrit/11045
Change-Id: I6ea32c677912185f55ab20faaa23c6c0a4c483b3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259492
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We gracefully handle cherry-pick errors, but none of the others
which means people get confusing Python tracebacks. Move the
main logic in a single GitError try block so we can show pretty
error messages for all of them.
Change-Id: I52cdf6468d21a98de7f65b86d5267b3caabd5af8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259854
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The git cherry-pick already supports this, so plumb the existing repo
option down. Otherwise it's confusing when people use -c --ff and it
doesn't use that behavior.
Change-Id: Id68932ffa09204bb30b92a21aff185c00394a520
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259852
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When the launcher handles the init subcommand, it takes care of
setting the repo url & branch itself when cloning. So we don't
need to pass them down to the checked out init subcommand.
Further, the init subcommand has never actually done anything
with those options, so there's no point in passing them.
We'll be changing the latter behavior so that init will reset
the url/branch when specified with an existing repo checkout
which means passing them through adds overhead: the launcher
will checkout to the right value, then chain to the sub-init
which will then reset the checkout to the same value.
Bug: https://crbug.com/gerrit/11045
Change-Id: Ia2a4ab9d86febc470aea4abd73d75bb10e848b56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259312
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The current subcmds design has singletons in all_commands. This isn't
exactly unusual, but the fact that our main & help subcommand will then
attach members to the classes before invoking them is. This makes it
hard to keep track of what members a command has access to, and the two
code paths (main & help) attach different members depending on what APIs
they then invoke.
Lets pull this back a step by storing classes in all_commands and leave
the instantiation step to when they're used. This doesn't fully clean
up the confusion, but gets us closer.
Change-Id: I6a768ff97fe541e6f3228358dba04ed66c4b070a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259154
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The branch->branches alias is setup in the main module when that
really belongs in the existing all_commands setup.
For help, rather than monkey patching all_commands to the class,
switch it to use the state directly from the module. This makes
it a bit more obvious where it's coming from rather than this one
subcommand having a |commands| member added externally to it.
Change-Id: I0200def09bf4774cad8012af0f4ae60ea3089dc0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259153
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
When we download git sources, we get a progress bar (good) and we get
a dump of all the refs we downloaded (bad) as it can easily be 100+ if
not 1000+ depending on the project (for each git repo!). Lets rework
the output behavior so that:
* quiet: Only errors.
* default: Progress bars (if on a tty).
* verbose: Full output (progress bars & downloaded refs).
Bug: https://crbug.com/gerrit/11293
Change-Id: I87a380075e79de6805f91095876dd1b37d32873a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256456
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
We've been overly lenient with boolean parsing by ignoring invalid
values as "false" even if the user didn't intend that. Turn all
unknown values into warnings to avoid breaking existing manifests,
and unify the parsing logic in a helper to simplify.
We've been stricter about numbers, but still copying & pasting
inconsistent code. Add a helper for this too. For out of range
sync-j numbers (i.e. less than 1), throw a warning for now, but
mark it for future hard failures.
Change-Id: I924162b8036e6a5f1e31b6ebb24b6a26ed63712d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256457
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Since most ref namespaces are shared among all worktrees, trying to
set the pseudo m/<branch> in the common git repo ends up clobbering
each other when using shared checkouts. For example, in CrOS:
<project path="src/third_party/kernel/v3.8"
name="chromiumos/third_party/kernel"
revision="refs/heads/chromeos-3.8" />
<project path="src/third_party/kernel/v3.10"
name="chromiumos/third_party/kernel"
revision="refs/heads/chromeos-3.10" />
Trying to set m/master in chromiumos/third_party/kernel.git/ will
keep clobbering the other.
Instead, when using git worktrees, lets set the m/ pseudo ref to
point into the refs/worktree/ namespace which is unique to each
git worktree. So we have in the common dir:
chromiumos/third_party/kernel.git/:
refs/remotes/m/master:
ref: refs/worktree/m/master
And then in each worktree we point refs/worktree/m/master to the
respective manifest revision expression. Now people can use the
m/master in each git worktree and have it resolve to the right
commit for that worktree.
Bug: https://crbug.com/gerrit/12404
Change-Id: I78814bdd5dd67bb13218c4c6ccd64f8a15dd0a52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256952
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Trying to use the config state when the git tree hasn't yet been
created hits bad side effects. Add a check to avoid probing the
config logic during the first run. It's not clear what's going
wrong at the lower layers, but this gets us back to the behavior
before we added worktree support, so lets settle the status quo.
Bug: https://crbug.com/gerrit/12387
Change-Id: I85b56797455f3c2e249d02c18496e060be05501d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256592
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We produce some simple "Get" messages that aren't super clear as to
what they're doing, especially for people not familiar with repo.
Rephrase these to explicitly state the thing we're doing so it's
clear why we're downloading a particular source.
Bug: https://crbug.com/gerrit/11293
Change-Id: I0749504f17c5385c6c65274a274e0ae25b117413
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256455
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Git likes to create .git files with read-only permissions which makes
it difficult to open+truncate+write in situ under Windows. Delete it
before we write the file content to workaround.
Change-Id: I3effd96525f0dfe0b90e298b6bf0b856ea26aa03
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256412
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Changing this to a file instead of using a symlink serves two purposes:
* We can insert some comments & doc links to help users learn what this
is for, discover relevant documentation, and to discourage them from
modifying things.
* Windows requires Administrator access to use symlinks. With this
last change, Windows users can get repo client checkouts with the new
--worktree option and not need symlinks anywhere at all. Which means
they no longer need to be an Administrator in order to `repo sync`.
Change-Id: I9bc46824fd8d4b0f446ba84bd764994ca1e597e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256313
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Windows requires Administrator access to create symlinks. We can
mitigate this a bit by falling back to hardlinks as those may be
created by any user on the system. Do this with the git hooks as
these are supposed to be internal only and people shouldn't be
modifying them. If they do, they'll have to delink first. This
seems worth it to allow repo usage without extra privileges.
Change-Id: I996ea9c9238f7bd7d27d1d9b1f2786593bf75ef7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256312
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We also need to check more things in the manifest/project handlers,
and use platform_utils in a few places to address Windows behavior.
Drop Python 2.7 from Windows testing as it definitely doesn't work
and we won't be fixing it.
Change-Id: I83d00ee9f1612312bb3f7147cb9535fc61268245
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256113
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We're using this for git worktrees because it handles the .git file
format, but it should work for all flows. Unify to simplify. This
also fixes the worktree logic which duplicated .git/config settings.
Change-Id: Ie3af2e206710859dccfc376b3593f415d6830738
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256034
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Switch the copyright holder to "The Android Open Source Project" to
match all the other source files in the tree, and move it to the top
of the file to match everything else we do.
Change-Id: Ie15d8e2bc004a626e45f715271deeaf3919dc44a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256235
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
- E129 visually indented line with same indent as next logical line
- E125 continuation line with same indent as next logical line
Fixed automatically by:
autopep8 --in-place --select E125,E129 subcmds/sync.py
Change-Id: Ia2f82f443e1e6a23ba22c6f9849c8485405aed0e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256092
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
This allows people to write ~/.repoconfig/config akin to ~/.gitconfig
and .repo/config akin to .git/config. This allows us to add settings
specific to repo without mixing up git, and to persist in general.
Change-Id: I1c6fbe31e63fb8ce26aa85335349c6ae5b1712c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255832
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Since deleting a source checkout involves a good bit of internal
knowledge of .repo/, move the DeleteProject helper out of the sync
code and into the Project class itself. This allows us to add git
worktree support to it so we can unlock/unlink project checkouts.
Change-Id: If9af8bd4a9c7e29743827d8166bc3db81547ca50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256072
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When using extensions, make sure we set the git repo format version
so git knows to check the extension compatibility. We can add a
helper to the Project API to simplify this and make it foolproof.
Change-Id: I9ab6c32d92fe2b8e5df6e2b080ca71556332e909
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256035
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
This provides initial support for using git worktrees internally
instead of our own ad-hoc symlink tree. It's been lightly tested
which is why it's not currently exposed via --help.
When people opt-in to worktrees in an existing repo client checkout,
no projects are migrated. Instead, only new projects will use the
worktree method. This allows for limited testing/opting in without
having to completely blow things away or get a second checkout.
Bug: https://crbug.com/gerrit/11486
Change-Id: Ic3ff891b30940a6ba497b406b2a387e0a8517ed8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254075
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Different Python & OS versions have different environ behavior wrt
accepted types & encoding. Since we're migrating to be Python 3 only,
lets change our code to assume strings always work as that's what the
newer Python 3 does. This will fail under Python 2 for some env vars,
mostly on Windows, but the effort of maintaining shim layers that can
handle these edge cases isn't worth it when we're dropping that code.
We leave the logic in the `repo` launcher for now as it is simple, and
we want it to be able to switch versions a bit longer than the rest of
the tree.
Here's the support table:
| *NIX | Windows |
Python 2 | ASCII string | str or bytes, not unicode |
Python 3 | str or bytes | str only |
Windows uses strings natively in its environment all the time. But it
doesn't allow unicode strings under Python 2, so we have to encode.
Python 2 on *NIX is funky in that it always lowers to ASCII, so we had
to manually encode to avoid errors regardless of unicode or str.
Python 3 on Windows & *NIX will accept strings. *NIX will also accept
bytes but Windows will not.
Bug: https://crbug.com/gerrit/12145
Change-Id: I3cf8f95a06902754ea1f08ad4b28503f7063531b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248972
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
For people used to running `repo xxx --help`, they might not realize
that there are detailed man pages behind `repo help xxx`. Add a note
to all --help commands to improve discoverability.
Change-Id: I84af58aa0514cc7ead185f6c2534a8f88e09a236
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255853
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The pager setting is tristate (where None means "auto"),
so make sure we still handle that setting.
Change-Id: I89fe352572dd15922c61e3bb65ac33f847d01ee0
Test: `repo help upload` triggers the pager
Test: `repo -p help upload` triggers the pager
Test: `repo --no-pager help upload` doesn't trigger the pager
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255852
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Named Temporary file defaults to mode 'w+b' which causes repo sync to
fail. By opening the tmpcookiefile in PersistentTransport.request as
writable, we are able to run sync successfully.
Bug: https://crbug.com/gerrit/12370
Test: Ran smartsync successfully
Change-Id: I01ddf915fc30eb3ff0e4d440a6f1aa261c63e88d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255692
Tested-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
If an older launcher script is used with newer repo source tree, we
might be issuing python version warnings. Plus, we want to be able
to roll Python version requirements independently of the launcher.
Add some version checking here too.
Change-Id: Ia35fc821f93c429296bdf5fd578276fef796b649
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255592
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This allows us to control sync output better by having three levels
of output: quiet (only errors), default (progress bars), verbose (all
the things). For now, we just put the chatty "already have persistent
ref" message behind the verbose level.
Bug: https://crbug.com/gerrit/11293
Change-Id: Ia61333fd8085719f3e99edb7b466cdb04031b67f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255414
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Includes the following commits (redacted to those that are relevant):
da300bd9bd8 - Do not create a change id if gerrit.createChangeId == false
731eb42b8ae - Do not strip out "-- >8 --" comment in commit-msg hook
627d07c2bfc - Handle messages with only comments in the commit-msg hook
68296f71804 - Simplify the hook script using git-interpret-trailers
Change-Id: I7a82836495427df3c5437ba88a9576b47629065f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255393
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
This gives us a bit of feedback by running our testsuite on Linux,
macOS, and Windows platforms. While Linux & macOS are passing,
Windows fails some of them. We can figure that out later. This
is better than what we have now which is manual one-offs.
Change-Id: I9d2d644be97ec76645db0bc15739e7679310a647
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255314
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We need to use the path separators provided by the python library,
and we need to set the git env vars so the name is always known.
Not all tests pass, but at least the basic frameworks work now.
Change-Id: Icea67098a8d7d58bbf918c78325681cf12a2e5f2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255313
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
- flake8 is a wrapper around pyflakes, so it's redundant to mention
both of them. Roll the explicit sections about coding errors and
coding style violations into a single section.
- After recent cleanups the project now has zero warnings or errors
from flake8. Reword the requirements so that it is now mandatory
to not introduce new warnings.
- Expand the section on suppression of warnings to differentiate
between suppressing inline individually and globally suppressing
for the whole project.
- Properly capitalize "Python Style Guide".
Change-Id: I4b333d013e985db252873441b16cb719ed5be5b5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255040
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
If programs emit non-UTF-8 output, we currently throw a fatal error.
We largely only care about the exit status of programs, and even the
output we do parse is a bit minimal. Lets make it into a warning and
mangle the invalid bytes into U+FFFD. This should complain enough to
annoy but not to break when it's not necessary.
Bug: https://crbug.com/gerrit/12337#c2
Change-Id: Idbc94f19ff4d84d2e47e01960dd17d5b492d4a8a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255272
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
All of the instances of this are related to Python 2 names that
don't exist in Python 3, and the warnings are raised when running
flake8 on Python 3.
All of these will go away once we completely remove support for
Python 2, so just suppress them inline. We don't globally suppress
the check so that we will still see legitimate errors if/when they
occur in new code.
Change-Id: Iccf955f50abfc9f83b371fc0af6cceb51037456f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255039
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Some versions of gpg on Windows mishandle native paths with homedir.
It manifests itself like:
gpg: keybox 'C:\Users\.../.repoconfig\gnupg/pubring.kbx' created
gpg: C:\Users\.../.repoconfig\gnupg/trustdb.gpg: trustdb created
gpg: key 16530D5E920F5C65: public key "Repo Maintainer <repo@android.kernel.org>" imported
gpg: can't connect to the agent: Invalid value passed to IPC
gpg: Total number processed: 1
gpg: imported: 1
fatal: registering repo maintainer keys failed
It seems gpg (at least version 2.2.17) needs paths to be specified
in cygwin form (e.g. "/c/Users/.../.repoconfig/gnupg") otherwise
it fails to talk to its own processes. We can work around this
with a minor trick: we cd to the right path and then invoke gpg
with --homedir . and let gpg itself resolve . to whatever form it
really wants.
This is a bit hacky, but we don't control gpg, and this allows us
to avoid having to muck with the environment. Since --homedir has
been around since at least gpg-1.4.x from 2004, backwards compat
shouldn't be an issue.
While we're here, touch up the output a bit: there's no need to
dump all the chatty gpg output if things don't fail, so always
swallow the output. If things do fail, our exception handler
takes care of dumping the full stdout & stderr.
Change-Id: I74ab98e1e61e95318fda6faf57c6a8699f775935
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255120
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We added support for `repo init -c` to main.py, but not to the
launcher, so the -c option only works after the first init has
run which kind of defeats its purpose. Rework the parser setup
so that we can tell it whether it's for "init" or "gitc-init"
and then add the -c option in the same way we do in main.py.
This has the benefit of getting the parser entirely out of the
module scope which makes it a lot easier to reason about, and
it means we can write some unittests.
Change-Id: Icbc2ec3aceb938d5a8f941d5fbce1548553dc5f7
Test: repo help init
Test: repo help gitc-init
Test: repo init -u https://android.googlesource.com/platform/manifest -c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255113
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We've been using a DSA/1024 key to sign our tags. Time to update to
something a bit newer. We'll include RSA & ECC keys, but only use
RSA keys initially for backwards compatibility and see how it goes
with our user base.
Change-Id: I683c97b6fbd860f220ed4ddc7b21f07db279a916
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255112
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We can't rely on subprocess.run yet as that requires Python 3.6,
but we can clean up the code we have with some ad-hoc replacement.
This unifies all the inconsistent subprocess.Popen usage we have.
Change-Id: I56af40a3df988ee47b299105d692ff419d07ad6b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254754
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
- W503 line break before binary operator
- W504 line break after binary operator
There doesn't seem to be a nice way of fixing all of these without
replacing W503 with W504 or vice-versa, or unwrapping them resulting
in excessively long lines. Let's just suppress them.
Change-Id: I7846d0124054f58e1cb480d4459cd9c86b737a50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254608
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
flake8 reports:
E722 do not use bare 'except'
Replace them with 'except Exception' per [1] which says:
Bare except will catch exceptions you almost certainly don't want
to catch, including KeyboardInterrupt (the user hitting Ctrl+C) and
Python-raised errors like SystemExit
If you don't have a specific exception you're expecting, at least
except Exception, which is the base type for all "Regular" exceptions.
[1] https://stackoverflow.com/a/54948581
Change-Id: Ic555ea9482645899f5b04040ddb6b24eadbf9062
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254606
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E301,E302,E303,E305,E306
Manually fix issues in project.py caused by misuse of block comments.
Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E121 continuation line under-indented for hanging indent
- E122 continuation line missing indentation or outdented
- E125 continuation line with same indent as next logical line
- E126 continuation line over-indented for hanging indent
- E127 continuation line over-indented for visual indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E121,E122,E125,E126,E127,E128,E129,E131
Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
The Google style guide for python [1] says the maximum line length
should be 80, but there are several lines in the code base that
exceed it:
git ls-files | grep py$ | xargs flake8 | grep E501 | wc -l
64
I don't think it's worth going through and re-wrapping all those,
so just increase the limit to 100 which seems to be a reasonable
compromise:
git ls-files | grep py$ | xargs flake8 | grep E501 | wc -l
6
Leave the re-rewrapping of those lines for a follow-up commit,
though.
[1] http://google.github.io/styleguide/pyguide.html#32-line-length
Change-Id: Ia37c34301163431fd1fb4fb6697a4a482d6be077
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254595
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
We have a few files that we optionally symlink from the work tree
.git/ to the .repo/projects/ path. If they don't exist when we
first initialize, then we skip creating symlinks. If the files
are created later on under the work tree .git/, repo gets upset.
This can happen with the packed-refs file: if we don't have any
packed refs initially, we don't symlink it. But if git tries to
pack refs later on and creates the file, the project gets wedged.
We could create an empty file initially and then symlink it, but
for some files, it's not clear we want to always do that (e.g.
the .git/shallow setting). Instead, lets make handling of these
paths more dynamic. If they show up later on in the work tree
.git/ only, we'll take care of relocating & symlinking. This
also makes repo a little more robust and autorecovers incase a
path goes missing in one of the dirs.
Ideally we wouldn't monkey around at all here, but considering
the only option we give to users currently is to blow things
away with --force-sync, this seems a bit better.
Bug: https://crbug.com/gerrit/12324
Change-Id: Ia6960f1896ac6d890c762d7d053684a1c6ab2c87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254632
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When upload hooks fail, people are forced to use --no-verify to upload
CLs anyways. When projects have flaky hooks, this trains people to
always use that option. This is obviously bad: hooks might get fixed,
or some of the hooks are always good & people should review.
Lets add an --ignore-hooks option. This still runs the hooks, but any
failures will be ignored and allow the user to upload anyways.
Bug: https://crbug.com/gerrit/12230
Change-Id: Ide2ac8a40a656bfcd6aae20c3ce8118e06bf909b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254452
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
flake8 reports:
F811 redefinition of unused 'test_src_block_dir' from line 259
which is caused by having two methods with the same name. Rename
them both to better desribe their purpose.
Change-Id: If7612a42001776d71bb1a6a80fc631d3d262e6ce
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254449
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
We were perhaps a bit too hasty to jump to git-2.10. Existing LTS
releases of Ubuntu are quite old still: Trusty has 1.9 while Xenial
has 2.5. While we plan on dropping support for those eventually as
we migrate to Python 3.6, we don't need to be so strict just yet on
the git versions.
We also want to disconnect the version the repo launcher requires
from the version the rest of the source tree requires. The repo
launcher doesn't need as many features, and being flexible there
allows us more freedom to upgrade & rollback as needed.
So we'll allow git-1.7 again, but start warning on any users older
than git-1.9. This aligns better with existing LTS releases, and
gives users a chance to start upgrading before we cut them off.
Change-Id: I140305dd8e42c9719c84e2aee0dc6a5c5b18da25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254573
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Going purely on upstream package release cycles doesn't tell the whole
story: a lot of people run LTS distros which will have older versions
of software we want to support.
Build out a table for us to quickly reference when making decisions as
to what versions of git/python we want to support, and when we can drop
them. This will also help to refer users to as why we made a specific
decision that might be affecting them.
Change-Id: I7aea24bbefd50e358aeacf11e8c15a346c8fb8a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254572
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Some automatic git operations will prune objects on us, and not just
the gc step. Normally we don't care, but with shared projects, we
will have multiple git checkouts with refs that the others cannot
see, but with a shared object dir. Any pruning of objects based on
refs in just one repo can easily break the others.
git-2.7.0 introduced a preciousObjects setting which tells git to
never prune objects for this exact scenario: there might be refs in
some location that git is unable to see.
Change-Id: I781de27c5bbe1d4c70f0187566141c9cce088bd8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254392
Reviewed-by: Nasser Grainawi <nasser@codeaurora.org>
Reviewed-by: David Riley <davidriley@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Execution of 'repo forall -p -c' doesn't work with Py3 and ends up
with an error:
Got an error, terminating the pool: TypeError: can only concatenate
str (not "bytes") to str
That's fixed by using the decode() method.
Change-Id: Ice01aaa1822dde8d957b5bf096021dd5a2b7dd51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253659
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Jiri Tyr <jiri.tyr@gmail.com>
(cherry picked from commit 83a3227b62)
The code has an ad-hoc check in that it requires the launcher major
version to not be less than the source code version. We don't really
care about that requirement, and it doesn't fit with our other version
checks. Rework it so we explicitly declare the min launcher version
that is supported.
We'll start with requiring repo launcher 1.15 which was released back
in 2012. Hopefully no one has anything older than that, although it's
not clear we work with even newer versions than that :). But let's be
a little conservative with the first update to this logic.
Bug: https://crbug.com/gerrit/10418
Change-Id: I611d70c60324d313c76874e978b8499a491a5d00
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254278
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Execution of 'repo forall -p -c' doesn't work with Py3 and ends up
with an error:
Got an error, terminating the pool: TypeError: can only concatenate
str (not "bytes") to str
That's fixed by using the decode() method.
Change-Id: Ice01aaa1822dde8d957b5bf096021dd5a2b7dd51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253659
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Jiri Tyr <jiri.tyr@gmail.com>
Currently our default behavior is:
* Try to sync all repos
* If any errors seen, exit
* Try to garbage collect all repos
* If any errors seen, exit
* Try to update local project list
* If any errors seen, exit
* Try to checkout out all local repos
* If any errors seen, exit
Users find these incomplete syncs confusing, so lets try to complete
as much as possible by default and printing out summaries at the end.
Bug: https://crbug.com/gerrit/11293
Change-Id: Idd17cc9c3bbc574d8a0f08a30225dec7bfe414cb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238554
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We want to start warning about Python 2 usage, but we can't do it
simply because the shebang is /usr/bin/python which might be an old
version like python2.7.
We can't change the shebang because program name usage is spotty at
best: on some platforms (like macOS), it's not uncommon to not have
a `python3` wrapper, only a major.minor one like `python3.6`. Using
python3 wouldn't guarantee a new enough version of Python 3 anyways,
and we don't want to require Python 3.6 exactly, just that minimum.
So we check the current Python version. If it's older than the ver
of Python 3 we want, we search for a `python3.X` version to run. If
those don't work, we see if `python3` exists and is a new enough ver.
If it's not, we die if the current Python 3 is too old, and we start
issuing warnings if the current Python version is 2.7. This should
allow the user to take a bit more action by installing Python 3 on
their system without having to worry about changing /usr/bin/python.
Once we require Python 3 completely, we can simplify this logic a bit
by always bootstrapping up to Python 3 and failing with Python 2.
We have a few KI with Windows atm though, so keep it disabled there
until the fixes are merged.
Bug: https://crbug.com/gerrit/10418
Change-Id: I5e157defc788e31efb3e21e93f53fabdc7d75a3c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253136
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
The git-2.10 series was released in 2016. Since we're moving to
require Python 3.6 which was also released in 2016, bumping up the
git version seems reasonable. Also we don't really test any git
versions close to as old as 1.7.2 which was released in 2010.
Change-Id: Ib71b714de6cd0b7dd50d0b300b108a560ee27331
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253134
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The use case is any situation where your manifest does
not exist on server, but where you still want to do
full sync for the projects, without having your
workspace manifest switched to other branch or
forwarded to latest or similar.
This allows syncing to a historical manifest in git log,
that does not have a branch, as well as when integrating
something together that has not been pushed upstream yet.
Changes can also exist locally on a manifest that is
behind head, meaning not requiring rebase to latest.
Tested using:
$ cd .repo/manifests/
$ git checkout <any hash 1>
$ <do local modifications>
$ repo sync --no-manifest-update
$ git checkout <any hash 2>
$ repo sync --no-manifest-update
Change-Id: I0c9773aa8bc5876813a2e7d7fec697abcb2d9e94
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/246445
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This commit supports for the 'remote' attribute in
<extend-project>. This avoids the need to perform a <remove-project>
followed by a <project> in local manifests.
Change-Id: I9f9347913337ec9d159bc264d15ce97881ae5398
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253092
Tested-by: Kyunam Jo <kyunam.jo@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reject paths in <copyfile> & <linkfile> that try to use symlinks or
non-file or non-dirs.
We don't fully validate <linkfile> when src is a glob as it's a bit
complicated -- any component in the src could be the glob. We make
sure the destination is a directory, and that any paths in that dir
are created as symlinks. So while this can be used to read any path,
it can't be abused to write to any paths.
Bug: https://crbug.com/gerrit/11218
Change-Id: I68b6d789b5ca4e43f569e75e8b293b3e13d3224b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233074
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
This reverts commit 1e01a74445.
Not all platforms support select.poll() currently it seems.
At least macOS's Python 2 doesn't (while macOS Python 3 does).
Lets back this out for the existing release series and once we
start repo-2 which is Python 3-only, we can put this back in.
Change-Id: I205206b0fa4fe2d755f4fbc6ec683ad125f27cc2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253072
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
After Ib546f5ebbc8a23875fbd14bf166fbe95b7dd244e, repo info now displays
the current project revision in the 'Current revision' field.
While the output is more consistent, there are use cases for the
revision expression as shown in the manifest. This patch re-adds the
manifest revision as a new 'Manifest revision' field.
Change-Id: I50c1559dcb7ceb69af07352b956d78f85b8f592e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/240799
Tested-by: Diogo Ferreira <deovferreira@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
select() has a limit of FD_SETSIZE file descriptors. If you run repo
sync -j500 you'll pretty quickly hit this limit and get "file descriptor
out of range for select" errors. poll() has no such limit.
Change-Id: I21f350e472bda1db03dcbcc437645c23dbc7a901
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248852
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Theodore Dubois <tbodt@google.com>
Before a2cd6aeae8, "repo mirror with --current-branch" fetches git data
using command
git fetch --progress --update-head-ok cros --tags
No refspec is specified, thus it fetches default refspec, which is +refs/heads/*:refs/heads/*
After a2cd6aeae8, the fetch command became
git fetch --progress --update-head-ok cros --tags +refs/tags/*:refs/tags/*
It did not only add tags refspec, but also suppressed the fetching of default refspec.
In other words, repo mirrors doesn't work if current_branch_only=True.
This CL explicitly adds the default refspec to command line if none is
specified.
Bug: https://crbug.com/gerrit/11990
Change-Id: Iadcf7b9aa50f53c47132cfe6c53b3fb2076ebca2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/246632
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Kuang-che Wu <kcwu@chromium.org>
The upload module tries to turn the strings into bytes before passing
to EditString, but it combines bytes & strings causing an error. The
return value might be bytes or string, but the caller only expects a
string. Lets simplify this by sticking to strings everywhere and have
EditString take care of converting to/from bytes when reading/writing
the underlying files. This also avoids possible locale confusion when
reading the file by forcing UTF-8 everywhere.
Bug: https://crbug.com/gerrit/11929
Change-Id: I07b146170c5e8b5b0500a2c79e4213cd12140a96
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/245621
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Series of steps:
* Create a local "b1" branch with `repo start b1` that tracks a remote
branch (totally fine)
* Manually create a local "b2" branch with `git branch --track b1 b2`
that tracks the local "b1" (uh-oh...)
* Delete the local "b1" branch manually or via `repo prune` (....)
* Try to process the "b2" branch with `repo prune`
Since b2 tracks a branch that no longer exists, everything blows up
at this point as we try to probe the non-existent ref. Instead, we
should flag this as unknown and leave it up to the user to resolve.
This probably could come up if a local branch was tracking a remote
branch that was deleted from the server, and users ran something like
`repo sync --prune` which cleaned up the remote refs.
Bug: https://crbug.com/gerrit/11485
Change-Id: I6b6b6041943944b8efa6e2ad0b8b10f13a75a5c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236793
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Kirtika Ruchandani <kirtika@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the local branch changed state since its last upload, the data
cached in .git/config related to the last uploaded CL might not be
that relevant. If we're able to fast forward merge to the latest
tree state, then let's do that. This would be akin to checking
out a detached head before syncing where we already switch state.
If we aren't able to fast forward merge, then it's not a big deal
as we'll continue on to the existing branch checking logic.
This would be easy to reproduce by doing something like:
$ repo start foo .
$ git revert HEAD
$ repo upload --cbr .
$ git reset --hard HEAD^
<CL is merged>
$ repo sync .
<we can fast forward>
Change-Id: I7d62f3d1ba5314a349d85b4dbb0ec8352eca18bb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238552
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
The current sync output displays "Fetching project" and "Checking out
project" messages and progress bar updates independently leading to a
lot of spam. Lets merge these periodic outputs with the status bar to
get a little bit tighter output in the normal case. This doesn't solve
all our problems, but gets us closer.
Bug: https://crbug.com/gerrit/11293
Change-Id: Icd627830af4dd934a9355b7ace754b56dc96cfef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244934
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If you run `repo sync -l foo` without first `repo sync -n foo`,
repo sets up an invalid gitdir tree that gets wedged and requires
manual recovery. Add a sanity check to abort cleanly first.
Change-Id: Iad865ea860a3f1fd2f39ce683fe66bd4380745a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244732
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When repo sync fails because some git trees are not in clean state and
as such can not be rebased automatically, it is a pain to figure out
which trees are the culprits.
With this patch the list of offending trees is printed when repo sync
reports checkout errors.
TEST=ran 'repo sync' and observed the proper list of directories show
up after the final error message
Bug: https://crbug.com/gerrit/11293
Change-Id: Icdf1a03e9014ecb184f331f513cc9a2efc7d11ed
Signed-off-by: Vadim Bendebury <vbendeb@google.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244053
Reviewed-by: Mike Frysinger <vapier@google.com>
Hitting Ctrl-C in the middle of this func will leave the .git in a
bad state that requires manual recovery. The code tries to catch
all exceptions and recover by deleting the incomplete .git dir, but
it omits KeyboardInterrupt which Exception misses.
We could add that to the recovery path, but we can make this more
robust with a different approach: set up everything in .git.tmp/
and only move it to .git/ once we've fully initialized it.
Change-Id: I0f5b97f2e19fc39cffc3e5e23993a2da7220f4e3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244733
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The plan previously documented was <=1.13.x is Python 2 and >=1.14.x
is Python 3. Other projects that migrated Python versions and drop
support for older have tended to take a more drastic version jump to
make it clearer to users. So lets adjust the plan to say <=1.x will
support Python 2, and >=2.x will be Python 3-only.
This also allows us to harmonize the repo launcher version. It is
currently sitting at v1.26 and has been incremented independently of
the repo version for the life of the project. While we might know
these lower nuances, pretty much no one else does and it just leads
to confusion: do I know version 1.26 or version 1.13.7? Or do I
have both? What does that even mean?
Once we update the major version to 2.0.0, we can also adjust the
launcher script to 2.0.0, and then the launcher release process will
be tied to a new repo release in general.
Bug: https://crbug.com/gerrit/10418
Change-Id: Idb2257371a06e56d2923cf717345c028f49176a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/240372
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
`repo forall <proj>` will look up all <proj> in the manifest for all
manifest groups regardless of which are active. If <proj> is checked
out to different locations depending on the group, this ultimately
fails as we're unable to locate all of them.
Simple fix is to only include projects that match the manifest groups
that we already passed down & initialized to the active set, and that
we already use when getting the default project list.
Bug: https://crbug.com/gerrit/11677
Bug: https://crbug.com/1011226
Change-Id: I975f10f9a9e5a1cad7d87344123f8003732dab27
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239652
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raul Rangel <rrangel@chromium.org>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The "Current revision" field shows the revision as listed in the
manifest. I think most users expect this to show the revision
that the git tree is checked out to instead. Switch the output
to show that revision instead, and add a "Current branch" if it
matches a local branch.
Change-Id: Ib546f5ebbc8a23875fbd14bf166fbe95b7dd244e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239240
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
imp is deprecatedon py3. It's also not used with py3, so just move it
to the py2 import block
Test: run `repo` command and verify warning is no longer present
Test: verify `repo sync` and `repo upload` function as expected
Change-Id: I9d59403d7819c4a478c9f54cbef114f8a96486a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239713
Tested-by: Rashed Abdel-Tawab <rashedabdeltawab@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If the repo client checkout is in an incomplete sync state, the work
git repo might be in a bad way. Turn errors parsing HEAD into None
since callers of CurrentBranch already need to account for it.
Change-Id: Ia7682e29ef4182006b1fb5f5e57800f8ab67a9f4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239239
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If the user passes in bad projects like `repo info asdf`, we currently
silently swallow those and do nothing. Allow NoSuchProjectError to
bubble up to main which will handle & triage this correctly for us.
Change-Id: Ie04528e7b7a164293063a636813a73eaabdd5bc3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239238
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Depending on where/how repo is invoked, the active version might be
from a git tree, and it might be different from the .repo/repo.git/
version in the current repo client checkout. Report both if they're
different so it's clearer.
Lets also include the two different User-Agent's that we set up when
talking to networked services.
Bug: https://crbug.com/gerrit/11144
Change-Id: I2ebb6e3ac30e374a8406cab3e4438087246a8c57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239234
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We've been setting the User-Agent header when making connections
from repo itself, but not when running git (as the latter will set
up User-Agent itself). Our Gerrit/Git admins say it'll be helpful
if we pass through the repo version settings even when running git.
We currently set GIT_HTTP_USER_AGENT and not GIT_USER_AGENT as it's
unclear if the extended form works over all protocols. We can wait
for a user request.
Bug: https://crbug.com/gerrit/11144
Change-Id: I21d293f49534058dbc23225152451df26c5b7bef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239233
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Convert the RepoUserAgent function into a UserAgent class. This
makes it cleaner to hold internal state, and will make it easier
to add a separate git User-Agent, although we don't do it here.
We make the RepoSourceVersion independent of GitCommand so that
it can be called by the class (later).
Bug: https://crbug.com/gerrit/11144
Change-Id: Iab4e1f974b8733a36b243b2d03f5085a96effa19
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239232
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Lets switch the default rebase behavior to align with our new sync
behavior: we try to rebase all projects by default and exit/summarize
things at the very end if there were any errors. Or if people want
to exit immediately, they can use the new --fail-fast option.
Change-Id: I436ac563f972b45de6ce9ad74da1e4870e584902
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238553
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Cut out some more standalone code from Execute to make this func a
bit more manageable. The manifest project update is pretty simple
and standalone, but still takes up a good chunk of what's left.
Change-Id: Idc2442d9def495eccd0a49cda203c44aef16f129
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236614
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
When displaying progress bars, we use \r to reset the cursor to the
start of the line before showing the new update. This assumes the
new line will fully erase whatever was displayed there previously.
The "done" codepath tries to handle this by including a few extra
spaces at the end of the message to "white out" what was there.
Lets replace that hack with the standard ECMA escape sequence that
clears the current line completely. This is the CSI "erase in line"
sequence that the terminal will use to delete all content. The \r
is still needed to move the cursor to the start of the line. Using
this sequence should be OK since we're already assuming the terminal
is ECMA compliant with our use of coloring sequences. We also put
the \r after the CSI sequence on the off chance the terminal can't
process it and displays a few bytes of garbage.
The other improvement is to the syncbuffer API. When it dumps its
status information, it almost always comes after a progress bar
update which leads to confusing comingled output. Something like:
Fetching projects: 100% (2/2) error: src/platform2/: branch ...
Since the progress bar is "throw away", have the syncbuffer reset
the current output to the start of the line before showing whatever
messages it has queued.
Bug: https://crbug.com/gerrit/11293
Change-Id: I6544d073fe993d98ee7e91fca5e501ba5fecfe4c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236615
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The smart sync logic takes up about 45% of the overall Execute func
and is about 100 lines of code. The only effect it has on the rest
of the code is to set the manifest_name variable. Since this func
is already quite huge, split the smart sync logic out.
Change-Id: Id861849b0011ab47387d74e92c2ac15afcc938ba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234835
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This makes it way easier to recover from forgetting to run repo start
before committing: just run `repo start -b new-branch`, instead of
all that tedious mucking around with reflogs.
Change-Id: I56d49dce5d027e28fbba0507ac10cd763ccfc36d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232712
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
In commit d9e5cf0e ("sync: invert --force-broken with --fail-fast") the
force-broken option has been deprecated. Accidentally the option has
been changed from Boolean to Value. This breaks all users of repo with:
main.py: error: -f option requires an argument
This is easy to avoid by keeping the type.
Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Change-Id: Ia8b589cf41ac756d10c61e17ec8d76ba8f7031f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/235043
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
A common pattern in our subcommands is to verify the arguments &
options before executing things. For some subcommands, that check
stage is quite long which makes the execution function even bigger.
Lets split that logic out of the execute phase so it's easier to
manage these.
This is most noticeable in the sync subcommand whose Execute func
is quite large, and the option checking makes up ~15% of it.
The manifest command's Execute can be simplified significantly as
the optparse configuration always sets output_file to a string.
Change-Id: I7097847ff040e831345e63de6b467ee17609990e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234834
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
People seem to not expect the sync process to halt immediately if an
error is encountered. It's also basically guaranteed to leave their
tree in an incomplete state. Lets invert the default behavior so we
attempt to sync (both fetch & checkout) all projects. If an error is
hit, we still exit(1) and show it at the end.
If people want the sync to abort quickly, they can use the new option
--fail-fast.
Bug: https://crbug.com/gerrit/11293
Change-Id: I49dd6c4dc8fd5cce8aa905ee169ff3cbe230eb3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234812
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Callers don't actually see -1 (they'll usually see 255, but the exact
answer here is complicated). Just switch to 1 as that's the standard
value tools use to indicate an error.
Change-Id: Ib712db1924bc3e5f7920bafd7bb5fb61f3bda44f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233553
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The partial clone rework (commit 745be2ede1
"Add support for partial clone") changed the behavior when a single repo
hit a failure: it would always call sys.exit() immediately. This isn't
even necessary as we already pass down an error event object which the
workers set and the parent checks. Just delete the exit entirely.
Change-Id: Id72d8642aefa2bde24e1a438dbe102c3e3cabf48
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233552
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The GitCommand Wait helper takes care of decoding bytes to strings
for us. That means we don't have to decode stdout ourselves which
is what our local rev list, ls-remote, and generic get_attr helpers
were doing.
If we don't use Wait though to capture the output but instead go
directly to the subprocess stdout, we do have to handle decoding
ourselves. This is what the diff helpers were doing.
Bug: https://crbug.com/gerrit/10418
Change-Id: I057ca245af3ff18d6b4a074e3900887f06a5617d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233076
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
As we convert repo to support Python 3, the version of Python that we
use might not be the version that repo hooks users have written for.
Since repo upgrades are not immediate, and not easily under direct
control of end users (relative to the projects maintaining the hook
code), allow hook authors to declare the version of Python that they
want to use.
Now repo will read the shebang from the hook script and compare it
against the version of Python repo itself is running under. If they
differ, we'll try to execute a separate instance of Python and have
it load & execute the hook. If things are compatible, then we still
use the inprocess execution logic that we have today.
This allows repo hook users to upgrade on their own schedule (they
could even upgrade to Python 3 ahead of us) without having to worry
about their supported version being exactly in sync with repo's.
Bug: https://crbug.com/gerrit/10418
Change-Id: I97c7c96b64fb2ee465c39b90e9bdcc76394a146a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228432
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
A new option, --partial-clone is added to 'repo init' which tells repo
to utilize git's partial clone functionality, which reduces disk and
bandwidth usage when downloading by omitting blob downloads initially.
Different from restricting clone-depth, the user will have full access
to change history, etc., as the objects are downloaded on demand.
Change-Id: I60326744875eac16521a007bd7d5481112a98749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
We're relying on sys.stdout.write() to flush its buffer which isn't
guaranteed, and is not the case in Python 3. Change to use print()
everywhere to be standard, and utilize the end= keyword to get the
EOL semantics we need.
We can't use print's flush= keyword as that's only in Python 3.
Leave behind a TODO to clean it up when we can drop Python 2.
Bug: https://crbug.com/gerrit/10418
Change-Id: I562128c7f1e6d154f4a6ecdf33a70fa2811dc2af
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/230392
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
This fix exception with python3 with stack-trace:
error: Cannot fetch platform_external_grpc-grpc-java.git (UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 640: invalid start byte)
[...]
File "[...]project.py", line 2255, in _IsValidBundle
if f.read(16) == '# v2 git bundle\n':
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
Even if we ask 16 characters, python buffered decoder will try to decode more in the buffer
The patch works for python2 and python3, and open the file in byte mode so that decoding is not attemped
Signed-off-by: Pierre Tardy <tardyp@gmail.com>
Change-Id: I837ae3c5cd724b34670fc2a84e853431f482b20d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/224642
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We've been truncating the git version info in the user agent to the
first three components. So given an example `git --version` like
"2.22.0.510.g264f2c817a-goog", we were cutting it down to "2.22.0".
For user-agent purposes, we usually want that full string, so use
the original full value instead.
Bug: https://crbug.com/gerrit/11144
Change-Id: I8ffe3186bdaac96164c34ac835a54bb3fc85527e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231056
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We were returning an e.g. tuple(1,2,3), but that strips off the full
version string which we might want in some places e.g. '1.2.3-rc3'.
Change the return value to a namedtuple so we can pass back up the
full version string. For code doing a compare with three elements
(all code today), things still work fine as the namedtuple will DTRT
in this scenario.
Bug: https://crbug.com/gerrit/11144
Change-Id: Ib897b5df308116ad1550b0cf18f49afeb662423e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231053
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
%prog represents the full subcommand ("repo" + subcommand name), not a
Windows-style environment variable for "repo". The current help output
shows
repo forall% forall ...
Correct the variable usage so it shows "repo forall ..." instead.
Change-Id: I1fea55572428cc922ddf24ace1168a3d8f82dad0
This takes a single argument (the error message), not multiple
arguments that get formatted implicitly.
Change-Id: Idfbc913ea9f93820edb7e955e9e4f57618c8cd1b
Currently we read the binary stream from the subprocess code directly
before waiting for it to finish, but there's no need to do so as we
aren't streaming the output to the user. This also means we pass up
binary data to the caller as we don't go through GitCommand's internal
logic which decodes the stream as utf-8.
Simplify the code by calling Wait first, then splitting the entire
captured output in one line.
Bug: https://crbug.com/gerrit/10418
Change-Id: I7a57904be8cb546a229980fb79c829fc3df31e7d
Python 3 returns bytes by default with urlopen. Adjust our code to
handle that scenario and decode as necessary.
Bug: https://crbug.com/gerrit/10418
Change-Id: Icf4cd80e7ef92d71a3eefbc6113f1ba11c32eebc
Neither of the fields here expect floats so make sure we use integer
division when calculating things.
Bug: https://crbug.com/gerrit/10418
Change-Id: Ibda068b16a7bba7ff3efba442c4bbff4415caa6e
This module uses print() so make sure we import the print function.
It doesn't really impact the current code due to the simple way it
is calling print, but we should be sane to avoid future issues.
Change-Id: I0b15344678c1dcb71207faa333c239b3fced1d62
We avoided this future import because Python 2.4 & 2.5 did not
support it. We've dropped support for Python 2.6 at this point,
and those versions are long dead. Since this workaround adds a
bit of complexity to the codebase, drop it. Considering we are
not running any actual tests against older versions, there's no
sense in trying to support them anymore.
Change-Id: Icda874861e8a8eb4fa07c624a9e7c5ee2a0da401
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.
Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
Use the `raise` statement directly.
Switch to using .items() instead of .iteritems(). Python 3 doesn't
have .iteritems() as .items() is a generator, and these are small
enough that the Python 2 overhead should be negligible.
We have to run .keys() through list() in a few places as Python 3
uses a generator and we sometimes want to iterate more than once.
That's why we don't change all .keys() or .items() calls -- most
are in places where generators are fine.
Bug: https://crbug.com/gerrit/10418
Change-Id: I469899d9b77ffd77ccabb831bc4b217407fefe6f
Standard utilities exit normally/zero when users explicitly request
--help, and they write to stdout. Exiting non-zero & using stderr
is meant for incorrect tool usage instead. We're already doing this
for `repo help <init|gitc-init>` calls, so lets fix `repo help` and
`repo --help|-h` to match.
Change-Id: Ia4f352b431c91eefef70dcafc11f00209ee69809
This isn't executable (+x), nor does it have a main func or code
that would run if it were. It's simply an imported module like
most others in here. Drop the shebang to avoid confusion.
Change-Id: I5e2881eb1de5e809a3fa9e8f49220ed797034fb1
The current implementation ignores the user-specified paths to
manifests. if the "repo diffmanifests" is invoked with absolute
file paths for one or both manifests, the command fails with message:
fatal: duplicate path ... in /tmp/manifest-old.xml
Also the current implementation fails to expand the absolute path to
manifest files if "repo diffmanifests" is invoked with relative
paths, i.e "repo diffmanifests manifest-old.xml manifest-new.xml".
fatal: manifest manifest-old.xml not found
This commit fixes the first issue by disabling the local manifest
discovery for diffmanifests command, and the second issue by
expanding paths to manifests within "diffmanifests" sub-command.
Test: repo manifest --revision-as-HEAD -o /tmp/manifest-old.xml
repo sync
repo manifest --revision-as-HEAD -o /tmp/manifest-new.xml
repo diffmanifests /tmp/manifest-old.xml /tmp/manifest-new.xml
Change-Id: Ia125d769bfbea75adb9aba81abbd8c636f2168d4
Signed-off-by: Vasyl Gello <vasek.gello@gmail.com>
During sync, repo runs `git read-tree --reset -u -v HEAD` which causes
git-lfs's smudge filter to run, which fails because git-lfs does not
work with bare repositories.
This was fixed in I091ff37998131e2e6bbc59aa37ee352fe12d7fcd to
automatically disable this smudge filter. However, later versions of
Git (2.11.0) introduced a new filter protocol [1], to avoid spawning
a new command for each filtered file. This was implemented in Git-LFS
1.5.0 [2].
This patch fixes the issue by setting the git lfs process filter, in
addition to the smudge filter. For any projects that have LFS objects,
`git lfs pull` must still be executed manually afterwards.
[1] edcc85814c
[2] https://github.com/git-lfs/git-lfs/pull/1617
Bug: https://crbug.com/gerrit/10911
Change-Id: I277fc68fdefc91514a2412b3887e3be9106cab48
If a user is asking for a shallow clone of the repos, they probably
expect a shallow clone of the manifest repo too. For very large
manifest repos, this can be a huge space and time savings. For one real-world
repo, a 'repo init --no-tags --current-branch' used 350MB of disk space and
took 7 minutes. Adding --depth 1 and this change reduced it to 10MB and 2.5
minutes.
Change-Id: I6fa662e174e623ede8861efc862ce26d65d4958d
When syncing a lot of projects in parallel, it is not otherwise
clear which one of them has failed to init work tree.
Change-Id: I8edfb4955023389a499e99cfa511bdc0d2850ba2
This allows projects to include submodules inside of
projects that use repo without repo incorrectly believing
the area is dirty just because a submodule has updates.
This is in line with git porcelain commands which generally
require a commandline flag to include submodules (git add,
git rebase).
Change-Id: Ide8a292162a42ab35145b5c4ca8ca0d020cdfe81
If "repo init" was run in a path containing "%", "repo info" would fail
printing the path with
File ".repo/repo/color.py", line 173, in f
return fmt % args
TypeError: not enough arguments for format string
as the "%" in the path name is interpreted as the start of a formatting
specifier. Avoid that by using the non-formatting printer for headtext
which does not require any formatting so there is no need to try to
expand "%" sequences.
Change-Id: Ie193b912191fe7cdabdce5c97bb100f0714f6e76
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
The newly introduced "Already have persistent ref" message prevents
repo from overwriting the last line when syncing quietly. Omit the
message when syncing quietly to clean up the output and to restore
the previous behaviour.
Change-Id: Idf42751c67f95924d6de50092ba54d4c6fe93096
The comment in _create_symlink is incorrect. The return value of
CreateSymbolicLink is as documented, it was just declared with
the wrong return type. The actual return type is BOOLEAN, not BOOL.
Fixing this allows us to simplify the code a bit.
Change-Id: I4d2190a50d45ba41dd9814bf7079a5784fc0a366
If a tag is rewritten on the server (which is bad), trying to recover
locally with `repo sync --force-sync` doesn't actually work. People
have to manually delete things themselves to fix syncing. While tags
should never be rewritten in practice, allow users to easily recover
from broken servers.
We updated some of these code paths already (see commit 6e53844f1e
"Allow clobbering of existing tags from remote."), but the incremental
update flow was missed.
Bug: b/120778183
Bug: chromium:932651
Test: delete local tag & recreate to diff commit, then check
`repo sync` & `repo sync --force-sync` behavior
Change-Id: I3648f7d2526732c06016b691a9a36c003157618d
Forcefully remove dirty projects if option '--force-remove-dirty' is given.
The '--force-remove-dirty' option can be used to remove previously used
projects with uncommitted changes. WARNING: This may cause data to be lost
since uncommitted changes may be removed with projects that no longer exist
in the manifest.
Change-Id: I844a6e943ded522fdc7b1b942c0a1269768054bc
Even if dots are used as separators for Git config keys, they are not
forbidden as part of submodule names. This fixes the issue of submodules
with a name like e.g. "long.js" to be skipped from checkout.
Change-Id: I77da07925ad207fa3d043067dfbbcb4a1ebdac4d
Running lots of sync processes in parallel can hit the failure:
Fetching projects: 23% (124/523)Exception in thread Thread-201:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/src/repo/subcmds/sync.py", line 278, in _FetchProjectList
success = self._FetchHelper(opt, project, *args, **kwargs)
File "/usr/local/src/repo/subcmds/sync.py", line 357, in _FetchHelper
start, finish, success)
File "/usr/local/src/repo/event_log.py", line 104, in AddSync
event = self.Add(project.relpath, task_name, start, finish, success)
File "/usr/local/src/repo/event_log.py", line 74, in Add
'id': (kind, next(self._next_id)),
ValueError: generator already executing
It looks like, while we lock the multiprocessing value correctly, the
generator that wraps the value isn't parallel safe. Since we don't
have a way of doing that (as it's part of the language), turn it into
a plain function call instead.
Bug: https://crbug.com/gerrit/10293
Change-Id: I0db03601986ca0370a1699bab32adb03e7b2910a
When --ne/--no-emails is added to 'repo upload' command line, gerrit
server will not generate notification emails.
project.py:Project.UploadForReview method is modified to accept a
string recognizable by gerrit to indicate different sets of destination
email addressees, but the upload command line allows only one option -
disable sending emails completely.
Default repo upload behavior is not being changed.
TEST=tried in the Chrome OS repo, observed that patches uploaded with
--ne or --no-emails indeed do not trigger any emails, while
patches uploaded without these command line options still trigger
email notifications.
Change-Id: I0301edec984907aedac277d883bd0e6d3099aedc
Pass options through the refspec for all transports, including ssh.
This means the behavior will be more consistent between the ssh and
https cases.
A downside is that this prevents passing special characters in
reviewer options. That already didn't work over https, so it seems
okay. It could be fixed by using push options instead.
Change-Id: Ia38d16e350cb8cb0de14463bfb3d9724e13bc4bf
* Add more file i/o wrappers in platform_utils to allow using
long paths (length > MAX_PATH) on Windows.
* Paths using the long path syntax ("\\?\" prefix) should never
escape the platform_utils API surface area, so that this
specific syntax is not visible to the rest of the repo code base.
* Forward many calls from os.xxx to platform_utils.xxx in various place
to ensure long paths support, specifically when repo decides to delete
obsolete directories.
* There are more places that need to be converted to support long paths,
this commit is an initial effort to unblock a few common use cases.
* Also, fix remove function to handle directory symlinks
Change-Id: If82ccc408e516e96ff7260be25f8fd2fe3f9571a
"repo init --reference" has two purposes: to decrease bandwidth used
at clone time, and to decrease disk usage afterward, by using the
reference repositories as an alternate store of objects even after
the clone. The downside is that it makes the borrowing repositories
dependent on the reference repositories, so it is easy to end up
with missing objects by mistake after a cleanup operation like "git
gc".
To prevent that, v2.3.0-rc0~30^2 (clone: --dissociate option to mark
that reference is only temporary, 2014-10-14), "git clone" gained a
--dissociate option that makes --reference reuse objects from the
reference repository at clone time but copy them over instead of
using the reference as an alternate. This is more straightforward to
use than plain --reference, at the cost of higher disk usage.
Introduce a --dissociate to "repo init" that brings the same benefits
to repo. The option is simply passed on to "git clone".
Change-Id: Ib50a549eb71e0a2b3e234aea57537923962a80d4
Since gitiles recommends using # headers over ---/=== underlines,
change the manifest-format.md over and all our help texts.
Change-Id: I96391d41fba769e9f26870d497cf7cf01c8d8ab3
pylint is not used since bb5b1a0. The pyflakes cleanup mentioned in that
commit has not been done, but given that this project is no longer being
actively developed I don't think it's worth spending time doing it.
Leaving the pylint suppressions causes confusion because it leads people
to think that we are still using pylint.
Change-Id: If7d9f280a0f408c780f15915ffdb80579ae21f69
Commit 27226e742d introduced a warning if
"repo" is not part of the bootstrapped REPO_URL. However, that check was
done too early, directly after the call to _Clone. As the _Clone function
does not actually clone but it only initializes and fetches, the check
needs to be moved to after the call to _Checkout.
To reproduce, call
repo init --no-clone-bundle --repo-branch=master -u https://android.googlesource.com/platform/manifest
which will currently always show the (bogus) warning message. With this
fix, the warning will only be shown if "repo" indeed does not exist.
While at it, also slightly improve the code by using os.path.join().
Change-Id: Ied89e24231addabab6075005065748df1ffa74c4
While on Linux stderr is unbuffered, it is buffered on Windows. Always
flush stderr on Windows to ensure any error messages appear in the right
order to ease diagnosing.
Change-Id: I37300e384ecd3a51a321a48818f0114d6f3357a0
Desktops and servers tend to have no power sensor, thus on_ac_power returns
255 ("unknown"). Thus, let's take any answer other than 1 ("battery") as
no contraindication to run gc.
If that tool returns "unknown", there's no point in querying other sources
as it already queried them, and is smarter than us (can handle multiple
adapters).
Reported by: Xin Li <delphij@google.com>
Signed-off-by: Adam Borowski <kilobyte@angband.pl>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
(cherry picked from git.git commit 781262c5e7ad4a7813c528803117ed0d2e8c5172)
Signed-off-by: Fredrik Roubert <roubert@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Change-Id: I51fe2eb1eb879492a61e8e09c86ee34d049036c1
It's convenient to set upstream for all projects in a manifest instead of
repeating the same value for each project.
Change-Id: I946b1de4efb01b351c332dfad108fa7d4f443cba
The hooks are run from the top of the manifest checkout, not from the
individual git repos. It's up to individual hooks to chdir as needed.
Change-Id: I53325e0c3dcaa9c250b02b223e78d238d2cbd36d
When someone does "repo download -c <project> <change>"
without specifying a patch number, by default patch 1 is
downloaded. An alternative is to look for the latest patch
and download the same when no explicit patch is given.
This commit does the same by identifying the latest patch
using "git ls-remote".
Change-Id: Ia5fa7364415f53a3d9436df4643e38f3c90ded58
This change adds support for the 'revision' attribute in
<extend-project>. This avoids the need to perform a <remove-project>
followed by a <project> in local manifests.
Change-Id: Id2834fcfc1ae0d74b3347bed3618f250bf696b1f
Allow the 'default' and 'project' element in the manifest
file to apply "--no-tags" option equivalent.
Change-Id: I7e0f8c17a0e25cca744d45df049076d203c52ff5
Signed-off-by: YOUNG HO CHA <ganadist@gmail.com>
Put the correctly-expanded relative paths in objects/info/alternates.
From gitrepository-layout(5), this path should be "relative to the
object database, not to the repository".
Change-Id: I7b2027ae23cf7d367b80f5a187603c4cbacdb2de
Issue: when subproject url is a relative in .gitmodules
repo tool cannot handle this and cause:
"fatal: '***' does not appear to be a git repository
fatal: Could not read from remote repository."
issue.
Signed-off-by: Shouheng Zhang <shouheng.zhang@intel.com>
Change-Id: I2a24c291ea0074ba13a740b32a11c0c25975e72b
If a user executes:
repo forall -c echo $REPO_PROJECT
then $REPO_NAME is expanded by the user's shell first, and passed
as $1 to the shell that executes echo. This will either result in
no output, or output of whatever REPO_NAME is set to in the user's
shell. Either way, this is an unexpected result.
The correct way to do it is:
repo forall -c 'echo $REPO_PROJECT'
such that $REPO_NAME is passed in to the shell literally, and then
expanded to the value set in the environment that was passed to
the shell.
Update the documentation to make this clearer.
Change-Id: I713caee914172ad8d8f0fafacd27026502436f0d
Make it explicit that the ssh wrapper we use for control master
support accepts OpenSSH-compatible command line arguments instead of
asking Git to guess.
The GIT_SSH_VARIANT setting was introduced in Git v2.13.0-rc0~3^2~2
(2017-02-01) as a more reliable detection method than relying on the
ssh command name. Fortunately the default variant was 'ssh' (i.e.,
OpenSSH-compatible) so this wasn't initially required.
Now Git wants to start using more OpenSSH features
(-o SendEnv=GIT_PROTOCOL), and in order to do so its ssh variant
detection will need to be tweaked. Set GIT_SSH_VARIANT explicitly
so this helper can continue to work regardless of how Git modifies
its autodetection.
Reported-by: William Yan <wyan@google.com>
Change-Id: I6bf2c53b4eb5303a429eae6cb68e0a5ccce89064
This change allows setting the EDITOR env. variable to point to a
program location that contains quotes and spaces.
For example:
> set EDITOR="C:\Program Files (x86)\Notepad++\notepad++.exe" -multiInst -nosession
> repo upload
Change-Id: Ic95b00f7443982b1956a2992d0220e50b1cf6bbb
os.remove raises an exception when deleting read-only files on
Windows. Replace all calls with calls to platform_utils.remove,
which deals with deals with that issue.
Change-Id: I4dc9e0c9a36b4238880520c69f5075eca40f3e66
Windows does not support pipe|fork, but we can simulate by creating
the pager as a child process, redirecting stdout/in/err appropriately
and then waiting for the child process to terminate after we are
done executing the repo command.
Change-Id: I5dd2bdeb4095e4d93bc678802e53c6d4eda0235b
Without this change, '.git\HEAD' files, for examples, are sometime
read incorrectly resulting in the current branch to be reset to
"master" when running a "repo init -b xxx" on an already initialized
repository.
Change-Id: I48c7ef85ff81626edf156914329a560e14252f2a
* changes:
Port os.rename calls to work on Windows
Workaround shutil.rmtree limitation on Windows
Add support for creating symbolic links on Windows
Make "git command" and "forall" work on Windows
When starting a branch, branch.merge is set to project revision unless
the revision is a SHA1. In that case, branch.merge is set to dest_branch
if defined or manifest default revision otherwise. This special handling
allows repo upload to work when the project revision is a SHA1.
Extend the special handling to also happen when the project revision
is a tag value or a change value so that repo upload will work in those
case as well.
Change-Id: Iff81ece40e770cd02535e80dcb023564d42dcf47
The output indicates that fetching happens even when it is skipped.
To avoid confusion, print a message when fetching is skipped for
an immutable ref so that the user knows when and why a fetch is skipped.
Change-Id: Id6e4812cebc5e57d379feb76a9d034af0b93043b
This option allow to bypass verification ssl certification while
establishing connection with Gerrit to upload review.
Change-Id: If2e15f5a273c18a700eb5093ca8a4d5a4cbf80cd
This reverts commit d88f53e2b9. I merged
it too hastily without paying enough attention to compatibility with
released Gerrit versions.
Change-Id: I4028d4737df1255f11e217da183a19a010597d5b
Considering that some users might expect changes created with
'-d' option are not public. Private changes may be a better
choice here than work-in-progress changes.
Change-Id: I46a8fb9ae38beb41cf96d6abe82bea6db2439669
This change adds options for git-repo tool to support private
changes and work-in-progress changes.
Change-Id: I343491f5949f06f1580d53f9cc0dee2dca09130f
$ git ls-files | grep py$ | xargs pyflakes
subcmds/stage.py:101: list comprehension redefines 'p' from line 63
subcmds/sync.py:784: list comprehension redefines 'p' from line 664
subcmds/upload.py:467: list comprehension redefines 'avail' from line 454
Change-Id: Ia65d1a72ed185ab3357e1a91ed4450c719e75a7c
The submodule argument to Sync_LocalHalf was missing in
MetaBranchSwitch, causing submodules not to get synced when the
-b/--manifest-branch argument to init is used.
Change-Id: Ie86d271abac2020725770be36ead83be3326e64b
Signed-off-by: Martin Kelly <mkelly@xevo.com>
With --force-broken it continue to fetch other projects but nothing
is added in directory because it abort some lines later.
Change-Id: I32c4a4619b3028893dc4f98e8d4e5bc5c09adb27
os.rename fails on Windows if the destination exists, so replace
os.rename to platform_utils.rename which handles the platform
differences.
Change-Id: I15a86f10f65eedee5b003b80f88a0c28a3e1aa48
By default, shutil.rmtree raises an exception when deleting readonly
files on Windows.
Replace all shutil.rmtree with platform_utils.rmtree, which adds an
error handler to make files read-write when they can't be deleted.
Change-Id: I9cfea9a7b3703fb16a82cf69331540c2c179ed53
Replace all calls to os.symlink with platform_utils.symlink.
The Windows implementation calls into the CreateSymbolicLinkW Win32
API, as os.symlink is not supported.
Separate the Win32 API definitions into a separate module
platform_utils_win32 for clarity.
Change-Id: I0714c598664c2df93383734e609d948692c17ec5
Python on Windows does not support non blocking file operations.
To workaround this issue, we instead use Threads and a Queue to
simulate non-blocking calls. This is happens only when running
with the native Windows version of Python, meaning Linux and Cygwin
are not affected by this change.
Change-Id: I4ce23827b096c5138f67a85c721f58a12279bb6f
See http://stackoverflow.com/questions/7004687/os-exec-on-windows:
execv on Windows does not behave as on Linux, i.e. a new process is
spawned and the parent process terminates right away, which makes the
shell prompt come back too soon.
Change-Id: I1f8d23208765988629f081e9b949c67cf71c08ae
Project names are stored as path using the '/' file separator, and
stored in a dictionary as keys.
Change-Id: Ide40dfe840958ac0d46caae5f77f1a49d71c9d90
The documentation of the XML file format contains DTD which contains
definition of all allowed elements and attributes. The "include" element
is defined but it's not referenced from the top-level "manifest"
element.
This patch is adding the "include" element into the list of elements of
the top-level "manifest" element.
Change-Id: I33beb8ef2846bbf42ffd42e6ae6888828566d604
This is required for setups, where Gerrit access using ssh is only available
for some networks.
For network without ssh access, repo will get ssh_info from Gerrit and
use ssh for communications - which will fail. To support this setup
we need to have an option to ignore the ssh_info provided by Gerrit and
use http(s).
Using git insteadOf as alternative results in the inability to add
reviewers using "repo upload --re=...", since the syntax of adding
reviewers differs for ssh and https. repo is assuming an ssh
connection and uses "git push --receive-pack=...", which will fail
since git silently uses https for push operation. repo must be aware
that https is used so it uses "git push remote ...:refs/for/...%r=..."
for upload.
Change-Id: Idd83baef0fb26ffcc9ac65e204b68d323ce177a1
As reported by pyflakes:
subcmds/abandon.py:84: undefined name 'p'
The name of the variable should be 'proj'.
Change-Id: Ic09eb92e8db6b510e99efce010bd0bb094d7cbfe
repo sync can sync submodules via the --fetch-submodules option.
However, if the manifest repo has submodules, those will not be synced.
Having submodules in the manifest repo -- while not commonly done -- can
be useful for inheriting a manifest from another project using <include>
and layering changes on top of it. In this way, you can avoid having to
deal with merge conflicts between your own manifests and the other
project's manifests (for example, if you're managing an Android fork).
Add a --submodule option to init that automatically syncs the submodules
in the manifest repo whenever the manifest repo changes.
Change-Id: I45d34f04517774c1462d7f233f482d1d81a332a8
Signed-off-by: Martin Kelly <mkelly@xevo.com>
Previously, this would always have exited with 1 on Windows, causing "git
gc --auto" to abort. Fix this by adding support for Windows.
Change-Id: Ie519b366a11b6b18b2d465e892e738de3f4bbc99
repo can be configured to download from any number of remote git repos.
However when one fails repo doesn't report which one. Example:
Fatal: remote error: Daily ls-remote rate limit exceeded for IP xx.xx.xx.xx
TEST=repo init -q -u https://chromium.googlesource.com/chromiumos/manifest.git
# Apply patch in ./.repo/repo/
# Simulate a git remote error:
sed -i -e 's#chromiumos/docs#chromiumos/XXdocs#' .repo/manifests/full.xml
repo sync --quiet --force-sync docs
# error message now shows the remote URL
Optional test tip: reduce the time.sleep(random(...)) in ./.repo/repo/project.py
Change-Id: I4509383b6a43a8e66064778e8ed612d8a735c8b6
If you don't know that the url to git-repo itself can be overridden via
REPO_URL, it's hard to debug cases where REPO_URL is accidentally set to
another repository, e.g. inside a Jenkins CI job. What makes is even
harder is that the ".repo/repo" directory gets silently removed in such
cases as verifications fails, which makes it impossible to look at the
cloned files to understand the problem.
To better protect against such an issue, warn if the cloned git-repo
repository does not contain a top-level "repo" file, and state that the
".repo/repo" directory will be removed in case of a clone failure.
Change-Id: I697b4999205a5967910c0237772ccaada01e74d4
Currently, if direct fetch of a sha1 is not supported by git server and
depth option is used, we fallback on syncing the upstream branch by
ignoring depth option.
This fallback doesn't work in next 2 cases:
(1) upstream attribute is not specified in manifest
(2) depth option is passed to repo init command
(not with clone-depth attribute in manifest)
This commit do the following:
- fixes (1) by updating condition used to apply fallback
first we retry with depth set to None, then by syncing all branches
- fixes (2) by passing depth as argument of _RemoteFetch() method
thus, its value is not set again to depth value passed to repo init
command when applying fallback
Change-Id: Ifd6fffafc49ba229df624b0d7b64c83d47619d17
repo already special-cases sso:// URLs to behave similarly to https://
and rpc:// elsewhere in repo, but it forgot to do so here.
Noticed when trying to use relative URLs in a manifest obtained using
an sso:// URL.
Change-Id: Ia11469a09bbd6e444dbc4f22c82f9bbe9f5fd083
A recent backward incompatible change created confusion and loss of
productivity and highlighted the very limited amount of information
provided when repo sync fails; merely recommending to --force-sync
and blow-up git repos without any hint as to why. The addition of
this basic _error(...) call would have provided a clue and will in
the future.
BUG=Issue 232
TEST=simulate a breakage similar to the ones reported at
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-dev/2-0oCy_CX5s
cd .repo/projects/src/third_party/libapps.git/
file info; rm info; ln -s wronglink info
cd -
repo sync src/third_party/libapps/
# error message now shows the failure
Change-Id: Idd2f177a096f1ad686caa8c67cb361d594ccaa57
The --quiet option reduces the output to just
a list of projects with modified workspaces (and
orphans if -o is specified)
A common use case is when performing a full-workspace
merge. The integrator will kick-off a merge via:
repo forall -c git merge <some tag>
And then produce a short list of conflicted projects via:
repo status -q
The integrator can then iteratively fix and clean up all conficted
components. The merge is complete when:
repo status -q
returns no output.
Change-Id: Ibbba8713eac35befd8287c95948874e23fd5c7e2
when you want to delete all local branches, you should be find
all branches' name, and type them behind 'repo abandon' command.
Usage:
repo abandon --all [<project>...]
Change-Id: I4d391f37fb9d89b8095488c585468eafc1a35f31
When repo syncs a manifest that utilizes multiple branches
in the same project, then the sync will use an extra
thread for each "duplicate". For example, if
the manifest includes the project "foo" and "bar"
twice, then "repo sync -jN" will fetch with N+2 threads.
This is caused by _FetchHelper() releasing the thread semaphore
object each time it's called, even though _FetchProjectList()
may call this function multiple times within the scope of a
single thread.
Fix by moving the thread semaphore release to
_FetchProjectList(), which is only called once per thread
instance.
Change-Id: I1da78b145e09524d40457db5ca5c37d315432bd8
When there's a symlink to a directory, os.walk still lists the symlink
in dirs, even if it isn't configured to follow symlinks. This will fail
the listdirs check if the symlink is broken (either before or during the
cleanup). So instead, check for directory symlinks and remove them using
os.remove.
Bug: Issue 231
Change-Id: I0ec45a26be566613a4a39bf694a3d9c6328481c2
When there are nested projects in a manifest, like on AOSP right now:
<project path="build" name="platform/build" />
<project path="build/blueprint" name="platform/build/blueprint" />
<project path="build/kati" name="platform/build/kati" />
<project path="build/soong" name="platform/build/soong" />
And the top "build" project is removed (or renamed to remove the
nesting), repo just wipes away everything under build/ and re-creates
the projects that are still there. But it only checks to see if the
build/ project is dirty, so if there are dirty files in a nested
project, they'll just be blown away, and a fresh worktree checked out.
Instead, behave similarly to how `git clean -dxf` behaves and preserve
any subdirectories that have git repositories in them. This isn't as
strict as git -- it does not check to see if the '.git' entry is a
readable gitdir, just whether an entry named '.git' exists.
If it encounters any errors removing files, we'll print them all out to
stderr and tell the user that we were unable to clean up the obsolete
project, that they should clean it up manually, then sync again.
Change-Id: I2f6a7dd205a8e0b7590ca5369e9b0ba21d5a6f77
Allow the 'remote' element in the manifest file to define an optional
'pushurl' attribute which is passed into the .git/config file.
Change-Id: If342d299d371374aedc4440645798888869c9714
Signed-off-by: Steve Rae <steve.rae@raedomain.com>
pylint reports a lot of warnings, but many of them are false positive,
and it's difficult to configure it. It also seems that for some reason
the included config file is not working well with the latest version.
Update the documentation to recommend using pyflakes and flake8 instead
of pylint. Remove the pylint config and add a basic flake8 config with
minimum settings:
- Maximum line length 80 columns
- Ignore warnings about indentation (repo uses 2 rather than expected 4)
- Ignore warnings about import placement
In this commit no code cleanup is done, and it's expected that most of
the files will throw up quite a few warnings, at least for flake8. These
can be cleaned up in follow-up commits.
The existing pylint suppression comments are left as-is. These will be
helpful when cleaning up pyflakes warnings later.
Change-Id: I2f7cb4340266ed07cc973ca6483b8f09d66a765b
The shared object stores confuse git and make it throw away objects which are
still in use. We'll avoid that problem by disabling automatic pruning on those
projects, but there's nothing preventing a user from changing the config back
or pruning a repository manually.
BUG=chromium:375945
TEST=Ran repo sync on fresh ChromeOS checkout, starting with a branch of repo
with this change. Verified that the kernel projects and no others were
identified as having shared object stores, and that repo successfully disabled
automatic pruning in their configs. Re-enabled pruning and ran repo sync just
on one of the kernel directories. Verified that pruning was re-disabled as a
result.
Change-Id: I728ed5b06f0087aeb5a23ba8f5410a7cd10af5b0
The requirement to explicitly specify the local project when starting
a new repo branch is somewhat counter intuitive.
This patch uses the current directory's git tree as the default
project.
Tested by running
'repo start <name>'
observed that the result is the same as if running
'repo start <name> .'
Change-Id: If106caa801b4cd5ba70dbe8354a227d59f100aa3
When nothing is pending, most of this code is already short-circuited.
Hoist the single check up to make this more obvious/slightly faster.
Change-Id: Iec3a7e08eacd23a7c5f964900d5776bf5252c804
if a gerrit server has ssh and https access enabled, but user access
(for some users) is limited to https, 'repo upload' command will fail
for them.
Gerrit returns a ssh configuration (gerrit/ssh_info), that does not work
for users limited to https.
With this patch repo will test, if the returned ssh configuration from
gerrit/ssh_info is working. if not, it will fall back to https for upload.
Change-Id: If98f472e994f350bf71f35610cd649b163f1ab33
this check can only detect errors that happen within 1 sec after launching
ssh. But this is typically enough to catch configuration issues like
'connection refused' or 'authentication failed'.
Change-Id: I00b6f62d4c2889b1faa6c820e49a198554c92795
Instead of
Do you want to allow this script to run (yes/yes-never-ask-again/NO)? (yes/always/NO)?
ask
Do you want to allow this script to run (yes/always/NO)?
Change-Id: I5f5a2d0e88086a8d85e54fb8623a62d74a20956a
Signed-off-by: Jonathan Nieder <jrn@google.com>
There have been a number of changes in the repo wrapper since the last
increment that was done in fee390ee:
- 9711a98 init: Add --no-clone-bundle option
- 631d0ec Support non-ASCII GNUPGHOME environment variable
- 4088eb4 repo: Cleaned up pylint/pep8 violations
- 5553628 repo: Add check of REPO_URL env variable
- 745b4ad Fix gitc-init behavior
- d3ddcdb Ignore clone.bundle on HTTP 501, i.e. Not Implemented
Change-Id: I3f763ef0ec2df2d726dff429021b48ad474148f1
The constant prompting when registered hooks change can be tedious and
has a large multiplication factor when the project is large (e.g. the
AOSP). It gets worse as people want to write more checks, hooks, docs,
and tests (or fix bugs), but every CL that goes in will trigger a new
prompt to approve.
Let's tweak our trust model when it comes to hooks. Since people start
off by calling `repo init` with a URL to a manifest, and that manifest
defines all the hooks, anchor trust in that. This requires that we get
the manifest over a trusted link (e.g. https or ssh) so that it can't
be MITM-ed. If the user chooses to use an untrusted link (e.g. git or
http), then we'll fallback to the existing hash based approval.
Bug: Issue 226
Change-Id: I77be9e4397383f264fcdaefb582e345ea4069a13
During sync, repo runs `git read-tree --reset -u -v HEAD` which causes
git-lfs's smudge filter to run. However this fails because git-lfs does
not work with bare repositories.
Add lfs.filter configuration to the project config as suggested in the
comments on the upstream git-lfs client issue [1]. This prevents the
smudge filter from running, and the sync completes successfully.
For any projects that have LFS objects, `git lfs pull` must be executed.
[1] https://github.com/github/git-lfs/issues/1422
Bug: Issue 224
Change-Id: I091ff37998131e2e6bbc59aa37ee352fe12d7fcd
Re-ordered to first create the symlink before checking the source
file and remove the destination if the source does not exists.
Change-Id: Iae923ba2ef0ba5a8dc1b8e42d8cc3f3708f773af
If upstream string is empty, current_branch_only variable will be assigned
to an empty string.
This is not what we expect here as this variable is a boolean.
Change-Id: Ibba935e25a74c2be1e50c88b4b403cf394ba365e
When sync-s="true" option is used, the checkout of a submodule will try
to use the revision attribute of the parent project.
If this revision is a named reference, the checkout will fail if there
is no reference with this name in the submodule.
The proposed solution is to use the git commit id as revisionExpr for
submodules.
Change-Id: Ie8390a11957fd6a9c61289c6861d13cb3fa11678
Adds the hook-scripts to .gitattributes due to the shell-scripts not
liking CRLF which they will get if a user sets 'autocrlf = true'
in their global gitconfig.
Further, since the python interpreter can handle either CRLF or LF,
python-scripts specific line-ending rules have been removed.
Change-Id: I2d6bfd491b2f626b9ca93c40a3a7f2cfba6c54f0
This ignores whitespaces errors, which we have quite a few of in argument
lists, for example:
************* Module git_config
C:209, 0: No space allowed around keyword argument assignment
def HasSection(self, section, subsection = ''):
^ (bad-whitespace)
C:320, 0: No space allowed around keyword argument assignment
capture_stdout = True,
^ (bad-whitespace)
C:321, 0: No space allowed around keyword argument assignment
capture_stderr = True)
^ (bad-whitespace)
C:427, 0: Exactly one space required after comma
'-o','ControlPath %s' % ssh_sock(),
^ (bad-whitespace)
C:436, 0: Exactly one space required after comma
check_command = command_base + ['-O','check']
^ (bad-whitespace)
C:464, 0: Exactly one space required after comma
% (host,port, str(e)), file=sys.stderr)
^ (bad-whitespace)
C:707, 0: No space allowed around keyword argument assignment
return self._config.GetString(key, all_keys = all_keys)
^ (bad-whitespace)
C:759, 0: No space allowed around keyword argument assignment
return self._config.GetString(key, all_keys = all_keys)
^ (bad-whitespace)
Change-Id: Ia8f154f6741ce609787551f65877d7584c457903
Signed-off-by: Stefan Beller <sbeller@google.com>
When the alias attribute is set for a remote, the RemoteSpec attached to
a Project only contains the alias name used by git, not the original
name used in the manifest. But that's not enough information to
reconstruct the manifest, so save off the original manifest name as
another RemoteSpec parameter, only used to write the manifest out.
Bug: Issue 181
Bug: Issue 219
Change-Id: Id7417dfd6ce5572e4e5fe14f22924fdf088ca4f3
Mention that the RPC endpoints are used when running repo
sync with the --smart-sync and --smart-tag options
Change-Id: I4b0b82e8b714fe923a5b325a6135f0128bf636ff
The repo script allows a manifest to specify a '.' as the path the
top-level directory, which co-locates the .git and .repo directories,
and places files from the git repository at the top-level:
<project name="proj_name" path="." />
<project name="sierra.other.git" path="other" />
Most commands work correctly with this setup. Some commands, however,
fail to find the project. For instance, 'repo sync' works, and 'repo sync .'
works in a sub-project ('other' in this case) but 'repo sync .' in the
top-level directory fails with the error:
error: project . not found
There are two reasons for this:
1. The self.worktree attribute of the Project object is not normalized,
so with a '.' for path its value would be '/my/project/root/.'. This is
fine when used as a path, since it's the same path as '/my/project/root',
but when used in a string comparison it fails. This commit applies
os.path.normpath() to that value before storing it.
2. The _GetProjectByPath method in command.py was not checking the path
against manifest.topdir, so even once it was normalized the project was
not found. This commit adds a check against manifest.topdir if the
loop drops out without finding a project.
Change-Id: Ic84d053f1bbb5a357cad566805d5a326ae8246d2
We weren't copying these lists, so the += was actually changing the
underlying lists.
When a new project was added to the manifest, we run _CheckDirReference
against the manifest project with share_refs=True, which added the
working_tree_* to the shareable_* lists. Then, when we load the new
manifest and create the new project, it uses the lists that already
contain the working_tree_* files, even though we passed
share_refs=False.
This happens reliably under the above conditions, but doesn't seem to
happen when syncing a fresh tree. So we've got a mixture of links that
may need to be cleaned up later. This patch will just stop it from
happening in the future.
Change-Id: Ib7935bfad78af1e494a75e55134ec829f13c2a41
Make it possible to exclude projects using regex/wildcard.
The syntax is similar to that of the -r option, e.g.:
repo forall -i ^platform/ ^device/ -c 'echo $REPO_PROJECT'
Change-Id: Id250de5665152228c044c79337d3ac15b5696484
A common design pattern is to use __file__ to find the location of the
active python module to assist in output or loading of related assets.
The current hook systems runs the pre-upload.py hook in a context w/out
that set leading to runtime errors:
$ repo upload --cbr .
ERROR: Traceback (most recent call last):
File ".../repo/project.py", line 481, in _ExecuteHook
self._script_fullpath, 'exec'), context)
File ".../repohooks/pre-upload.py", line 32, in <module>
path = os.path.dirname(os.path.realpath(__file__))
NameError: name '__file__' is not defined
Define this variable in this context so code can safely use it.
Change-Id: If6331312445fa61d9351b59f83abcc1c99ae6748
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a few violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the command.py script.
This script now has a pylint score of 10.0.
Change-Id: Ibb35fa9af0e0b9b40e02ae043682b3af23286748
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a number of violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the project.py script.
This script now has a pylint score of 10.0, and no violations reported
by pep8.
Change-Id: I1462fd84f5b6b4c0dc893052671373e7ffd838f1
It was only displaying 'Project list error: GitError()'
without any useful info about the project nor the error
Change-Id: Iad66cbaa03cad1053b5ae9ecc90d7772aa42ac13
This commit adds additional instructions on getting patches submitted,
based on my recent experience doing so.
Change-Id: I8e0d37d316214cc9a39383414773aad181f83f18
I noticed when running pylint (as the SUBMITTING_PATCHES file directs)
that there were a number of violations reported. This makes it difficult
to see violations I might have introduced. This commit corrects all
pylint violations in the repo script.
First I ran this to clean up the formatting:
autopep8 --max-line-length=80 --indent-size 2 repo
Following that the following violations remained:
% pylint --rcfile=.pylintrc repo
************* Module repo
W:220,21: Redefining name 'init_optparse' from outer scope (line 156)
(redefined-outer-name)
W:482, 2: No exception type(s) specified (bare-except)
C:704, 0: Old-style class defined. (old-style-class)
For line 220, the parameter to _GitcInitOptions was renamed so as not to
mask the init_optparse global.
For line 482, a pylint directive was added to disable the bare-execpt
violation for just that line.
For line 704, the _Options class was changed to subclass object.
Additionally, the comments at lines 107-113 were spaced out to line up
with the comment at line 112 that autopep8 moved.
This script now has a pylint score of 10.0
Change-Id: I779b66eb6b061a195d3c4372b99dec1b6d2a214f
We want to be able to run repo on a system that is not connected to
the Internet and cannot access https://gerrit.googlesource.com. We
can put a clone of that repos there, but would prefer to use the
stable version of the repo script instead of a locally modified
version.
This commit adds a check for the REPO_URL environment variable. If
that is set and not empty its value will be set in the REPO_URL
global in repo. Otherwise the standard path will be used.
Change-Id: I0616f5f81ef75f3463b73623b892cb5eed6bb7ba
As soon as we wrote the gitc manifest, the folder for that repo became
empty, causing the next GetProjects lookup to fail. Reorder the
GetProjects calls so that they all happen while we still have the
repository contents available.
If you were already in a subdir, for cases like 'repo start <branch> .',
this would still fail, since the working directory would disappear out
from under you. That's fine most of the time, since we shouldn't be
doing operations based on the local directory, but git has a realpath
function that tries to restore CWD by chdir'ing back to it. So if the
working directory no longer exists, chdir to the topdir before
continuing.
Change-Id: Ibdf6cd37ff6e5a5f8338347c3919175491f7166f
Some teams have a continuous build server that would mark certain
manifest green and safe to sync to. Then team members could repo
sync to that particular manifest file and make sure they always
sync to a green build. But if she/he has some local changes and
wants to rebase, currently it would be a manual process to find the
correct version to rebase onto. This patch helps with that use
case by automating the process to rebase onto the currently synced
manifest version.
Change-Id: I847c9eb6addf7f84fd3f5594fbf8c0bcc103f9a5
We don't really use HEAD much in the bare git repositories, but there
have been reports of errors in git-symbolic-ref:
symbolic-ref: fatal: Refusing to point HEAD outside of refs/
That happen when the bare git repo is in the detached head state. It's
possible that previous operations were killed while we were pruning
branches.
Use DetachHead instead of SetHead if we're restoring the repo into a
detached head state.
Change-Id: I9062e8957bc70367d3ded399685ac026fbb421fc
When repo sync is used with -f (--force-error) and a project fails to
sync, the sync will continue but then exit with an error status.
However if -n (--network-only) is also used, the exit code is 0, even
when a project failed.
Modify the logic to make sure the sync exits with the correct status.
Bug: Issue 214
Change-Id: I0b5d97a34642c5aa3743750ef14a42c9d5743c1d
See git commit 33cfccbbf35a -- some protocols allow arbitrary command
execution as part of the URL. Instead of blindly allowing those,
whitelist the allowed URL protocols unless the user has already done so.
Bug: Issue 210
Change-Id: I6bd8e721aa5e3dab53ef28cfdc8fde33eb74ef76
If a linkfile is a broken link (destination does not exist), and it
needs to be updated, we didn't notice that it needed to be removed
first. Use lexists instead of exists to check for this condition.
Change-Id: I1f6a1f0193d3fd2b9f7a647836044997f6ab32eb
By passing --prune to the sync command, the --prune option is
given to the `git fetch`, causing refs that no longer exist on
the remote to be removed.
Change-Id: I3cedacce14276d96ac2d5aabf2d07fd05e92bc02
Adds windows as one of the allowed platforms flags.
Fixes -p foo to append 'platform-foo', instead of each letter (list.extend
expects a list and thus appends each char in the string, rather than the
string itself).
Change-Id: I73a92127ac29a32fc31b335cc54a246302904140
With gitc-init, a gitc client may be specified using '-c'. If we're
not currently in that client, we need to change directories so that
we don't affect the local checkout, and to ensure that repo is
checked out in the new client.
This also makes '-c' optional if already in a gitc client, to match
the rest of the init options.
Change-Id: Ib514ad9fd101698060ae89bb035499800897e9bd
The .gitcookies file generated by googlesource.com does not have
the header:
# (Netscape) HTTP Cookie File
which causes python's MozillaCookieJar.load to fail with the
error:
"does not look like a Netscape format cookies file"
Prepend the expected header onto the generated cookie file.
We don't bother to check if the header already exists on the
file; repeating it does not cause any problem.
Bug: Issue 207
Change-Id: I7d39720a1d36a6aae00f70691156514ebc04e579
repo gitc-delete deletes a GITC client and all the locally
saved sources. Useful for removing unnecessary clients and
recovering disk space.
Change-Id: Idf23addcea52b8713d268c34a7b37da0c5e5cd26
These won't show up as common commands in the help text unless in a GITC
client, and will refuse to execute.
Change-Id: Iffe82adcc9d6ddde9cb4b204f83ff018042bdab0
The "copyfile" element is available since 2009 and
have been used in every Android manifest; the "linkfile"
element is available since 2014.
Now it's a good time to add both to the documentation
Change-Id: Ia987edf5f69a006235fbd3f33b744e9794a6d964
Signed-off-by: Ruslan Bilovol <ruslan.bilovol@gmail.com>
This way any changes made to the main manifest are reflected in the gitc
manifest. It's also necessary to use both manifests to sync since the
information required to update the gitc manifest is actually in the repo
manifest.
This also fixes a few issues that came up when testing. notdefault
groups weren't being saved to the gitc manifest in a method that matched
'sync'. The merge branch wasn't always being set to the correct value
either.
Change-Id: I435235cb5622a048ffad0059affd32ecf71f1f5b
This way any changes made to the main manifest are reflected in the gitc
manifest. It's also necessary to use both manifests to sync since the
information required to update the gitc manifest is actually in the repo
manifest.
This also fixes a few issues that came up when testing. notdefault
groups weren't being saved to the gitc manifest in a method that matched
'sync'. The merge branch wasn't always being set to the correct value
either.
Change-Id: I5dbc850dd73a9fbd10ab2470ae4c40e46ff894de
The source or destination attributes may have changed even if the source
didn't, so we need to make sure that these are up to date.
Change-Id: I266ef3598ddda7e8c23bc9c6a049905ddc586348
If the clone.bundle is out of date, repo may be installed with an old
version. It will upgrade with the next sync a day later, or when "repo
selfupdate" is run.
This behavior was added to normal project downloads, but was never added
to the repo launcher.
Change-Id: Ib04bef3a658c98fe1b6c53b3e8d0067165a5e3f7
Updates the repo launcher and gitc_utils to pull the manifest
directory location out of the gitc config file.
Change-Id: Id08381b8a7d61962093d5cddcb3ff6afbb13004b
The GITC filesystem does not understand relative URLs for remotes,
so now if a remote uses a relative URL, it will be be expanded to
be relative to the manifest URL.
Change-Id: Ie1210758560aeb1934da3f71496aaf19c2728214
Add repo start support for GITC checkouts. If the user is in
the GITC FS view, they can now run repo start to check out
the sources and create a new working branch.
When "repo start" is called on a GITC project, the revision
tag is set to an empty string and saved in a new tag:
old-revision. This tells the GITC filesystem to display the
local copy of the sources when being viewed. The local copy
is created by pulling the project sources and the new branch
is created based off the original project revision.
Updated main.py to setup each command's gitc_manifest when
appropriate.
Updated repo sync's logic to sync opened projects and
updating the GITC manifest file for the rest.
Change-Id: I7e4809d1c4fc43c69b26f2f1bebe45aab0cae628
If a hook file has been modified locally, it will not be replaced.
Improve the message to make this clearer.
Also change it from an error to a warning.
Change-Id: I62c635390f24d2868db17717c247861b0381c99f
Use the _error method instead of directly calling `print`.
Also add a new _warn convenience method.
Change-Id: Ia332c14ef8d9d1fe2df128dbf36b5521802ccdf1
Don't emit a message when the netrc file doesn't exist or couldn't
be opened.
Instead of trying to unpack the result of info.authenticators() and
catching the resulting TypeError when it's None, first store it to
a local and only unpack it if it has a value.
Also remove an unused import.
Change-Id: I5c404d91e48c261c1ab850c3e5f040c4f4c235cb
Use the same cookies and proxy that git traffic goes through for
persistent-http[s] to support authentication for smart-sync.
Change-Id: I20f4a281c259053a5a4fdbc48b1bca48e781c692
Passing the force_sync variable into the string formatting results in
the message:
"Retrying clone after deleting None"
or
"Retrying clone after deleting True".
Pass the name of the git directory instead.
Also, move the print inside the if-block so it's only displayed
when the retry is actually going to be attempted.
Change-Id: I76d9ecc176cecee4ad512d13e9d1f6bd36aacbbb
Add repo sync support for GITC checkouts. If the user is in the
GITC client directory they can still pull the sources as normal
if they pass in the --force-gitc argument. Otherwise the user
should call repo sync in the GITC view to update the user's
remote view. (This works because .repo in the GITC view will
link to .repo in the client config directory.)
Part of the support for this change is the refactoring of GITC
related code into gitc_utils.py.
Change-Id: I2636aaa50b450b6f091309db8dd0e8f4dbdad579
This argument wasn't being copied, which caused syncs from generated
manifests to pull down too much of the git history.
Change-Id: I269bab788d4557267c081628b3f8c6aec7744e81
Adds the new gitc-init command to set up a GITC client. Gitc-init
sets up the client directory and calls repo init within it. Once
the repo is initialized, then generates a GITC manifest file
by using git ls-remote on each project and retrieving the HEAD SHA
to use as the revision attribute.
Gitc-init inherits from and has all the options as repo init.
Change-Id: Icd7e47e90eab752a77de7c80ebc98cfe16bf6de3
For projects that have been cloned outside of the repo command (or
cloned a long time ago), commit abaa7f312f
introduced an error message to invite the user to use --force-sync.
However, due to the risk of data loss, it's useful to know which
project's git directory is being replaced before deciding whether or not
to provide --force-sync.
This change updates the exception's associated value to include the
project's relative path and explain to the user how they can resolve the
issue. A previous version of this commit used the project name. However,
for projects that have multiple work trees, the name can be ambiguous,
while the path clearly identifies which git directory will be replaced.
Change-Id: If717e66fda4d19accc0a8e889a91f4cd4ff14dff
The existing code here makes sure that switching clone-depth from on to
off actually causes the history to be fully restored. Unfortunately, it
does this by fetching the full history every time the fetch spec
changes. Switching between two clone-depth="1" branches will fetch far
more than the top commit.
Instead, when not using clone-depth, pass --depth=2147483647 to git
fetch so that it ensures that we have the entire history. That is
slightly less efficient, so limit it to only when there are shallow
objects in the project by checking for the existance of the 'shallow'
file.
Change-Id: Iee0cfc9c6992c208344b1d9123769992412db67b
Previously repo would only print the failing project path if
Sync_NetworkHalf returned false/empty, but if it threw an
exception the print() was never called.
Change-Id: I58c41de43930df5e34b21561c205e062a72e290f
This fixes these errors:
...
File ".repo/repo/project.py", line 2371, in _ReferenceGitDir
os.symlink(os.path.relpath(src, os.path.dirname(dst)), dst)
OSError: [Errno 17] File exists
Which was happening for checkouts that were created before v1.12.8, when
project-objects was created. Nothing had yet been forcing these
checkouts to use project-objects, until the recent verification changes.
In this OSError case, we already created the symlink, so src == dst, and
the directory did not exist. This caused us to run os.makedirs the
os.symlink on the same file.
dst really should be the file in gitdir, not the target of that symlink
if it exists. So just use realpath for the dotgit portion of the path.
Change-Id: Iff5396a2093de91029c42cf38aa57131fd22981c
Enable operating against groups of repositories. As it stands, it isn't
compatible with `-r/--regex`.
`repo forall -g groupname -c pwd` will run `pwd` for all projects in
groupname.
`repo forall -g thisgroup,-butnotthisone -c pwd` will run `pwd` for all
projects in `thisgroup` but not `butnotthisone`.
`repo list -g groupname -n` will list all the names of repos in
`groupname`.
Change-Id: Ia75c50ce52541d1c8cea2874b20a4db2e0e54960
In some cases, a user may wish to continue with a sync even though
it would require overwriting an existing git directory. This behavior
is not safe as a default because it could result in the loss of some
user data, but as an optional flag it allows the user more flexibility.
To support this, add a --force-sync flag to the sync command that will
attempt to overwrite the existing git dir if it is specified and the
existing git dir points to the wrong obj dir.
Change-Id: Ieddda8ad54e264a1eb4a9d54881dd6ebc8a03833
repo info will crash when using a manifest with no default element despite
default being an optional element. Output nothing for "Manifest Branch" if no
default element exists (or if no default revision exists).
Change-Id: I7ebffa2408863837ba980f0ab6e593134400aea9
If _InitGitDir fails, it leaves any progress it had made on the file
system. This can cause subsequent calls to repo sync to behave
differently. This is especially evident when _CheckDirReference() fails,
since it will not be invoked when sync is retried because both the
source and destination directories already exist.
To address this, have _InitGitDir() clean up any directories it has created
if it catches an exception. Also behave the same way for _InitWorkTree().
Change-Id: Ic16bb3feea649e115b59bd44be294e89e3692aeb
For some users it is not desirable to remove refs that don't exist
on the remote server when syncing a mirror repo.
This reverts commit b4d43b9f66.
Change-Id: Ie849b66682138ef88da6cd1a5fbb27e993197dd7
The fetch logic for the case where depth is set and revision is a
SHA1 has several failure modes that are not handled well by the
current logic.
1) 'git fetch <SHA1>' requires git version >= 1.8.3
2) 'git fetch <SHA1>' can be prevented by a configuration option on the server.
3) 'git fetch --depth=<N> <refspec>' can fail to contain a SHA1 specified by
the manifest.
Each of these cases cause infinite recursion when _RemoteFetch() tries to call
itself with current_branch_only=False because current_branch_only is set to
True when depth != None.
To try to prevent the infinite recursion, we set self.clone_depth to None
before the first retry of _RemoteFetch(). This will allow the Fetch to
eventually succeed in the case where clone-depth is specified in the manifest.
A user specified depth from the init command will still recurse infinitely.
In addition, never try to fetch a SHA1 directly if the git version being used
is not at least 1.8.3.
Change-Id: I802fc17878c0929cfd63fff611633c1d3b54ecd3
* changes:
forall: use smart sync override manifest if it exists
sync: Remove smart sync override manifest when not in smart sync mode
forall: Don't try to get lrev of projects in mirror workspace
sync: Improve error message when writing smart sync manifest fails
Previously, in running the `manifest` command, we wouldn't output the
upstream if the default upstream would include the pinned sha1.
However, now that fetching refs/heads/* doesn't guarantee that we will
have the sha1, we need to always output the specified upstream branch.
Change-Id: Ib8b409a8ecd439397b38ee9649c530407797f841
If a workspace is synced with the -s or -t option, the included projects
may be different to those in the original manifest. However, when using
the forall command, the list of the projects from the original manifest
is used.
If the smart sync manifest file exists, use it to override the original
manifest.
Change-Id: Iaefcbe148d2158ac046f158d98bbd8b5a5378ce7
When syncing with the -s or -t option, a smart_sync_override.xml file
is created. This file is left in the file system when syncing again
without the -s or -t option.
Remove the smart sync override manifest, if it exists, when not using
the -s or -t option.
Change-Id: I697a0f6405205ba5f84a4d470becf7cd23c07b4b
git rev-parse fails for projects that don't have an explicit revision
specified, and don't have a branch of the same name as the default
revision. This can be the case in a workspace synced with the smart
sync (-s) or smart tag (-t) option.
Change-Id: I19bfe9fe7396170379415d85f10f6440dc6ea08f
The error message only states that writing the manifest failed.
Include the exception message, so it's easier to track down the reason
that the write failed.
Change-Id: I06e942c48a19521ba45292199519dd0a8bdb1de7
After performing the actual cherry-pick operation, the code
in cherry_pick.py opens a pipe to 'git commit -F' to rewrite the commit
message, emits the fixed-up commit msg to the pipe, then waits
for 'git commit' to complete. The child 'git' process winds up
hanging while reading from the pipe, however, since the parent
process still has it open. To fix the hang, change the parent process
to close its end of the pipe after it has emitted the message.
Change-Id: I5929371e69a5b076f09009d00d40a2c72ac8ac33
output from a process is in bytes in python3. we need
to decode it.
in Python3, bytes don't have an encode attribute. use this
to identify it.
Change-Id: I152f2ec34614131027db680ead98b53f9b321ed5
This allows a project to use globs in the linkfile src attribute. When
a glob is used in the src the dest field must be a directory. Then
_LinkFile._Link(self) calls will create symbolic links in the dest
directory to all of the entries in the src as defined by the glob
specification.
Below all of the entries in master-configs/ will have symbolic links
in <root dir>/configs directory:
<project name="helloworld.git" path="apps/helloworld">
<linkfile src="master-configs/*" dest="configs"/>
</project>
Change-Id: Idfed8fa47c83d2ca6e2b8e867731b8e2f9e2eb47
The source (target) of the symlink is specified relative to a project
within a tree, and the destination is specified relative to the top
of the tree, so it should always be possible to create a relative symlink
to the target file. Relative symlinks will allow moving an entire tree
without breaking the symlink, and copying a tree (with -p) without leaving
a symlink to the old tree.
Change-Id: I16492a8b59a137d2abe43ca78e3b212e2c835599
Pressing ctrl-c during repo sync often hangs for 30 to 45 seconds
due to the time.sleep and retry in _RemoteFetch. If git exits with
a signal, for example -2 for SIGINT triggered by ctrl-c, skip the
sleep and retry.
Change-Id: I32da12c2dcc96d9cc0b12a066e824b12ebfb52a0
If the generator that produces per-project worker arguments raises an
exception it triggers python bug http://bugs.python.org/issue8296.
Rewrite the generator expression as a generator function, and catch
Exceptions and KeyboardInterrupts to end the iteration.
Also add a pool worker initializer to disable SIGINT to prevent
KeyboardInterrupts inside multiprocessing.Pool in the worker threads
causing the same problem.
Fixes easy-to-reproduce hangs when hitting ctrl-c during
repo forall -c echo
Change-Id: Ie4a65b3e1e07a64ed6bb6ff20f3912c4326718ca
There are a set of cases that can cause the git directory in
.repo/projects to point to a directory in .repo/project-objects that
is not the one specified in the manifest. This results in a tree that
is not sane, and so should cause a failure.
In order to reproduce the failure case:
1) Sync to any manifest
2) Change the 'name' of a project to a different repository. Leave the
'path' the same.
3) Resync the modified project. The project-objects directory will not
be created, and the projects directory will remain pointed at the old
project-objects.
Change-Id: Ie6711b1c773508850c5c9f748a27ff72d65e2bf2
In 2fb6466f79 an optimisation was
added to avoid fetching from remotes if the project is fixed to
a revision and the revision is already available locally.
This causes problems for users who expect all objects to be
fetched by default.
Change the logic so that the optimized behaviour is only enabled if
an option is explicitly given to repo sync.
Change-Id: I3b2794ddd8e0071b1787e166463cd8347ca9e24f
When syncing a mirror repo, add the --prune option to the fetch
command to force removal of stale refs from the mirror.
Change-Id: I4b43b2a5c86b9915627887c16f6569066f3ab978
Previously, we used a regex that would only remove a phony string from
a url if it existed, but we recently replaced that with a slice. This
change goes back to the previous behavior.
Change-Id: I8baf527be01c4b49d45b903b31a1cd6315563d5b
Appending the branch to the fetch spec causes sync of a mirror to
fail for projects that don't have an explicit revision specified,
and don't have a branch of the same name as the default revision.
For example, a manifest defining a default revision:
<default revision="master">
having a project without an explicit revision:
<project name="path/to/project">
and not having a branch named "master", will cause repo sync to
fail for that project with the error:
Couldn't find remote ref refs/heads/master
Modify the logic to not append the branch onto the fetch spec when
syncing to a mirror.
Change-Id: I5c4457bd125519abf27abe682dea62ad708978c9
Before, a list was generated, which is why there was a massive delay.
Using a generator will allow processes to start straight away
Change-Id: Ia325b0b340cc328c08c9bcc92a6709bbdaf6a664
buflist was being used, which isn't available in Python 3.
`Execute` was using StringIO to capture the output of `PrintWorkTreeStatus`,
only to redirect it straight to stdout.
Instead, just let `PrintWorkTreeStatus` do it's own thing directly to stdout.
for handling `_FindOrphans`, we swap StringIO for a list. Nothing was done
that needed a a file like object.
Change-Id: Ibdaae137904de66a5ffb590d84203ef0fe782d8b
When running repo branch, the git merge line (in many circumstances)
is set to the revision of the project specified in the manifest. If
this is a branch name that is not fully-qualified, we will end up with
something like "merge = master" instead of "merge = refs/heads/master".
This change examines the revision if we are going to use that and
changes branch short names to fully qualified branch names.
Change-Id: Ie1be94fb8d45df8eeac44a47f729a3819a05fa81
Instead of using regex, append the netloc and relative
scheme lists with the custom scheme.
The schemes will only be appended when needed, instead
of passing X amount of regex replaces.
see http://bugs.python.org/issue18828 for more details.
Change-Id: I10d26d5ddc32e7ed04c5a412bdd6e13ec59eb70f
Switch the GitCommand program to always capture the output for stdout
and stderr. And by default print the output while running.
The options capture_stdout and capture_stderr have effectively become
options to supress the printing of stdout and stderr.
Update the 'git fetch' to use '--progress' so that the progress messages
will be displayed. git checks if the output location isatty() and if it
is not a TTY it will by default not print the progress messages.
Change-Id: Ifdae138e008f80a59195f9f43c911a1a5210ec60
This reverts commit 38e4387f8e.
A "repo init" followed by "repo sync" is meant to be as safe as
"git clone". In particular it should not run arbitrary code provided
by the manifest owner.
It would still be nice to have support for manifest-defined git hooks
--- they'd just need a prompt like the upload RepoHook has. Hopefully
a later change can bring them back.
Change-Id: I5ecd90fb5c2ed64f103d856d1ffcba38a47b062d
Signed-off-by: Jonathan Nieder <jrn@google.com>
If the server returns HTTP 401 (unauthorized) when attempting to
download clone bundle files, ignore it and continue, rather than
exiting with a fatal error.
Change-Id: I2c7ee03e149c354c7e4ad6ea1ebf266534778fe1
Currently, paths longer than 39 chars have no space after them so it looks
like this:
project path/branch master
Change-Id: I4c1bb13648ac099ade8a8d4ebafa04131571f842
As a fix for issue #149, this patch add support for the remote '.'
(local).
As an alias for the local repository, remote '.' is lacking a fetch =
config in .git/config.
Without such refspec, repo info --overview is not able to process a
local tracking branch.
v2: Check for name == '.' before checking if merge starts with refs/,
since the case where it's not is invalid.
Signed-off-by: Yann Droneaud <ydroneaud@opteya.com>
Signed-off-by: Filipe Brandenburger <filbranden@google.com>
Change-Id: I8c8fd8602cd68baecb530301ae41d37d751ec85d
The stderr output generated by git during a RemoteFetch was not being
printed. This information is useful so print it.
Change-Id: I6e6ce12c4a57e5ca2359f76ce14f2fcbbc37a5ef
If the repo project is synced with partial depth, then the tags
won't be fetched and users will be told the newest sha1 in the
stable branch isn't signed.
Change-Id: I107df97b4836b928c76aa33a700fa35d1705ae09
Handle the case when this error occurs:
error: some local refs could not be updated; try running
'git remote prune origin' to remove any old, conflicting branches
This is usually caused by a reference getting changed from a file to a
directory.
For example:
Initially someone creates a branch 'foo' and it is stored as:
.git/refs/remotes/origin/foo
Then later on it is decided to change the layout structure where 'foo'
is a directory with branches below it:
.git/refs/remotes/origin/foo/master
The problem occurs when someone still has
'.git/refs/remotes/origin/foo' on their system and does a repo sync.
When this occurs the error message for needing to do a
'git remote prune origin' occurs.
Now when doing a 'git fetch' if the error message from git says that a
'git remote prune' is needed, it will do the prune and then retry the
fetch.
Change-Id: I4c6f5aa6bd932f0ef7a39134400bedd52e82f633
Signed-off-by: John L. Villalovos <john.l.villalovos@intel.com>
When working within a team or corporation it is often
useful/required to use predefined git templates. This
change teaches repo to use a per-remote git hook template
structure.
The implementation is done as a continuation of the
existing projecthook functionality. The terminology is
therefore defined as projecthooks.
The downloaded projecthooks are stored in the .repo
directory as a metaproject separating them from the users
project forest.
The projecthooks are downloaded and set up when doing a
repo init and updated for each new repo init.
When downloading a mirror the projecthooks gits are
not added to the bare forest since the intention is to
ensure that the latest are used (allows for company policy
enforcement).
The projecthooks are defined in the manifest file in the
remote element as a subnode, the name refers to the
project name on the server referred to in the remote.
<remote name="myremote ..>
<projecthook name="myprojecthookgit" revision="myrevision"/>
</remote>
The hooks found in the projecthook revision supersede
the stock hooks found in repo. This removes the need for
updating the projecthook gits for repo stock hook changes.
Change-Id: I6796b7b0342c1f83c35f4b3e46782581b069a561
Signed-off-by: Patrik Ryd <patrik.ryd@stericsson.com>
Signed-off-by: Ian Kumlien <ian.kumlien@gmail.com>
Currently, we only use the depth flag when cloning. The result is that when
new project history has merges, the entire history of the merged branch is
brought in and the project becomes unshallow very quickly. --depth and
clone-depth are often used to save on space, not just network load, so this
seems less than ideal.
This change uses --depth on every fetch (when the user has depth specified),
not just the initial clone. The result is that the given project stays
consistently shallow as opposed to growing over time, especially when merges
are involved.
Change-Id: Iac706cfdad4a555c72f9d9f1119195d38d91df12
When doing a shallow checkout SHA1 pinned repos with repo init --depth=1 and
repo sync -c, repo would try to fetch only some reference and fail if the exact
SHA1 repo was missing.
Instead, when depth is set, fetch only the specific commit.
Change-Id: If3f799d0e78c03faea47f796380bb5e367b11998
We currently delete all hooks in .git/hooks for each project before
symlink'ing in the standard project hooks. This can be annoying for
users who have installed custom git hooks.
There's no reason to delete all existing hooks. Just rip out the
deletion code.
Change-Id: I5062a6cd20af700f6d6a17b11ad6c94853987c57
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The persistent proxy may choose to present a per-process cookie file
that gets cleaned up after the process exits, to help with the fact
that libcurl cannot save cookies atomically when a cookie file is
shared across processes. We were letting this cleanup happen
immediately by closing stdin as soon as we read the configuration
option, resulting in a nonexistent cookie file by the time we use the
config option.
Work around this by converting the cookie logic to a context manager
method, which closes the process only when we're done with the cookie
file.
Change-Id: I12a88b25cc19621ef8161337144c1b264264211a
If you want to turn off colors for commands, you have to manually adjust
the git config settings (in various locations). If you're writing scripts
though, you often don't want to modify those locations. Add a commandline
option to explicitly control things.
The default behavior is unchanged -- we still scan the config files.
Change-Id: I54a3fd8e1918bac180aadd7c7d3004f069b02522
The invalid clone.bundle file warning is not typically user actionable,
and can be confusing. So don't show it when -q flag is in effect.
Change-Id: If9fef4085391acf54b63c75029ec0e161c38eb86
This reverts commit 565480588d.
We are reverting this change for 2 reasons:
1) It introduced a bug for users using sync -c with a reference mirror.
2) The fetch specs have recently changed to cause git to properly fail
when we request a non-existent branch of a manifest, removing the need
for this change.
Change-Id: I0f63da9bfb40cf5ffafb7979f1b8c929a738fc7b
When there are uncommitted files in the tree, 'repo upload' stops to
ask if it is OK to continue, but does not report the actual names of
uncommitted files.
This patch adds plumbing to have the outstanding file names reported
if desired.
BUG=None
TEST=verified that 'repo upload' properly operates with the following
conditions present in the tree:
. file(s) modified locally
. file(s) added to index, but not committed
. files not known to git
. no modified files (the upload proceeds as expected)
Change-Id: If65d5f8e8bcb3300c16d85dc5d7017758545f80d
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Signed-off-by: Vadim Bendebury <vbendeb@google.com>
While not typical, some users might have an upstream that isn't in
the usual refs/heads/* namespace. There's no reason not to use
those refs as the value for the upstream attribute, so support
doing so.
Change-Id: I5b119f1135c3268c20e7c4084682e860d3ee1fb1
'upstream' attribute is now transferred to the new manifest xml
that is created when using command 'repo manifest -o -'.
Manifest help is updated for the attributes 'sync-c','sync-s' and
'sync-j'.
Bug: Issue 164
Change-Id: If63f781e91d25c5b5b5ea0696b0c04337b0a686a
When current is "split" (i.e. some projects are current while others are not):
- Disable 'not in' printout (i.e. will print out all projects)
- Disable printing of multiple projects on one line
- Print current projects in green, non-current in white
Since using color to differentiate current from non-current in "split" cases:
- In non-split cases also print out project names in color (green for current
white for non-current)
Change-Id: Ia6b826612c708447cecfe5954dc767f7b2ea2ea7
Enable '--jobs' ('-j') option in the forall subcommand. For -jn
where n > 1, the '-p' option can no longer guarantee the
continuity of console output between the project header and the
output from the worker process.
SIG_INT is sent to all worker processes upon keyboard interrupt
(Ctrl+C).
Bug: Issue 105
Change-Id: If09afa2ed639d481ede64f28b641dc80d0b89a5c
Currently, if a local manifest wants to add groups to an existing
project, it must use remove-project and then re-add the project with
the new groups. This makes the local manifest more fragile, requiring
updates to the local manifest if the original manifest changes.
Add a new extend-project tag, which supports adding groups to an
existing project.
Change-Id: Ib4d1352efd722a65dd263d02644b9ea5ab6ed400
If a user reinits to a different manifest or the manifest updates so
that a project no longer has a fixed depth, we need to use --unshallow
when we fetch.
Change-Id: I6d3f15e5464b5eaad9205654bc24354947a78aea
Use JSON as it is shown to be much faster than pickle.
Also clean up the loading and saving functions.
Change-Id: I45b3dee7b4d59a1c0e0d38d4a83b543ac5839390
Some projects use multiple remotes.
In some cases these remotes have different naming conventions.
Add an option to define a revision in the remote configuration.
The `project` revision takes precedence over `remote` and `default`.
The `remote` revision takes precedence over `default`.
The `default` revision acts as a fall back as it originally did.
Change-Id: I2b376160d45d48b0bab840c02a3eef1a1e32cf6d
iterator.next() was replaced with iterator.__next__() in Python 3.
Use next(iterator) instead which will select the correct method for
returning the next item.
Change-Id: I6d0c89c8b32e817e5897fe87332933dacf22027b
Only warn about using Python 3 when running the repo script directly.
This prevents the user being warned twice.
Change-Id: I2ee51ea2fa0127ea310598320e460ec9f38c6488
dict.keys() produces a dict_keys object in Python 3, which does
not support .sort(). Use sorted() which will give the same outcome.
Change-Id: If6b33db07a31995b4e44959209d08d8fb74ae339
dict.values() produce dict_values objects rather than list objects.
Convert this to a list to maintain functionality with certain functions.
Change-Id: Ie76269e19f8d68479a1d7ae03aa965252d759a9e
A recent optimization (2fb6466f79) skips
performing a remote fetch if we already know we have the sha1 we want.
However, that optimization skipped initialization steps that ensure HEAD
points to the correct sha1. This change makes sure not to skip those
steps.
Here is an example of how to test this change:
"""""""""
url=<manifest url>
branch1=<branch name>
branch2=<branch name>
project=<project with revision set to different sha1 in each branch>
repo init -u $url -b $branch1 --mirror
repo sync $project
first=$(cd $project.git; git rev-parse HEAD)
repo init -b $branch2
repo sync $project
second=$(cd platform/build.git; git rev-parse HEAD)
if [[ $first == $second ]]
then
echo 'problem!'
else
echo 'no problem!'
fi
"""""""""
This fixes the bug that kept clients from doing things like `git log`
in projects using the clone-depth feature.
Change-Id: Ib4024a7b82ceaa7eb7b8935b007b3e8225e0aea8
It's just like copyfile and runs at the same time as copyfile but
instead of copying it creates a symlink instead. This is needed
because copyfile copies the target of the link as opposed to the
symlink itself.
Change-Id: I7bff2aa23f0d80d9d51061045bd9c86a9b741ac5
If a manifest includes projects with a clone-depth=1 attribute, and a
workspace is initialised from that manifest using the --mirror option,
any workspaces initialised and synced from the mirror will fail with:
fatal: attempt to fetch/clone from a shallow repository
on the projects that had the clone-depth.
Ignore the clone-depth attribute when fetching from the remote to a
mirror workspace. Thus the mirror will be synched with a complete
clone of all the repositories.
Change-Id: I638b77e4894f5eda137d31fa6358eec53cf4654a
For long-running forall commands sometimes it's useful to know which
iteration is currently running. Add REPO_I and REPO_COUNT environment
variables to reflect the current iteration count as well as the total
number of iterations so that the user can build simple status
indicators.
Example:
$ repo forall -c 'echo $REPO_I / $REPO_COUNT; git gc'
1 / 579
Counting objects: 41, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (19/19), done.
Writing objects: 100% (41/41), done.
Total 41 (delta 21), reused 41 (delta 21)
2 / 579
Counting objects: 53410, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (10423/10423), done.
Writing objects: 100% (53410/53410), done.
Total 53410 (delta 42513), reused 53410 (delta 42513)
3 / 579
...
Change-Id: I9f28b0d8b7debe423eed3b4bc1198b23e40c0c50
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Part of the cleanup path for _Init is removing the .repo
directory. However, _Init can fail before creating the .repo directory,
so trying to remove it raises another exception:
fatal: invalid branch name 'refs/changes/53/55053/4'
Traceback (most recent call last):
File "/home/mitchelh/bin/repo", line 775, in <module>
main(sys.argv[1:])
File "/home/mitchelh/bin/repo", line 749, in main
os.rmdir(repodir)
OSError: [Errno 2] No such file or directory: '.repo'
Fix this by only removing .repo if it actually exists.
Change-Id: Ia251d29e9c73e013eb296501d11c36263457e235
Currently, the --no-tags option is ignored if the user asks to only
fetch the current branch. There is no reason for this restriction. Fix
it.
Change-Id: Ibaaeae85ebe9955ed49325940461d630d794b990
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The old "manifest required for this command -- please run
init" is replaced by a more helpful message that lists the
command repo was trying to execute (with arguments) as well
as the str() of the NoManifestException. For example:
> error: in `sync`: [Errno 2] No such file or directory:
> 'path/to/.repo/manifests/.git/HEAD'
> error: manifest missing or unreadable -- please run init
Other failure points in basic command parsing and dispatch
are more clearly explained in the same fashion.
Change-Id: I6212e5c648bc5d57e27145d55a5391ca565e4149
For complex .gitconfig url rewrites, multiple insteadOf lines may be
used for a url. Search all of them for the right rewrite.
Change-Id: If5e9ecd054e86226924b0baf513801cd57c389cd
The `review.URL.autocopy` setting sends email notification to the
named reviewers, but does not add them as reviewer on the uploaded
change.
Add a new setting `review.URL.autoreviewer`. The named reviewers
will be added as reviewer on the uploaded change.
Change-Id: I3fddfb49edf346f8724fe15b84be8c39d43e7e65
Signed-off-by: bijia <bijia@xiaomi.com>
In existing workspaces where the manifest specifies a commit id in the
manifest, we can avoid doing a fetch from the remote if we have the
commit locally. This substantially improves sync times for fully
specified manifests.
Change-Id: Ide216f28a545e00e0b493ce90ed0019513c61613
Commit 8d201 "repo: Support multiple branches for the same project."
(Change id is I5e2f4e1a7abb56f9d3f310fa6fd0c17019330ecd) caused missing
mirroring manifest repository when 'repo sync' after 'repo init --mirror'.
When the function _AddMetaProjectMirror() is called to add two of
meta projects - git-repo itself and manifest repository to mirror,
it didn't add them into self._paths which has list of projects to be
sync'ed by 'repo sync'.
In addition, because member var of Project 'relpath' is used as a key
of self._paths, it should be set with proper value other than None.
Since this is only for meta projects which are not described in manifest
xml, 'relpath' is name of the projects.
Change-Id: Icc3b9e6739a78114ec70bf54fe645f79df972686
Signed-off-by: Kwanhong Lee <kwanhong.lee@windriver.com>
This command allows a deeper diff between two manifest projects.
In addition to changed projects, it displays the logs of the
commits between both revisions for each project.
Change-Id: I86d30602cfbc654f8c84db2be5d8a30cb90f1398
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
The fetch logic is now shared between the jobs == 1 and
jobs > 1 cases. This refactoring also fixes a bug where
opts.force_broken was not honored when jobs > 1.
Change-Id: Ic886f3c3c00f3d8fc73a65366328fed3c44dc3be
When we do an initial fetch and have not specified any branch etc,
the following fetch command will not error:
git fetch origin --tags +refs/heads/*:refs/remotes/origin/*
In this change we make sure something got fetched and if not we report
an error.
This fixes the bug that occurs when we init using a bad manifest url and
then are unable to init again (because a manifest project has been
inited with no manifest).
Change-Id: I6f8aaefc83a1837beb10b1ac90bea96dc8e61156
Currently if you run repo download -c on a change and the cherry-pick
runs into a merge conflict a Traceback is produced:
rob@rob-i5-lm ~/Programming/repo_test/repo1 $ repo download -c repo1 3/1
From ssh://rob-i5-lm:29418/repo1
* branch refs/changes/03/3/1 -> FETCH_HEAD
error: could not apply 0c8b474... 2
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
Traceback (most recent call last):
File "/home/rob/Programming/git-repo/main.py", line 408, in <module>
_Main(sys.argv[1:])
File "/home/rob/Programming/git-repo/main.py", line 384, in _Main
result = repo._Run(argv) or 0
File "/home/rob/Programming/git-repo/main.py", line 143, in _Run
result = cmd.Execute(copts, cargs)
File "/home/rob/Programming/git-repo/subcmds/download.py", line 90, in Execute
project._CherryPick(dl.commit)
File "/home/rob/Programming/git-repo/project.py", line 1943, in _CherryPick
raise GitError('%s cherry-pick %s ' % (self.name, rev))
error.GitError: repo1 cherry-pick 0c8b4740f876f8f8372bbaed430f02b6ba8b1898
This amount of error message is confusing to users and has the side effect
of the git message telling you the actual issue being ignored.
This change introduces a message stating that the cherry-pick couldn't
be completed removing the Traceback.
To reproduce the issue create a change that causes a conflict with one currently
in review and use repo download -c to cherry-pick the conflicting change.
Change-Id: I8ddf4e0c8ad9bd04b1af5360313f67cc053f7d6a
Gerrit no longer requires 'p/', and this causes unexpected behavior.
In this change we stop appending 'p/' to the urls.
Change-Id: I72c13bf838f4112086141959fb1af249f9213ce6
This commit implements a Kerberos HTTP authentication handler. It
uses credentials from a local cache to perform an HTTP authentication
negotiation using the GSSAPI.
The purpose of this handler is to allow the use Kerberos authentication
to access review endpoints without the need to transmit the user
password.
Change-Id: Id2c3fc91a58b15a3e83e4bd9ca87203fa3d647c8
Previously, we would remove 'persistent-' then tack it on at the end
if it had been previously found. However, this would ignore urljoin's
decision on whether or not the second path was relative. Instead, we
were always assuming it was relative and that we didn't want to use
a different absolute url with a different protocol.
This change handles persistent-https:// in the same way we handled the
absense of an explicit protocol. The only difference is that this time
instead of temporarily replacing it with 'gopher://', we use 'wais://'.
Change-Id: I6e8ad1eb4b911931a991481717f1ade01315db2a
The repo launcher version needs to be updated so some users can take
advantage of the more robust version number parsing.
Change-Id: Ibcd8036363311528db82db2b252357ffd21eb59b
'repo' and 'git_command.py' had their own git version parsing code.
This change shares that code between the modules. DRY is good.
Change-Id: Ic896d2dc08353644bd4ced57e15a91284d97d54a
This takes the wrapper importing code from main.py and moves it into
its own module so that other modules may import it without causing
circular imports with main.py.
Change-Id: I9402950573933ed6f14ce0bfb600f74f32727705
The last change regarding version parsing lost handling of version
hyphenation, this restores that. In otherwords,
1.1.1-otherstuff is parsed as (1,1,1) instead of (1,1,0)
Change-Id: I3753944e92095606653835ed2bd090b9301c7194
In CL:50715, I updated repo to handle multiple projects, but the
remove-projects code path was not updated accordingly. Update it.
Change-Id: Icd681d45ce857467b584bca0d2fdcbf24ec6e8db
the value of Manifest.projects has changed from being the dictionary
to the values of the dictionary. Here we handle this change
correctly on a PostRepoUpgrade.
From a `git diff v1.12.7 -- manifest_xml.py`:
+ @property
def projects(self):
self._Load()
- return self._projects
+ return self._paths.values()
self._paths does contain the projects according to this line of
manifest_xml.py:
484 self._paths[project.relpath] = project
Change-Id: I141f8d5468ee10dfb08f99ba434004a307fed810
Fetching a new branch on a shallow client may download the entire
project history, as the depth parameter is not passed to git
fetch. Force the fetch to only download the current branch.
Change-Id: Ie17ce8eb5e3487c24d90b2cae8227319dea482c8
This code checks whether a dir exists before creating it. In between the
check and the mkdir call, it is possible that another process will have
created the directory. We have seen this bug occur many times in
practice during our 'repo init' tests.
Change-Id: Ia47d39955739aa38fd303f4e90be7b4c50d9d4ba
The backtrace currently occurs when one uses the "--cbr" argument with
the repo upload subcommand if the current branch is not tracking an
upstream branch. There may be other cases that would backtrace as well,
but this is the only one I found so far.
Change-Id: Ie712fbb0ce3e7fe3b72769fca89cc4c0e3d2fce0
The docs on the annotations say that zero or more may exist as a child
of a project, so that means that a "*" instead of a "?" should be used.
Change-Id: Iff855d003dfb05cd980f285a237332914e1dad70
This significantly reduces sync time and used brandwidth as only
a tar of each project's revision is checked out, but git is not
accessible from projects anymore.
This is relevant when git is not needed in projects but sync
speed/brandwidth may be important like on CI servers when building
several versions from scratch regularly for example.
Archive is not supported over http/https.
Change-Id: I48c3c7de2cd5a1faec33e295fcdafbc7807d0e4d
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
If the top-level .repo directory is moved somewhere else (e.g. a
different drive) and replaced with a symlink, _InitHooks() will create
broken symlinks. Resolving symlinks before computing the relative path
for the symlink keeps the path within the repo tree, so the tree can
be moved anywhere.
Change-Id: Ifa5c07869e3477186ddd2c255c6c607f547bc1fe
If git-remote-persistent-https fails, we use an iter() and then
subsequently a .read() on stderr. Python doesn't like this and
gives the following error message:
ValueError: Mixing iteration and read methods would lose data
This change removes the use of iter() to avoid the issue.
Change-Id: I980659b83229e2a559c20dcc7b116f8d2476abd5
* Switching from python2 to python3 in the same workspace isn't
currently supported, due to a change in the pickle version (which
isn't supported by python2)
* Basic functionality does work with python3, however not everything
is expected to
Change-Id: I4256b5a9861562d0260b503f972c1569190182aa
* Add .decode('utf-8') where needed
* Add 'b' to `open` where needed, and remove where unnecessary
Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
git-repo uses 2 space indentation. A couple of recent changes
introduced 4 space indentation in some modules.
Change-Id: Ia4250157c1824c1b5e7d555068c4608f995be9da
It is often useful to be able to include the same project more than
once, but with different branches and placed in different paths in the
workspace. Add this feature.
This CL adds the concept of an object directory. The object directory
stores objects that can be shared amongst several working trees. For
newly synced repositories, we set up the git repo now to share its
objects with an object repo.
Each worktree for a given repo shares objects, but has an independent
set of references and branches. This ensures that repo only has to
update the objects once; however the references for each worktree are
updated separately. Storing the references separately is needed to
ensure that commits to a branch on one worktree will not change the
HEAD commits of the others.
One nice side effect of sharing objects between different worktrees is
that you can easily cherry-pick changes between the two worktrees
without needing to fetch them.
Bug: Issue 141
Change-Id: I5e2f4e1a7abb56f9d3f310fa6fd0c17019330ecd
Currently, an error is raised if more than one default is defined.
When including another manifest, it is likely that a default has
been defined in both manifests.
Don't raise an error if all the defaults defined have the same
attributes.
Change-Id: I2603020687e2ba04c2c62c3268ee375279b34a08
Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
Example:
- `repo init -b master` / sync a project
- In one project: `git checkout -b work origin/branch-thats-not-master`
- make some changes, `git commit`
- `repo upload .`
- Upload will now be skipped with a warning instead of being uploaded to
master
Change-Id: I990b36217b75fe3c8b4d776e7fefa1c7d9ab7282
Passing a project revisionExpr to UploadForReview will cause it to
try to push to refs/for/<sha> if the revision points to a sha
instead of a branch. Pass None for dest_branch if no destination
branch has been specified, which will cause UploadForReview to
upload to the merge branch.
There is room for further improvement, the user prompts will
still print "Upload project <project> to remote branch <sha>",
and then upload to the merge branch and not the sha, but that
is the same behavior that was in 1.12.2.
Change-Id: I06c510336ae67ff7e68b5b69e929693179d15c0b
Previously, change I7150e449341ed8655d398956a095261978d95870
had broken alias support in order to fix the manifest command to keep
it from spitting projects that point to an alias that wasn't recorded.
This commit reverts that commit and instead solves the issue more
correctly, outputting the alias in the remote node of the manifest and
respecting that alias when outputting the list of projects.
Change-Id: I941fc4adb7121d2e61cedc5838e80d3918c977c3
When uploading a change for review, we sometimes request /ssh_info to
get the correct port number for uploading via ssh (regardless of
whether or not we intend to upload over ssh).
If we have trouble accessing /ssh_info (for authentication reasons,
etc), we now press on under the assumption that we will upload via http
instead of aborting.
Change-Id: Ica6bbeac1279e89424a903529649b7f4af0b6937
Python 2.4 and 2.5 do not have a print_function available, so we need a
compatible print function for displaying an error message when the user
has an older version of Python.
Change-Id: I54d7297be98bb53970e873b36c6605e6dad386c3
This is to avoid the following AttributeError:
line 681, in _ParseProject
AttributeError: '_Default' object has no attribute 'destBranchExpr'
Change-Id: Ia9f7e2cce1409d22d71bc8a74b33edf2b27702ca
When the RPC call fails, the error message returned by the server
is printed, but it is not obvious that this is caused by RPC call
failure.
Prefix the error message with a descriptive message that explains
what went wrong.
Change-Id: I4b77af22aacc2e9843c4df9d06bf54e41d9692ff
When syncing using smart sync or smart tag mode, print the url of
the manifest server that is being used.
This is useful in organisations that have multiple manifest servers
used in different manifest branches.
Change-Id: Ib5bc2de5af6f4a942d0ef735c65cbc0721059a61
cco3@android.com has a new gpg key, so this needs to be updated in the
repo scripts so that he can sign updates.
Change-Id: I9f058263b35bd027502d6e3b814d7aeb801a1e6e
The command `repo upload --cbr -D <some branch>` will display
the default revision, and not the actual destination branch.
Fix that and display the branch to which the change will be
uploaded to.
Change-Id: I712ed0871c819dce6774c47254dac4efec4532e0
* Previously, it would run `git fetch <remote.name> <change refspec>
<remote.fetch>, which would fetch all the branches, even if 'sync-c'
was set to true in the manifest.
* Fix that, since all it needs to fetch is the change that was asked
for, and nothing else.
* For some more info, refer to the discussion on:
I42a9d419b51f5da03f20a640ea68993cda4b6500
Change-Id: Ibc801695d56fc16e56f999e0f61393f54461785f
* manifest_name was never set if opt.smart_sync or opt.smart_tag is used.
* Set it earlier, so that the code handles it correctly when it is None.
* An UnboundLocalError is raised if running `repo sync` without any options:
local variable 'manifest_name' referenced before assignment
* This fixes the above regression caused by commit
53a6c5d93a
Change-Id: I57086670f3589beea8461ce0344f6ec47ab85b7b
Revert "Fix "'module' object is not callable" error", and fix it properly.
* The urlparse module is renamed to urllib.parse in Python 3.
* This commit fixes the code to use "urllib.parse.urlparse"
instead of creating a new module urlib and setting
urlib.parse to urlparse.urlparse.
* Fixes an AttributeError:
'function' object has no attribute 'uses_relative'
This reverts commit cd51f17c64.
Change-Id: I48490b20ecd19cf5a6edd835506ea5a467d556ac
In a couple of files the urlparse method was not being set up
correctly for python < 3 and this resulted in an error being
thrown when trying to call it.
Change-Id: I4d2040ac77101e4e228ee225862f365ae3d96cec
This was broken in b2bd91c, which updated the manifest after it had
been overridden, which made it fall back to the original file (and
not the one from the manifest server).
This builds on 0766900 and overrides the manifest by the one
downloaded from the manifest server completely.
Change-Id: Ic3972390a68919b614616631d99c9e7a63c0e0db
It doesn't make sense to print the relpath, since there's nothing
checked out there and the dir shouldn't even exist.
Change-Id: Id43631a8e0895929d3a5ad4ca8c2dc9e3d233e70
git-remote-persistent-https proxy implementations may pass cookie file
configuration to git-remote-https. When fetching bundles for
persistent-http(s) URLs, use the -print_config flag (if supported) to
extract this information from the proxy binary and pass it to curl,
overriding http.cookiefile from .gitconfig.
This adds a few ms overhead per clone.bundle fetch, which should be
acceptable since it happens only on the initial clone, which takes
much longer anyway.
Change-Id: I03be8ebaf8d3005855d33998cd8ecd412e8ec287
Server auth middleware may return a 200 from a clone.bundle request
that is not a bundle file, but instead a login or access denied page.
Instead of just checking the file size, actually check the first few
bytes of the file to ensure it is a bundle file before proceeding.
Change-Id: Icea07567c568a24fd838e5cf974c58f9e4abd7c0
This fixes dest-branch display with >1 branch being uploaded to at
once, and correctly handles setting the target branch in that case.
Change-Id: If5e9c7ece02cc0d903e2cb377485ebea73a07107
The call to `urlopen` can raise `HTTPException`, but this is not
caught which results in a python Traceback.
Add handling of the exception. Because `HTTPException` and its
derived classes do not have any message, print the name of the
exception in the error message instead.
Change-Id: Ic90fb4cc0e92702375cd976d4a03876c8ce8bffc
This adds the ability to have reviews pushed to a different branch
from the one on which changes are based. This is useful for "gateway"
systems without smartsync.
Change-Id: I3a8a0fabcaf6055e62d3fb55f89c944e2f81569f
Add a new module with methods for checking the Python version.
Instead of handling Python3 imports with try...except blocks, first
check the python version and then import the relevant modules. This
makes the code a bit cleaner and will result in less diff when/if we
remove support for Python < 3 later.
Use the same mechanism to handle `input` vs. `raw_input` and add
suppression of pylint warnings caused by redefinition of the built-in
method `input`.
Change-Id: Ia403e525b88d77640a741ac50382146e7d635924
Also-by: Chirayu Desai <cdesai@cyanogenmod.org>
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
Several files have local suppression of pylint warnings. We don't
need these to be reported; code review should catch any unnecessary
suppressions.
Change-Id: Ie71ba3e910714ef3fe44554a71bb62519d0a891d
When running 'repo init --reference=<mirror>', the mirror will be
used for all projects except the manifest project. This is because
the _InitGitDir function uses the 'repo.reference' git config
value specified in the manifest git, which has no effect when
creating the manifest git as that value will be set after the git
has been successfully cloned.
Information about where the manifest git is located on the server
is only known when performing the 'repo init', so that information
has to be provided when cloning the git in order for it to set up
a proper mapping.
Change-Id: I47a2c8b3267a4065965058718ce1def4ecb34d5a
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
Fetching all tags for a shallow git results in an
inconstent git and forces git to fetch more than
the depth specified.
This change teaches repo not to fetch any tags in a
repository initialised with the depth option.
Change-Id: I557ead9f88fa0d6a19b1cb55b23bba1100fcb8f2
Signed-off-by: Patrik Ryd <patrik.ryd@stericsson.com>
When running pylint over the entire code base, it reports the
warning:
R0801: Similar lines in 2 files
for several pairs of files.
The code referred to is boilerplate code related to imports and
error handling. It is not practical to change the code to avoid
the warnings, so simply disable them in the config.
Change-Id: Ie685fdf1cab60fb8f1503be560166a14058698d8
There are several modules that have imports to support various
versions of Python. Pylint reports the following errors when
run in a version of Python that does not have the module or the
method/class in the module.
F0401: Unable to import 'module'
E0611: No name 'name' in module 'module'
Disable these warnings to reduce the noise on the output.
Change-Id: I97e7e2698bccb1e501a45a0869f97f90d54adfb7
Make a list of compiled patterns once, and then iterate over that
per project, instead of compiling the patterns again on every project.
Change-Id: I91ec430d3060ec76d5e6b61facf6b13e343c90a7
Filter the project list based on regex or wildcard matching
of strings, then we can handle subset of all projects.
Change-Id: Ib6c23aec79e7d981f7b6a5eb0ae93c44effec467
Signed-off-by: Zhiguang Li <muzili@gmail.com>
The name of the project is shown as "repo" in the project list in
the Eclipse workspace.
This change renames it to "git-repo" to match the name of the git
repository.
The existing project in Eclipse must be removed (it is not necessary
to delete project contents on disk) and re-imported for the change to
take effect.
Change-Id: I2ac022d22f46e5361dfe49c0dbcad482aaefe628
* Fix imports.
* Use python3 syntax.
* Wrap map() calls with list().
* Use list() only wherever needed.
(Thanks Conley!)
* Fix dictionary iteration methods
(s/iteritems/items/).
* Make use of sorted() in appropriate places
* Use iterators directly in the loop.
* Don't use .keys() wherever it isn't needed.
* Use sys.maxsize instead of sys.maxint
TODO:
* Make repo work fully with python3. :)
Some of this was done by the '2to3' tool [1], by
applying the needed fixes in a way that doesn't
break compatibility with python2.
Links:
[1]: http://docs.python.org/2/library/2to3.html
Change-Id: Ibdf3bf9a530d716db905733cb9bfef83a48820f7
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
* Print project name if the "quiet" option is not used.
Change-Id: I99863bb50f66e4dcbaf2d170bdd05971f2a4e19a
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
In some cases, especially when local manifest files exist, users may want
to force the mirrored repositories to be created in folders according to
their 'path' attribute in the manifest, rather than according to the name
of the repositories.
To enable this functionality for specified mirror, add a new attribute
'force-path' for that project in the manifest, set its value to 'true'.
Change-Id: I61df8c987a23d84309b113e7d886ec90c838a6cc
Signed-off-by: Scott Fan <fancp2007@gmail.com>
If the clone-depth attribute is set on a project, its value will
be used to set the depth when fetching the git. The value, if
given, must be a positive integer.
The value in the clone-depth attribute overrides any value given to
repo init via the --depth command line option.
Change-Id: I273015b3724213600b63e40cca4cafaa9f782ddf
`repo list -n` prints only the name of the projects.
`repo list -p` prints only the path of the projects.
Change-Id: If7d78eb2651f0b1b2fe555dc286bd2bdcad0d56d
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
Change Details:
* Make "default" a special manifest group that matches any project that
does not have the special project group "notdefault"
* Use "default" instead of "all,-notdefault" when user does not specify
manifest group
* Expand -g option help to include example usage of manifest groups
Change Benefits:
* Allow a more intuitive and expressive manifest groups specification:
* "default" instead of "all,-notdefault"
* "default,foo" instead of "all,-notdefault,foo"
* "default,-foo" instead of "all,-notdefault,-foo"
* "foo,-default" which has no equivalent
* Default manifest groups behavior can be restored by the command
'repo init -g default'. This is significantly more intuitive than the
current equivalent command 'repo init -g all,-notdefault'.
Change-Id: I6d0673791d64a650110a917c248bcebb23b279d3
Long story short, w/out this modification the manifest dump points
at the alias, rather than the actual remote for the project. This
breaks sync'ing for scenarios where the alias doesn't have the same
repos available as the remote, plus just plain is wrong.
Change-Id: I7150e449341ed8655d398956a095261978d95870
Repo now re-initialises the git-hooks reference directory
when updating the forest. This allows for any new template
files to be made available throughout the project forest
when updating the forest. Previous functionality required
the user to recreate the forest.
Change-Id: I9051265028a9e77d6139791547fff095bc195077
Signed-off-by: Patrik Ryd <patrik.ryd@stericsson.com>
This is currently the only generated file not present in the .gitignore
Apparently it comes from the usage of the "imp" module in main.py
Change-Id: I685dc252d0c822818434a8e5f493f77b63a66f03
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
List the user's manifest groups when running `repo info`.
These groups are passed to `repo init` using the -g/--groups flag.
Change-Id: Ie8a4ed74a35b8b90df3b1ee198fe725b1cd68ae7
Several messages are printed with the `print` method and the message
is split across two lines, i.e.:
print('This is a message split'
'across two source code lines')
Which causes the message to be printed as:
This is a message splitacross two source code lines
Add a space at the end of the first line before the line break:
print('This is a message split '
'across two source code lines'
Also correct a minor spelling mistake.
Change-Id: Ib98d93fcfb98d78f48025fcc428b6661380cff79
HTTP can't use the older style of passing options as part of
the git receive-pack command line. Use the new style as defined
by https://gerrit-review.googlesource.com/42652 when connecting
over HTTP.
If the Gerrit server is too old to understand the % option
syntax used here one of two outcomes is possible:
- If no topic name was sent the server will fail with an error
message. This happens because the user tried to do an upload to
"refs/for/master%r=alice", and the branch does not exist.
The user can delete the options and retry the upload.
- If a topic was set the options will be read as part of the
topic string and shown on the change page in the topic field.
Either outcome is slightly better than the current behavior of
just dropping the data on the floor and forgetting whatever the
user tried to supply.
Change-Id: Ib2df62679e5bf3ee93d6b18c12ab6474f96d9106
Add an option to pass `--no-tags' to `git fetch'.
Change-Id: I4158cc369773e08e55a167091c38ca304a197587
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The current printer always expands on the arguments which is a problem
for strings containing %.
Instead of forcing manual string expansion before printing allow for a
no format printer option which simply accepts and prints the string.
Part of fix for issue #131:
http://code.google.com/p/git-repo/issues/detail?id=131
Change-Id: I08ef94b9c4ddab58ac12d2bd32ebd2c413e4f83b
When running repo sync, the local_manifest.xml deprecation warning
is shown twice.
Add a flag to ensure that it is only displayed once.
Change-Id: Icfa2b0b6249c037c29771f9860252e6eda3ae651
If the .repo/local_manifests folder includes a local manifest file
that cannot be parsed, the current behaviour is to catch the parse
exception, print a warning, and continue to process remaining files.
This can cause any errors to go unnoticed.
Remove the exception handling, so that the exception is instead
caught in main._Main, and repo exits with a fatal error.
Change-Id: I75a70b7b850d2eb3e4ac99d435a4568ff598b7f4
When a local_manifest.xml file is present, a deprecation warning
is printed telling the user to put local manifest files in the
`local_manifests` directory.
Include the full path to the `local_manifests` directory in the
warning, to reduce confusion about where the directory should be
located. Also enclose the directory name in backticks.
Change-Id: I85710cfbd6e77fb2fa6b7b0ce66d77693ccd649f
`repo cherry-pick` was broken because we were referencing stderr
instead of sys.stderr. This should fix it.
Change-Id: I67f25c3a0790d029edc65732c319df7c684546c8
Add a .gitattributes file to prevent /bin/sh scripts from
getting clobbered by git config core.autocrlf=true setting.
Change-Id: I3dfc992a9c275fceae64c9719168d81e60d911bd
If the server returns HTTP 403 (forbidden) when attempting to
download clone bundle files, ignore it and continue, rather than
exiting with a fatal error.
Change-Id: Icf78cba0332b51b0e7b622f7c7924369b551b6f6
If a workspace is initialised with:
repo init -u git://path/to/manifest -b manifest-branch
and the default.xml specifies the default revision as `other-branch`,
running `repo info -d` results in a GitError:
fatal: bad revision 'refs/remotes/m/other-branch..'
The repo info command uses the default revision to build the symlink
to the remote revision which is passed to the `git rev-list` command.
This is incorrect; the manifest's branch name should be used.
Change-Id: Ibae5b91869848276785facfaef433e38d49fd726
Documentation of the "sync-j", "sync-c", "sync-s" and "upstream"
attributes is missing/incomplete. Add it.
Change-Id: I74977f819f603c520ef3818f85c3b51399cd2b94
'repo status --orphans' searches for non-repo objects
(not within a project), which is particularly helpful
before removing a working tree.
Change-Id: I2239c12e6bc0447b0ad71129551cb50fa671961c
In the information message displayed after running repo init, there
is a missing space:
If this is not the directory in which you want to initializerepo
Add a space.
Change-Id: I20467673ba7481cfe782ba58ff6ed2f7ce9824a5
If repo init is run with the --mirror option, repo checks if there
is already a workspace initialized in the current location, and if
so, exits with an error message:
--mirror not supported on existing client
This error can cause confusion; the users do not understand what
is wrong and what they need to do to fix it.
Change the error message to make it a bit clearer.
Change-Id: Ifd06ef64fd264bd1117e4184c49afe0345b75d8c
If the user's git config specifies a cookie file for HTTP, use it when
fetching clone.bundle, as it may contain the required login credentials
to get access (e.g. when used with Compute Engine service accounts).
Change-Id: I12ee9e38694822ef1de4ed62138c3876c43f431b
This fixes the following python error message if the
current working directory is not .repo/local_manifests:
IOError: [Errno 2] No such file or directory: 'local_manifest.xml'
Signed-off-by: Tobias Droste <tdroste87@gmail.com>
CC: David Pursehouse <david.pursehouse@sonymobile.com>
Change-Id: I4668dc04219b6233c7ff6ca3b4138bec9ee3529f
Some versions of Python will only attempt to resolve a relative
URL if they understand the URL scheme. Convert persistent-http://
and persistent-https:// schemes to the more typical http:// and
https:// versions for the resolve call.
Change-Id: I99072d5a69be8cfaa429a3ab177ba644d928ffba
When trying to render manifest with SHAs, projects in group notdefault
caused the following crash:
Traceback (most recent call last):
File ".repo/repo/main.py", line 385, in <module>
_Main(sys.argv[1:])
File ".repo/repo/main.py", line 365, in _Main
result = repo._Run(argv) or 0
File ".repo/repo/main.py", line 137, in _Run
result = cmd.Execute(copts, cargs)
File ".repo/repo/subcmds/manifest.py", line 129, in Execute
self._Output(opt, manifest)
File ".repo/repo/subcmds/manifest.py", line 79, in _Output
peg_rev = opt.peg_rev)
File ".repo/repo/manifest_xml.py", line 199, in Save
p.work_git.rev_parse(HEAD + '^0'))
File ".repo/repo/project.py", line 2035, in runner
capture_stderr = True)
File ".repo/repo/git_command.py", line 215, in __init__
raise GitError('%s: %s' % (command[1], e))
error.GitError: rev-parse: [Errno 2] No such file or directory: 'prebuilts/eclipse-build-deps'
This patch resolves the issue by making sure that -notdefault is always
used as a default manifest group so that notdefault projects are not
rendered out by the manifest subcmd.
Change-Id: I4a8bd18ea7600309f39ceff1b1ab6e1ff3adf21d
Signed-off-by: Matt Gumbel <matthew.k.gumbel@intel.com>
If the current manifest is broken then "repo sync" fails because it
can't retrieve the default value for --jobs. Use 1 in this case, in
order that you can "repo sync" to get a fixed manifest (assuming someone
fixed it upstream).
Change-Id: I4262abb59311f1e851ca2a663438a7e9f796b9f6
(Previous submission of this change broke Android buildbot due to
incorrect regular expression for parsing git-config output. During
investigation, we also found that Android, which pulls Chromium, has a
workaround for Chromium's submodules; its manifest includes Chromium's
submodules. This new change, in addition to fixing the regex, also
take this type of workarounds into consideration; it adds a new
attribute that makes repo not fetch submodules unless submodules have a
project element defined in the manifest, or this attribute is
overridden by a parent project element or by the default element.)
We need a representation of git-submodule in repo; otherwise repo will
not sync submodules, and leave workspace in a broken state. Of course
this will not be a problem if all projects are owned by the owner of the
manifest file, who may simply choose not to use git-submodule in all
projects. However, this is not possible in practice because manifest
file owner is unlikely to own all upstream projects.
As git submodules are simply git repositories, it is natural to treat
them as plain repo projects that live inside a repo project. That is,
we could use recursively declared projects to denote the is-submodule
relation of git repositories.
The behavior of repo remains the same to projects that do not have a
sub-project within. As for parent projects, repo fetches them and their
sub-projects as normal projects, and then checks out subprojects at the
commit specified in parent's commit object. The sub-project is fetched
at a path relative to parent project's working directory; so the path
specified in manifest file should match that of .gitmodules file.
If a submodule is not registered in repo manifest, repo will derive its
properties from itself and its parent project, which might not always be
correct. In such cases, the subproject is called a derived subproject.
To a user, a sub-project is merely a git-submodule; so all tips of
working with a git-submodule apply here, too. For example, you should
not run `repo sync` in a parent repository if its submodule is dirty.
Change-Id: I4b8344c1b9ccad2f58ad304573133e5d52e1faef
The manifest URL and mirror location can be specified in environment
variables which will be used if the options are not passed on the
command line
Change-Id: Ida87968b4a91189822c3738f835e2631e10b847e
Extend the Command base class to allow options to be set from values
in environment variables, if the user has not given the option on the
command line and the environment variable is set.
Derived classes of Command can override the implementation of the method
_GetEnvironmentOptions to configure which of its options may be set from
environment variables.
Change-Id: I7c780bcf9644d6567893d9930984c054bce7351e
If a local manifest includes a 'remove-project' element that refers to
a project that does not exist in the manifest, the error message is a
bit cryptic.
Change the error message to make it clearer what is wrong.
Change-Id: I0b1043aaec87893c3128211d3a9ab2db6d600755
When a command (eg, `repo forall`) expects the manifest project to
exist, but there is no manifest, an IOException gets raised. This
change defines a new Exception type to be raised in these cases and
raises it when project.py fails to read the manifest.
Change-Id: Iac576c293a37f7d8f60cd4f6aa95b2c97f9e7957
Add a new option (-e, --abort-on-errors) which will cause forall to
abort without iterating through remaining projects if a command
exits unsuccessfully.
Bug: Issue 17
Change-Id: Ibea405e0d98b575ad3bda719d511f6982511c19c
Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
Instead of using a nested try (which repo is plagued with), use a single
try when executing the appropriate subcommand.
Change-Id: I32dbfc010c740c0cc42ef8fb6a83dfe87f87e54a
Change Ia6032865f9296b29524c2c25b72bd8e175b30489 improved the
help text for the init command, but the same improvement was not made
in repo.
Change-Id: Idc34e479b5237137b90e8b040824776e4f7883b0
The info command will print information regarding the current manifest
and local git branch. It will also show the difference of commits
between the local branch and the remote branch.
It also incorporates an overview command into info which shows commits
over all branches.
Change-Id: Iafedd978f44c84d240c010897eff58bbfbd7de71
Enable the following Pylint warnings:
C0322: Operator not preceded by a space
C0323: Operator not followed by a space
C0324: Comma not followed by a space
And make the necessary fixes.
Change-Id: I74d74283ad5138cbaf28d492b18614eb355ff9fe
The repo coding style is to indent at 2 characters, but there are
many places where this is not followed.
Enable pylint warning "W0311: Bad indentation" and make sure all
indentation is at multiples of 2 characters.
Change-Id: I68f0f64470789ce2429ab11104d15d380a63e6a8
When prompting for yes/no answers, convert the answer to lower
case before comparing. This makes it easier to catch answers
like "Yes", "yes", and "YES" with a comparison only for "yes".
Change-Id: I06da8281cec81a7438ebb46ddaf3344d12abe1eb
Documentation of the remove-project element still refers explicitly
to local_manifest.xml.
Change it to the more generic "a local manifest".
Change-Id: I6278beab99a582fae26a4e053adc110362c714c2
Clean up a few more unnecessary usages of lambda in `repo` that were missed
in the previous sweep that only considered files ending in .py.
Remove a duplicate import.
Change-Id: I03cf467a5630cbe4eee6649520c52e94a7db76be
It should be assumed that on modern development environments, python
is accessible to /usr/bin/env
Change the shebang as necessary and remove the magic hack.
This also means losing the -E option on the call to python, so that
PYTHONPATH and PYTHONHOME will be respected and local configuration
problems in those vars would be noticed
Change-Id: I6f0708ca7693f05a4c3621c338f03619563ba630
Local manifest files stored in the local_manifests folder are loaded
in alphabetical order, so it's easier to know in which order project
removals/additions/modifications will be applied.
If local_manifests.xml exists, it will be loaded before the files in
local_manifests.
Change-Id: Ia5c0349608f1823b4662cd6b340b99915bd973d5
Add handling of manifest parse errors in the main method, and
print an error. This will prevent python tracebacks being
dumped in many cases.
Change-Id: I75e73539afa34049f73c993dbfda203f1ad33b45
In the current implementation, an error is raised if a remote with the
same name is defined more than once. The check is only that the remote
has the same name as an existing remote.
With the support for multiple local manifests, it is more likely than
before that the same remote is defined in more than one manifest.
Change the check so that it only raises an error if a remote is defined
more than once with the same name, but different attributes.
Change-Id: Ic3608646cf9f40aa2bea7015d3ecd099c5f5f835
The preferred way to specify local manifests is to drop the file(s)
in the local_manifests folder. Print a deprecation warning when
the legacy local_manifest.xml file is used.
Change-Id: Ice85bd06fb612d6fcceeaa0755efd130556c4464
Add support for multiple local manifests stored in the local_manifests
folder under the .repo home directory.
Local manifests will be processed in addition to local_manifest.xml.
Change-Id: Ia0569cea7e9ae0fe3208a8ffef5d9679e14db03b
Catch ExpatError and exit gracefully with an error message, rather
than exiting with a python traceback.
Change-Id: Ifd0a7762aab4e8de63dab8a66117170a05586866
The --manifest-server-* flags broke the smartsync subcmd since
the corresponding variables weren't getting set. This change
ensures that they will always be set, regardless of whether we are
using sync -s or smartsync.
Change-Id: I1b642038787f2114fa812ecbc15c64e431bbb829
In Python3, range() creates a generator rather than a list.
None of the parameters in the ranges changed looked large enough
to create an impact in memory in Python2. Note: the only use of
range() was for iteration and did not need to be changed.
This is part of a series of changes to introduce Python3 support.
Change-Id: I50b665f9296ea160a5076c71f36a65f76e47029f
This option causes the git call to fail, which probably indicates a
programming error; callers should check the git version and change the
call appropriately if -c is not available. Failing loudly is preferable
to failing silently in the general case.
For an example of correctly checking at the call site, see I8fd313dd.
If callers prefer to fail silently, they may set GIT_CONFIG_PARAMETERS
in the environment rather than using the config kwarg to pass
configuration.
Change-Id: I0de18153d44d3225cd3031e6ead54461430ed334
This minimum version is required for the -c argument to set config on
the command line. Without this option, git by default uses as many
threads per invocation as there are CPUs, so we cannot safely
parallelize without hosing a system.
Change-Id: I8fd313dd84917658162b5134b2d9aa34a96f2772
Fixing some more pylint warnings:
W1401: Anomalous backslash in string
W0623: Redefining name 'name' from outer scope
W0702: No exception type(s) specified
E0102: name: function already defined line n
Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
If the parent of current directory has an initialized repo,
for example, if the current directory is
'/home/users/harry/platform/ics', and there is an initialized repo
in harry's home directory '/home/users/harry/.repo', when user
run 'repo init' command, repo is always initialized to parent
directory in '/home/users/harry/.repo', but most of time user
intends to initialize repo in the current directory, this patch
tells user how to do it.
Change-Id: Id7a76fb18ec0af243432c29605140d60f3de85ca
Previously, if a key was added, a client wouldn't add the key during
the sync step. This would cause issues if a new key were added and a
subsequent release were signed by that key.
Change-Id: I4fac317573cd9d0e8da62aa42e00faf08bfeb26c
We can't just let this run wild with a high (or even low) -j, since
that would hose a system. Instead, limit the total number of threads
across all git gc subprocesses to the number of CPUs reported by the
multiprocessing module (available in Python 2.6 and above).
Change-Id: Icca0161a1e6116ffa5f7cfc6f5faecda510a7fb9
The repo list -r command will execute a regex search for every
argument provided on both the project name and the project
worktree path.
Useful for finding rarely used gits.
Change-Id: Iaff90dd36c240b3d5d74817d11469be22d77ae03
Try to more accurately estimate which projects take the longest to
sync by keeping an exponentially weighted moving average (a=0.5) of
fetch times, rather than just recording the last observation. This
should discount individual outliers (e.g. an unusually large project
update) and hopefully allow truly slow repos to bubble to the top.
Change-Id: I72b2508cb1266e8a19cf15b616d8a7fc08098cb3
Some projects may consistently take longer to fetch than others, for
example a more active project may have many more Gerrit changes than a
less active project, which take longer to transfer. Use a simple
heuristic based on the last fetch time to fetch slower projects first,
so we do not tend to spend the end of the sync fetching a small number
of outliers.
This algorithm is probably not optimal, and due to inter-run latency
variance and Python thread scheduling, we may not even have good
estimates of a project sync time.
Change-Id: I9a463f214b3ed742e4d807c42925b62cb8b1745b
"except Exception as e" instead of "except Exception, e"
This is part of a transition to supporting Python 3. Python >= 2.6
support "as" syntax.
Note: this removes Python 2.5 support.
Change-Id: I309599f3981bba2b46111c43102bee38ff132803
Repo is dropping support for Python <2.5 soon, so this updates the
PyDev configuration appropriately.
Change-Id: If327951e3a9fd9ff7513b931bfcfe6172dc8e4c5
pylint configuration file (.pylintrc) is added, and submission
instructions are updated to include pylint usage steps.
Deprecated pylint suppression (`disable-msg`) is updated in a few
modules to make it work properly with the latest version (0.26).
Change-Id: I4ec2ef318e23557a374ecdbf40fe12645766830c
We need a representation of git-submodule in repo; otherwise repo will
not sync submodules, and leave workspace in a broken state. Of course
this will not be a problem if all projects are owned by the owner of the
manifest file, who may simply choose not to use git-submodule in all
projects. However, this is not possible in practice because manifest
file owner is unlikely to own all upstream projects.
As git submodules are simply git repositories, it is natural to treat
them as plain repo projects that live inside a repo project. That is,
we could use recursively declared projects to denote the is-submodule
relation of git repositories.
The behavior of repo remains the same to projects that do not have a
sub-project within. As for parent projects, repo fetches them and their
sub-projects as normal projects, and then checks out subprojects at the
commit specified in parent's commit object. The sub-project is fetched
at a path relative to parent project's working directory; so the path
specified in manifest file should match that of .gitmodules file.
If a submodule is not registered in repo manifest, repo will derive its
properties from itself and its parent project, which might not always be
correct. In such cases, the subproject is called a derived subproject.
To a user, a sub-project is merely a git-submodule; so all tips of
working with a git-submodule apply here, too. For example, you should
not run `repo sync` in a parent repository if its submodule is dirty.
Change-Id: I541e9e2ac1a70304272dbe09724572aa1004eb5c
Fixing more issues found with pylint. Some that were supposed to
have been fixed in the previous sweep (Ie0db839e) but were missed:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
And some more:
W0631: Using possibly undefined loop variable 'name'
W0223: Method 'name' is abstract in class 'name' but is not overridden
W0231: __init__ method from base class 'name' is not called
Change-Id: Ie119183708609d6279e973057a385fde864230c3
If repo could not find authentication credentials from ~/.netrc, this
patch tries to get user and password from user's console input. This
could be a good choice if user doesn't want to save his plain password
in ~/.netrc or if user doesn't know about the netrc usage.
The user will be prompted only if authentication infomation does not
exist in the password manager. Since main.py firstly loads auth
infomation from ~/.netrc, this will be executed only as fallback
mechanism.
Example:
$ repo upload .
Upload project xxx/ to remote branch master:
branch yyy ( 1 commit, ...):
to https://review.zzz.com/gerrit/ (y/N)? y
(repo may try to access to https://review.zzz.com/gerrit/ssh_info and
will get the 401 HTTP Basic Authentication response from server. If no
authentication info in ~/.netrc, This patch will ask username/passwd)
Authorization Required (Message from Web Server)
User: pororo
Password:
....
[OK ] xxx/
Change-Id: Ia348a4609ac40060d9093c7dc8d7c2560020455a
Fix the following issues reported by pylint:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
W0612: Unused variable 'name'
W0613: Unused argument 'name'
W0102: Dangerous default value 'value' as argument
W0105: String statement has no effect
Also fixed a few cases of inconsistent indentation.
Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
Change 9bb1816b removed part of a block of code, but left the
remaining part unreachable. Remove it.
Change-Id: Icdc6061d00e6027df32dee9a3bad3999fe7cdcbc
The definition of `ImportError` redefines the Python built-in
class of the same name.
It is not used anywhere, so remove it.
Change-Id: I557ce28c93a3306fff72873dc6f477330fc33128
Currently when doing a sync against a revision locked manifest,
sync has no option but to fall back to sync'ing the entire refs space;
it doesn't know which ref to ask for that contains the sha1 it wants.
This sucks if we're in -c mode; thus when we generate a revision
locked manifest, record the originating branch- and try syncing that
branch first. If the sha1 is found within that branch, this saves
us having to pull down the rest of the repo- a potentially heavy
saving.
If that branch doesn't have the desired sha1, we fallback to sync'ing
everything.
Change-Id: I99a5e44fa1d792dfcada76956a2363187df94cf1
Change Details:
* Switch first default group to 'all' instead of 'default'
Change Benefits:
* More consistent with default_groups in the counterpart Save() function
* Fixes bug where command 'repo manifest' added an extra 'default'
group to every output project element groups attribute. This bug was
particularly confusing for projects which had 'groups="notdefault"'
as they were output as 'groups="notdefault,default"' by 'repo manifest'
Change-Id: I5611c027a982d3394899466248b971910bec8c6b
Add two new command line options, -u/--manifest-server-username and
-p/--manifest-server-password, which can be used to specify a username
and password to authenticate to the manifest server when using the
-s/--smart-sync or -t/--smart-tag option.
If -u and -p are not specified when using the -s or -t option, use
authentication credentials from the .netrc file (if there are any).
Authentication credentials from -u/-p or .netrc are not used if the
manifest server specified in the manifest file already includes
credentials.
Change-Id: I6cf9540d28f6cef64c5694e8928cfe367a71d28d
manifest_xml: import `HEAD` and `R_HEADS` from correct module
version: import `HEAD` from correct module
`HEAD` and `R_HEADS` should be imported from the git_refs module,
where they are originally defined, rather than from the project
module.
repo: remove unused import of readline
cherry_pick: import standard modules on separate lines
smartsync: import subcmd modules explicitly from subcmd
Use:
`import re
import sys`
and
`from subcmds.sync import Sync`
Instead of:
`import sys, re`
and
`from sync import Sync`
Change-Id: Ie10dd6832710939634c4f5c86b9ba5a9cd6fc92e
When using the --smart-sync or --smart-tag option, and the specified
manifest server is hosted on a server that requires authentication,
repo sync fails with the error: HTTP 401 Unauthorized.
Add support for getting the credentials from the .netrc file.
If a .netrc file exists in the user's home directory, and it contains
credentials for the hostname of the manifest server specified in the
manifest, use the credentials to authenticate with the manifest server
using the URL syntax extension for Basic Authentication:
http://user:password@host:port/path
Credentials from the .netrc file are only used if the manifest server
URL specified in the manifest does not already include credentials.
Change-Id: I06e6586e8849d0cd12fa9746789e8d45d5b1f848
If the first line of manifest.xml is a XML comment, root.childNodes[0]
is not a 'manifest' element node. The python minidom module will makes
a 'Comment' node as root.childNodes[0]. Since the original code only
checks whether the first child node is 'manifest', it couldn't do any
command including 'sync' due to the 'ManifestParseError' exception. This
patch could allow the comments between '<?xml ...?>' and '<manifest>' in
the manifest.xml file.
Change-Id: I0b81dea4f806965eca90f704c8aa7df49c579402
`R_HEADS` is imported twice, from both the git_refs and project
modules.
It is actually defined in git_refs, and in project it is imported
from there, so the import of `R_HEADS` from project in the sync
module is redundant. Remove it.
`HEAD` is imported from project, but like `R_HEADS` it is actually
defined in git_refs. Import it from git_refs instead.
Change-Id: I8e2b0217d0d9f9f4ee5ef5b8cd0b026174ac52f4
When connecting to the manifest server, exceptions can occur but
are not caught, resulting in the repo sync exiting with a python
traceback.
Add handling of the following exceptions:
- IOError, which can be raised for example if the manifest server
URL is malformed.
- xmlrpclib.ProtocolError, which can be raised if the connection
to the manifest server fails with HTTP error.
- xmlrpclib.Fault, which can be raised if the RPC call fails for
some other reason.
Change-Id: I3a4830aef0941debadd515aac776a3932e28a943
Catch curl failures to download clone.bundle; don't let git try to parse
the 404 page as a bundle file (was causing much user confusion).
This should eliminate false error messages from init and sync such as:
error: '.repo/manifests.git/clone.bundle' does not look like a v2 bundle file
fatal: Could not read bundle '.repo/manifests.git/clone.bundle'.
error: RPC failed; result=22, HTTP code = 400
Signed-off-by: Matt Gumbel <matthew.k.gumbel@intel.com>
Change-Id: I7994f7c0baecfb45bb5a5850c48bd2a0ffabe773
Instead of every group being in the group "default", every project
is now in the group "all". A group that should not be downloaded
by default may be added to the group "notdefault".
This allows all group names to be positive (instead of removing groups
directly in the manifest with -default) and offers a clear way of
selecting every project (--groups all).
Change-Id: I99cd70309adb1f8460db3bbc6eff46bdcd22256f
The threaded 'repo sync' implementation would very often freeze the
process when interrupted by the user with Ctrl-C. The only solution
being to kill -9 the process explicitly from another terminal.
The reason for this is best explained here:
http://snakesthatbite.blogspot.fr/2010/09/cpython-threading-interrupting.html
This patch makes all helper sync threads 'daemon', which allows the
process to terminate immediately on Ctrl-C.
Note that this will forcefully kill all threads in case of interruption; this
is generally a bad thing, but:
1/ This is equivalent to calling kill -9 in another terminal, which
is the _only_ thing that can currently stop the process.
2/ There doesn't seem to be a way to tell the worker threads to
gently stop when they are in a blocking operation anyway (even
in the non-threaded case).
+ Do the same for "repo status -j<count>".
Change-Id: Ieaf45b0eacee36f35427f8edafd87415c2aa7be4
master's original purpose was to forge ahead on using git submodules,
but this route has been abandoned.
Change-Id: I164a9efc7821bcd1b941ad76649764722046081b
One of the recent changes introduced implicit path:xxx and name:xxx groups
to every project, however they are not being stripped when generating
a manifest using "repo manifest" command resulting in clutter
Change-Id: Iec8610ba794b2fe4a6cdf0f59ca561595b66f9b5
urllib2 is not thread safe and may be causing sync to lock up or
not work correctly on various platforms. Instead use the command
line curl program.
Change-Id: I36eaf18bb4df089d26ea99d533cb015e7c616eb0
The Content-Length when resuming is the number of bytes that
remain in the file. To compute the total size as expected by
the progress meter, we must add the bytes already stored.
While we are in this method fix uses of % operator to ensure
a tuple is always supplied.
Change-Id: Ic899231b5bc0ab43b3ddb1d29845f6390e820115
The `alias` is an optional attribute in element `remote`. It can be
used to override attibute `name` to be set as the remote name in each
project's .git/config. Its value can be duplicated while attribute
`name` has to be unique across the manifest file. This helps each
project to be able to have same remote name which actually points
to different remote url.
It eases some automation scripts to be able to checkout/push to same
remote name but actually different remote url, like:
repo forall -c "git checkout -b work same_remote/work"
repo forall -c "git push same_remote work:work"
for example:
The manifest with 'alias' will look like:
<?xml version='1.0' encoding='UTF-8'?>
<manifest>
<remote alias="same_alias" fetch="git://git.external1.org/" name="ext1"
review="http://review.external1.org"/>
<remote alias="same_alias" fetch="git://git.external2.org/" name="ext2"
review="http://review.external2.org"/>
<remote alias="same_alias" fetch="ssh://git.internal.com:29418" name="int"
review="http://review.internal.com"/>
<default remote="int" revision="int-branch" sync-j="2"/>
<project name="path/to/project1" path="project1" remote="ext1"/>
<project name="path/to/project2" path="project2" remote="ext2"/>
<project name="path/to/project3" path="project3"/>
...
</manifest>
In each project, use command "git remote -v"
project1:
same_alias git://git.external1.org/project1 (fetch)
same_alias git://git.external1.org/project1 (push)
project2:
same_alias git://git.external2.org/project2 (fetch)
same_alias git://git.external2.org/project2 (push)
project3:
same_alias ssh://git.internal.com:29418/project3 (fetch)
same_alias ssh://git.internal.com:29418/project3 (push)
Change-Id: I2c48263097ff107f0c978f3e83966ae71d06cb90
The overview command shows an overview of each branch in all (or the
specified) projects. The overview lists any local commits that have
not yet been merged into the project.
The report output is inspired by the report displayed following a
"repo prune" event, with the addition of listing the one-line log
messages for each commit that is not yet merged.
The report can also be filtered to show only active branches; by
default all branches that have commits beyond the upstream HEAD will
be listed.
Change-Id: Ibe67793991ad1aa38de3bc9747de4ba64e5591aa
For CrOS, we have scenarios were people checkout a smaller version
of our manifest via groups, and enable individual repositories as
needed for their work. Previously this was via local_manifest
manipulation, which breaks via manifest-groups would require a
remove-project tag.
Via injecting the projects name into the projects groups, this
allows us to instead manipulate the configured groups allowing
the user to turn on/off projects as necessary.
Change-Id: I07b7918e16cc9dc28eb47e19a46a04dc4fd0be74
The fix for issue #46 in 5d016502eb appears to break syncing in some
situations: the branch is deleted after the point where it's been
configured, which deletes part of its configuration and causes the
config to change each time you call `repo init`, alternating between a
configuration that works and one that doesn't.
Instead of deleting the branch with git branch -D, use git update-ref -d
which just deletes the ref (to avoid the rebase) without touching the
configuration for the branch that was set up during the first repo init.
This appears to ensure the config is left in a valid state all the time
no matter what combination of repo init commands you run, without
reintroducing the rebasing issue.
Change-Id: Iaadaa6e56a46840bbc593fa5b35cb5b34cd3ce69
Git requires the values in this environment variable to be
single quoted. repo must wrap the expression into '' before
adding it to the environment.
Change-Id: I20a1fb8772f9aa6e9fd5a0516c981c2ca020ef05
This patch fixes repo behaviour when running sync -d with unmodified
topic branches.
Prior to this patch sync -d would see the latest revision is already
checked out, thus staying on the branch. Since "-d" means detach we
should follow git's behaviour and actually detach from the branch in
that case.
Basic test case - after a fresh repo init + sync -
* repo start --all testdetach
* repo sync -d
* repo status
-> status shows active topic branch "testdetach",
should show :
nothing to commit (working directory clean)
Change-Id: Ic1351e6b5721b76557a51ab09f9dd42c38a4b415
Currently repo-rebase requires that all modifications be committed
locally before it will allow the rebase. In high-velocity environments,
you may want to just pull in newer code without explicitly creating
local commits, which is typically achieved using git-stash.
If called with the --auto-stash command line argument, and it is
determined that the current index is dirty, the local modifications
are stashed, and the rebase continues. If a stash was performed, that
stash is popped once the rebase completes.
Note that there is still a possibility that the git-stash pop will
result in a merge conflict.
Change-Id: Ibe3da96f0b4486cb7ce8d040639187e26501f6af
See repo issue #46 :
https://code.google.com/p/git-repo/issues/detail?id=46
When using repo init -b on an already existing repository,
the next sync will try to rebase changes coming from the old manifest
branch onto the new, leading in the best case scenario to conflicts
and in the worst case scenario to an incorrect "mixed up" manifest.
This patch fixes this by deleting the "default" branch in the local
manifest repository when the -d init switch is used, thus forcing
repo to perform a fresh checkout of the new manifest branch
Change-Id: I379e4875ec5357d8614d1197b6afbe58f9606751
Calculation of where the include file lives was broken by 23acdd3f14
since it resulted in looking for the first include in .repo, rather
than .repo/manifests.
While people can work around it via setting their includes to
manifests/<include-target>, that breaks down since each layer of
includes would then have to be relative.
As such, restore the behaviour back to 2644874d; manifests includes
are calculated relative to the manifest root (ie, .repo/manifests);
local manifests includes are calculated relative to .repo/ .
Change-Id: I74c19ba614c41d2f08cd3e9fd094f3c510e3bfd1
Ancient versions of Git don't understand the -c command line flag
that we tried to use to pass http_proxy down into Git on Darwin.
Use the environment variable instead, to more gracefully degrade
with these old versions.
Change-Id: Iffffa32088c1fd803895b990b3377ecfec6a1b14
This patch adds the option to include topic branches by adding the
following to a .gitconfig file:
uploadtopic = true
This option is only read in when the -t option is not already
specified at the command line.
Change-Id: I0e0eea49438bb4e4a21c2ac5bd498b68b5a9a845
This is basically the same repository, but may be slightly more
up-to-date than the one on code.google.com/p/git-repo.
Change-Id: I5c99539f53231958eefb6993f00997c9adf0a3c9
Fix detection for Git not being in $PATH during the initial
run of `repo init` in a new directory.
Change-Id: I2b1fcce1fb8afc47271f5c3bd2a28369009b2fb7
The system libcurl library seems to ignore http_proxy on Mac OS
X systems. Copy the http_proxy environment variable (if set) as
`git -c http.proxy` whenever running a Git command.
Change-Id: I0ab29336897178f70b85092601f9fcc306dd17e1
Barring any kernel bugs, if this directory exists and there is
a symlink in there (which will point to the battery object),
that means there is a battery known to the kernel.
No symlink should mean no battery as far as the kernel is concerned.
Change-Id: Ib12819a5bbb816f0ae5ca080e5812a2db08441e9
Mirror manifest and repo projects are outside the manifest and
have no groups. Allow project groups to be None for these
projects.
Change-Id: I3e1c4add894fe1c43aa4e77a1fc1558aa10dd191
Combine manifest and local_manifest into a single list of elements
before parsing. This will allow elements in the local_manifest to
affect elements in the main manifest.
Change-Id: I4d34c9260b299a76be2960b07c0c3fe1af35f33c
Having the ability to include other manifests is a very practical feature
to ease the managment of manifest. It allows to divide a manifest into separate
files, and create different environment depending on what we want to release
You can have unlimited recursion of include, the manifest configs will simply be concatenated
as if it was in a single file.
command "repo manifest" will create a single manifest, and not recreate the manifest hierarchy
for example:
Our developement manifest will look like:
<?xml version='1.0' encoding='UTF-8'?>
<manifest>
<default revision="platform/android/main" remote="intel"/>
<include name="server.xml"/> <!-- The Server configuration -->
<include name="aosp.xml" /> <!-- All the AOSP projects -->
<include name="bsp.xml" /> <!-- The BSP projects that we release in source form -->
<include name="bsp-priv.xml" /> <!-- The source of the BSP projects we release in binary form -->
</manifest>
Our release manifest will look like:
<?xml version='1.0' encoding='UTF-8'?>
<manifest>
<default revision="platform/android/release-ext" remote="intel"/>
<include name="server.xml"/> <!-- The Server configuration -->
<include name="aosp.xml" /> <!-- All the AOSP projects -->
<include name="bsp.xml" /> <!-- The BSP projects that we release in source form -->
<include name="bsp-ext.xml" /> <!-- The PREBUILT version of the BSP projects we release in binary form -->
</manifest>
And it is also easy to create and maintain feature branch with a manifest that looks like:
<?xml version='1.0' encoding='UTF-8'?>
<manifest>
<default revision="feature_branch_foobar" remote="intel"/>
<include name="server.xml"/> <!-- The Server configuration -->
<include name="aosp.xml" /> <!-- All the AOSP projects -->
<include name="bsp.xml" /> <!-- The BSP projects that we release in source form -->
<include name="bsp-priv.xml" /> <!-- The source of the BSP projects we release in binary form -->
</manifest>
Signed-off-by: Brian Harring <brian.harring@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
Change-Id: I833a30d303039e485888768e6b81561b7665e89d
Allows to ff-only a gerrit patch
This patch is necessary to automatically ensure that the patch will
be correctly submitted on ff-only gerrit projects
You can now use:
repo download (--ff-only|-f) project changeid/patchnumber
This is useful to automate verification of fast forward status of a patch
in the context of build automation, and commit gating (e.g. buildbot)
Change-Id: I403a667557a105411a633e62c8eec23d93724b43
Signed-off-by: Erwan Mahe <erwan.mahe@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
BZ: 4779
Allows to revert a gerrit patch
This patch is necessary for the on-demand creation of
engineering builds using buildbot
You can now use:
repo download [--revert|-r project changeid/patchnumber
This is useful to automate reverting of a patch
in the context of build automation, and regression bisection
Change-Id: I3985e80e4b2a230f83526191ea1379765a54bdcf
Signed-off-by: Erwan Mahe <erwan.mahe@intel.com>
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
default option uses git checkout, and thus overwrite the previous
checkouts. this is a problem for automated builds of several
changesets in the same project for daily builds of pending submission
You can now use:
repo download [--cherry-pick|-c] project changeid/patchnumber
This will parse the manifest, cd to the corresponding project
download the changes to FETCH_HEAD and cherry-pick the result.
This is useful to automate cherry-picking of a patch
in the context of build automation, and commit gating (e.g. buildbot)
Change-Id: Ib638afd87677f1be197afb7b0f73c70fb98909fe
Signed-off-by: Pierre Tardy <pierre.tardy@intel.com>
repo status should output filenames one by one instead of trying to
build a string from incompatible encodings (like utf-8 and sjis
filenames)
Change-Id: I52282236ececa562f109f9ea4b2e971d2b4bc045
There are use-cases when fetching all branch is impractical and
we really need to fetch only one branch/tag.
e.g. there is a large project with binaries and every update of a
binary file is put to a separate branch.
The whole project history might be too large to allow users fetch it.
Add 'sync-c' option to 'project' and 'default' tags to make it possible
to configure 'sync-c' behavior at per-project and per-manifest level.
Note that currently there is no possibility to revert boolean flag from
command line. If 'sync-c' is set in manifest then you cannot make
full fetch by providing a repo tool argument.
Change-Id: Ie36fe5737304930493740370239403986590f593
Previous incarnations of groups support left "groups=" in the
repo .config, which is now treated as "delete all the projects".
Treat empty groups configuration the same as no groups
configuration.
Change-Id: I57dab8dac55bdbf4cc181e2748cd2e4e510764f5
Fixes three errors:
Python doesn't like the line wrap after 'and'.
platform.system is a function, needs to be platform.system().
Typo all_platfroms instead of all_platforms.
Change-Id: Ia875e521bc01ae2eb321ec62d839173c00f86c2d
Projects may optionally specify their platform
(eg, groups="platform-linux" in the manifest).
By default, repo will automatically detect the platform. However,
users may specify --platform=[auto|all|linux|darwin].
Change-Id: Ie678851fb2fec5b0938aede01f16c53138a16537
Every project is in group "default". "-default" does not remove
it from this project. All group names specified in the manifest
are positive names as opposed to a mix of negative and positive.
Specified groups are resolved in order. If init is supplied with
--groups="group1,-group2", the following describes the project
selection when syncing:
* all projects in "group1" will be added, and
* all projects in "group2" will be removed.
Change-Id: I1df3dcdb64bbd4cd80d675f9b2d3becbf721f661
Allow the optional addition of "annotation" nodes nested under
projects. Each annotation node must have "name" and "value"
attributes. These name/value pairs will be exported into the
environment during any forall command, prefixed with "REPO__"
In addition, an optional "keep" attribute with case insensitive "true"
or "false" values can be included to determine whether the annotation
will be exported with 'repo manifest'
Change-Id: Icd7540afaae02c958f769ce3d25661aa721a9de8
Signed-off-by: James W. Mills <jameswmills@gmail.com>
Previously repo had incorrect code that did not really check
if sha1 presents in a project. It worked for tags though.
Check if a revision (either tag or sha1) is present by using
'git rev_parse' functionality.
Change-Id: I1787f3348573948573948753987394839487572b
Allows specifying a list of groups with a -g argument to repo init.
The groups act on a group= attribute specified on projects in the
manifest.
All projects are implicitly labelled with "default" unless they are
explicitly labelled "-default".
Prefixing a group with "-" removes matching projects from the list
of projects to sync.
If any non-inverted manifest groups are specified, the default label
is ignored.
Change-Id: I3a0dd7a93a8a1756205de1d03eee8c00906af0e5
Reviewed-on: https://gerrit-review.googlesource.com/34570
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
The -u option causes 'repo diff' to generate diff output
with file paths relative to the repository root,
so the output can be applied to the Unix 'patch' command.
The name '-u' was selected for convenience, because
both 'diff' and 'git diff' accept the option with the same name
to generate an 'unified diff' output suitable for 'patch' command.
Change-Id: I79c8356db4ed20ecaccc258b3ba139db76666fe0
Reviewed-on: https://gerrit-review.googlesource.com/34380
Reviewed-by: Shawn Pearce <sop@google.com>
Tested-by: Shawn Pearce <sop@google.com>
A convenient equivalent to `repo upload --br=<current git branch>`.
Note that the head branch will be selected for each project
uploaded by repo, so different branches may be uploaded for
different projects.
Change-Id: I10ad8ceaa63f055105c2d847c6e329fa4226dbaf
401: Unauthorized, authentication may be required. This is usually
handled internally by the HTTP client in Python. If it reaches
our code in repo, the Python HTTP client didn't find a password
in ~/.netrc that it could use.
403: Authentication was supplied, but is incorrect. It might be
that the CDN doesn't want to offer this clone.bundle file
to the client, but the Git fetch operation would still be
successful. This might arise if branch level read controls
were used in Gerrit Code Review and the /clone.bundle file
contained branches not visible to the client.
404: The server has no /clone.bundle file available.
In all of these cases, sliently ignore the /clone.bundle file HTTP
error and let the Git operation take over.
Change-Id: I1787f3cac86b017351952bbb793fe5874d83c72b
In case of manifest/smart sync repo changes ".merge" config
option from branch to SHA. Doing 'repo upload' fails as
repo tries to upload to a remote branch that looks like SHA
(e.g. refs/for/23423423423423423423423)
Do not update the .merge in case if revision is SHA.
Change-Id: I9139708fa17f21eec5a7e23c3255333626bf529e
Clients might be using their own special git-remote-* helper that
has a hypen in its name. Permit - in the scheme part of the URL
when trying to decide if it is an SSH URL and assume it is *not*
SSH if the URL matches "foo-bar://" style.
Change-Id: I7ba2d810a614f6e605a441d5972902c4a14e73fd
repo tool supports only Basic authentication for now. For those
who want to use this tool to manage their own projects, in case
the administrator has configured the Apache server with Digest
authentication method, users will fail to be authenticated when
they run the command 'repo init'.
Add the digest authentication password manager to the handler
list will fix this issue.
Since Git HTTP protocol will require the user be authenticated
for fetch operation first before pushing commits to the remote,
it is unlikely for the administrator to implement anonymous
read (aka pull) access and write access (aka push) for
authenticated user. Both read and write have to be authenticated.
Be aware that the user may have to add an extra line in his
~/.netrc file:
-------------------
account example.com
-------------------
where 'example.com' is the realm for Apache Digest authentication.
Change-Id: I76eb27b205554426d9ce1965deaaf727b87916cd
Signed-off-by: Xiaodong Xu <stid.smth@gmail.com>
repo status just prints "# on branch oprofile" if you have branched
in clean status. This doesn't really tell which branch is meant.
Instead we can use the same syntax with modified gits which will
give us detailed information.
Change-Id: I55fe5154d278e10a814281dd2ba501ec6e956730
This new attribute can prevent 'repo sync' from automatically rebasing.
I hit a situation in where one of the git repositories I was tracking
was actually an external repository that I wanted to pull commits
into and merge myself. (NOT rebase, since that would lose the merge
history.) In this case, I'm not using 'repo upload', I'm manually
managing the merges to and from this repository.
Everything was going great until I typed 'repo sync' and it rebased
my manually-merged tree. Hence the option to skip it.
Change-Id: I965e0dd1acb87f4a56752ebedc7e2de1c502dbf8
Occassionally, the content-length may be missing when using urlib
in python 2.6 and 2.7. This change assumes the value of the header is
0 if it doesn't exist
Change-Id: Iaf1c8a796bc667823d4d7c30f9b617644b271d00
This parameter changes the manifest used by 'repo sync' for only
this execution. It should be useful for developers wishing to get
the repo temporarily into a known state, without clobbering their
existing manifest.
Tested by shifting Chrome OS between minilayout and full, and
between several release-builder-generated manifests.
Change-Id: I14194b665195b0e78f368d9ec8b8a83227af2627
If SSH is not available, Gerrit returns NOT_AVAILABLE to the /ssh_info
query made by repo upload. In this case fallback to the /p/$PROJECT URL
that Gerrit also exports and use that for uploads.
Change-Id: I1e3e39ab709ecc0a692614a41a42446426f39c08
This reverts commit ee1c2f5717.
This breaks a lot of buildbot systems. Rolling it back for now
until we can understand what the breakage was and how to fix it.
When repo is trying to figure out branches the repository has by
traversing refs/heads, add exception handling for readline.
Change-Id: If3b2a3720c6496f52f629aa9a2539f186d6ec882
Allows scripts driving repo to know when git failures have
occurred, not just repo internal errors.
Change-Id: Id20fbbb405c35a148e72c87b822da3f3bf93839c
If the user has already configured a workspace, use these values
when re-running 'repo init'.
Otherwise, if the user has global name and e-mail set, use these.
It's always possible to override this and be prompted by specifying
--config-name when running 'repo init'.
Change-Id: If45f0e4b14884071439fb02709dc5cb53f070f60
A default manifest URL can be specified using:
git config --global repo-manifest.<id>.url <url>
A default manifest server can be specified using:
git config --global repo-manifest.<id>.server <url>
A default git mirror reference can be specified using:
git config --global repo-manifest.<id>.reference <path>
This will allow the user to use 'repo init -u <id>' as
a shorter alternative to specifying the full URL.
Also, manifest server will not have to be specified in the
manifest XML and the reference will not have to be specified
on the command line. If they are, they will override these
default values however.
Change-Id: Ifdbc160bd5909ec7df9efb0c5d7136f1d9351754
Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
Several times one have done an upload only to later notice in gerrit
that the upload was done to the wrong branch as the git has not yet
been branched for the current git. This change will make repo print
what the destination branch is when asking the user if she wants to
go through with the upload.
Change-Id: Ia9c3a92a6a04c022edfebf4f8d651ac062bb1f3b
It is common in command line tools to indicate what the default answer
will be if the user simply hits enter. In repo, the display is just
"y/n" with no indication as to which is the default. So change the n
to N in the messages since that is how repo operates.
Change-Id: I81819ae630355072eb0365e59168b0921289498f
When commit with comment that has non-ASCII characters,
UnicodeDecodeError will be raised
while uploading multiple project/branch changes.
Because some strings in script are not str type, but unicode.
So all the strings are decoded to unicode,
and python use ascii to do this,
it can not decode non-ASCII characters,
so UnicodeDecodeError raised.
Signed-off-by: chenguodong <chenguodong@huawei.com>
Change-Id: I46447f489a4b9760a5899c7ba9d764b688594e46
Make repo use the standard way in python to work with pipes.
Communication via pipes to sub processes is done by calling
communicate(). This will make repo not hang every now and
then.
Change-Id: Ibe2c4ecbdbcbe72f0b725ca50d54088e5646fc5d
There is also shortcuts in case if the "current branch" is
a persistent revision such as tag or sha1. We check if the
persistent revision is present locally and if it does - do
no fetch anything from the server.
This greately reduces sync time and size of the on-disk repo
Change-Id: I23c6d95185474ed6e1a03c836a47f489953b99be
A bug introduced by relative urls caused projects such as manifest.git
to be placed in the root directory instead of the directory they should
by in.
This fix creates and refers to a resolvedFetchUrl in the _XmlRemote
class in order to get a fetchUrl that is never relative.
Python 2.6.6 has the same bug as Python 2.7, where HTTP
authentication just stops working, but does not have the
setter method to clear the retry counter. Work around by
setting the field directly if it exists.
Change-Id: I6a742e606bb7750dc66c33fc7c5d1310541db2c8
Signed-off-by: Shawn O. Pearce <sop@google.com>
help sync crashed as sync required the manifest to be configured to
create the option parser, as the default number of jobs is required.
Change-Id: Ie75e8d75ac0e38313e4aab451cbb24430e84def5
Signed-off-by: Shawn O. Pearce <sop@google.com>
REPO_HOST_PORT_INFO can be set to 'host:port' and be used
instead of the review URL given in the manifest.
Change-Id: I440bdecb2c2249fe5285ec5d0c28a937b4053450
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the remote is using authenticated HTTP, but does not have
$GIT_URL/clone.bundle files in each repository, an initial sync
would fail around 8 projects in due to the library not resetting
the number of failures after getting a 404.
Work around this by updating the retry counter ourselves.
The urllib2 library is also not thread-safe. Make it somewhat
safer by wrapping the critical section with a lock.
Change-Id: I886e2750ef4793cbe2150c3b5396eb9f10974f7f
Signed-off-by: Shawn O. Pearce <sop@google.com>
Not every version of urllib2 supplies a reason object on the
HTTPError exception that it throws from urlopen(). Work around
this by using str(e) instead and hope the string formatting includes
sufficient information.
Change-Id: I0f4586dba0aa7152691b2371627c951f91fdfc8d
Signed-off-by: Shawn O. Pearce <sop@google.com>
urllib2 returns a malformed HTTPError object in certain situations.
For example, urllib2 has a couple of places where it creates an
HTTPError object with no fp:
if self.retried > 5:
# retry sending the username:password 5 times before failing.
raise HTTPError(req.get_full_url(), 401, "basic auth failed",
headers, None)
When it does that, HTTPError's ctor doesn't call through to
addinfourl's ctor:
# The addinfourl classes depend on fp being a valid file
# object. In some cases, the HTTPError may not have a valid
# file object. If this happens, the simplest workaround is to
# not initialize the base classes.
if fp is not None:
self.__super_init(fp, hdrs, url, code)
Which means the 'headers' slot in addinfourl is not initialized and
info() fails. It is completely insane that urllib2 decides not to
initialize its own base class sometimes.
Change-Id: I32a0d738f71bdd7d38d86078b71d9001e26f1ec3
Signed-off-by: Shawn O. Pearce <sop@google.com>
* maint:
help: Don't show empty Summary or Description sections
sync: Run `git gc --auto` after fetch
Add "repo branch" as an alias for "repo branches"
upload: Catch and cleanly report connectivity errors
forall: Silently skip missing projects
Fix to display the usage message of the command download when the user don't provide any arguments to 'repo download'.
Use os.environ.copy() instead of dict()
Make path references OS independent
* stable: (33 commits)
Added feature to print a <notice> from manifest at the end of a sync.
sync: Use --force-broken to continue other projects
upload: Remove --replace option
sync --quiet: be more quiet
sync: Enable use of git clone --reference
Only delete corrupt pickle config files if they exist
Don't allow git fetch to start ControlMaster
Check for existing SSH ControlMaster
Fix for handling values of EDITOR which contain a space.
upload: Fix --replace flag
rebase: Pass through more options
upload: Allow review.HOST.username to override email
upload -t: Automatically include local branch name
Warn users before uploading if there are local changes
sync: Try fetching a tag as a last resort before giving up
rebase: Automatically rebase branch on upstrea
upload: Automatically --cc folks in review.URL.autocopy
Fix format string bugs in grep
Do not invoke ssh with -p argument when no port has been specified.
Allow files to be copied into new folders
...
Conflicts:
git_config.py
manifest_xml.py
subcmds/init.py
subcmds/sync.py
subcmds/upload.py
Change-Id: I4756a6908277e91505c35287a122a775b68f4df5
* stable:
Automatically install Gerrit Code Review's commit-msg hook
Fail sync when encountering "N commits behind."
Check that we are not overwriting a local repository when syncing.
Honor url.insteadOf when setting up SSH control master connection
sync: Fix split call on malformed email addresses
Fixing project renaming bug.
Conflicts:
hooks/commit-msg
project.py
subcmds/sync.py
Change-Id: I5eaf8fef8cbe4a95d124368112293a9ca64325bf
If a user aborts a commit, the commit-msg hook is still called,
but with an empty file. We need to leave the empty file alone.
Change-Id: I13766135dac267823cb08ab76f67d2000ba2d1ce
Signed-off-by: Shawn O. Pearce <sop@google.com>
This version fixes a bug where Change-Id lines become the subject
line, if the subject used a pattern like the subject of this
message does.
Change-Id: I7f7e0363091d03eb05dead2992fc19763214de65
Signed-off-by: Shawn O. Pearce <sop@google.com>
If a project is missing locally, it might be OK to skip over it
and continue running the same command in other projects.
Bug: REPO-43
Change-Id: I64f97eb315f379ab2c51fc53d24ed340b3d09250
Signed-off-by: Shawn O. Pearce <sop@google.com>
Instead of giving a Python backtrace when there is a connectivity
problem during repo upload, report that we cannot access the host,
and why, with a halfway decent error message.
Bug: REPO-45
Change-Id: I9a45b387e86e48073a2d99bd6d594c1a7d6d99d4
Signed-off-by: Shawn O. Pearce <sop@google.com>
Most users of repo are also using Gerrit Code Review, and will want
the commit-msg hook to be automatically installed into their local
projects so that Change-Ids are assigned when commits are created,
not when they are first uploaded.
Change-Id: Ide42e93b068832f099d68a79c2863d22145d05ad
Signed-off-by: Shawn O. Pearce <sop@google.com>
By running `repo manifest --uprade` an administrator can update the
current manifest format from the XML format to submodule format, but
we need all projects to be checked out in a work tree for this to
function correctly.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If a manifest top level directory contains '.gitmodules' we now
assume this is a git module format manifest and switch to using
that code, rather than the legacy XML based manifest.
At the same time, we move the bare repository for a project from
$TOP/.repo/projects/$REPO_PATH.git to be $REPO_NAME.git instead.
This makes it easier for us to later support a repo init from an
existing work tree, as we can more accurately predict the path of
the project's repository in the workspace. It also means that the
$TOP/.repo/projects/ directory is layed out like a mirror would be.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Some types of manifests might prefer to put their meta project work
tree under topdir, rather than inside of the .repo/ directory. We
can support that by allowing relpath to be optionally passed in.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the manifest format changes during init or sync we need to do
a full reparse of the manifest, and possibly allow the new object
to reconfigure the local workspace to match its expectations.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The -o option permits the user to control the name of the manifest's
remote, which normally is hardcoded to be 'origin', but can differ
because we derive it at runtime from the configuration file.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If we don't clear the cache, there can be a timestamp race between
the pickle file and the raw text file, and we may not pick up the
edit when we create a new config object around the same path name.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the manifest's work tree is actually inside of the rest of
the client work tree then its only fair that we include it as
a project that the user can access.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This permits the XML style manifest to use 'default', while other
types can use their own creation strategy for the current branch.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the caller knows exactly what the manifest type must be we
can now ask the loader to directly construct that type, rather
than guessing it from the working directory.
Signed-off-by: Shawn O. Pearce <sop@google.com>
I plan to have the new submodule manifest format use a different
layout for the m refs than the XML manifest format has used in
the past. Thus we need to move the behavior management into the
manifest object, and out of the project, so we can change it.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Users may wind up with a lot of loose object content in projects they
don't frequently make changes in, but that are modified by others.
Since we bypass many git code paths that would have otherwise called
out to `git gc --auto`, its possible for these projects to have
their loose object database grow out of control. To help prevent
that, we now invoke it ourselves during the network half of sync.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the manifest is the newer SubmoduleManifest style, then the -m
option makes no sense, as you cannot select a specific file within
the current branch.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the manifest isn't a single file format manifest, the -o option
makes no sense, as you cannot export multiple files to a single
stream for display or redirection.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Some of the help text is only related to the XML formatted manifest,
so only display that text if that is the current format.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The _LinkWorkTree method can now be used to relink the work tree,
such as if the real repository was moved to a different location
on disk.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If commands modify the git config too rapidly we might not notice
the .git/config file has been modified, as they could run in the
same filesystem timestamp window and thus not cause a change on
the config's mtime. This can cause repo to miss re-reading the
config file after running a command.
Allowing the cache to be flushed forces us to re-read the config.
Signed-off-by: Shawn O. Pearce <sop@google.com>
2009-07-03 11:00:16 -07:00
143 changed files with 32533 additions and 7040 deletions
| repo.partialcloneexclude | `--partial-clone-exclude` | Comma-delimited list of project names (not paths) to exclude while using [partial git clones] |
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
