When a hard revision ID is provided in a regular project tag then the
revisionId is updated as well if it is a commit hash. The difference
is that if the revisionExpr is a commit, git-repo needs to update
refs/remotes/m/master with update-ref not symbolic-ref, as the latter
must refer to another ref, not to a specific commit.
Change-Id: I215a62dabb30225e480ad2c731416d775fc0c750
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310963
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This allows us to move the repository to a new location in the source
tree without having to remove-project + add a new project tag.
Change-Id: I4dba6151842e57f6f2b8fe60cda260ecea68b7b4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310962
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
Some of the file removal calls are subject to race conditions (if
something else deletes the file), so extend our remove API to have
an option to ignore ENOENT errors. Then update a bunch of random
call sites to use this new functionality.
Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195
Reviewed-by: Sean McAllister <smcallis@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
For older git-repo versions, we might have only fetched the SHA1
revision that was provided by the project, but have remote branch left
intact as long as they exist. When the remote branch become stale,
some repo operations like rebase would fail, and repo sync would not
correct this situation.
Fix this by tightening the requirement to also require the superproject
provided SHA1 be an ancestor or equal to the tip-of-tree of the remote
branch.
Bug: [google internal] b/193798453
Change-Id: Ie34c5d860dabb1cbd9f822da929088ec69c79cf6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312642
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
It is possible that a clone bundle contained the object referenced by
the branch in the manifest and in the superproject, but not the branch
itself (for example, the branch may be newly created from an existing
branch, or is not vislble to the user downloading the clone bundle).
When --use-superproject is enabled, because we are overriding
revisionExpr with the SHA1 revision provided by the superproject, the
verification step would succeed, but because the expected branch do not
exist, it would confuse git-repo at a later time, as it is expecting the
remote branch to exist in the local clone.
In project.py, fix this by making SetRevisionId() to always remember
the actual branch name and verify it in _CheckForImmutableRevision()
so that we only skip the fetch step when both objects exists locally.
Bug: [google internal] b/191974277
Change-Id: I49d3ca0667f524c8c45f416492faf95b1dd822fb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
We always pass in output_redir when syncing, but that's the common case:
there are a few situations (like `repo init`) where we don't pass in a
buffer, and if any errors show up in that case, we'd crash. Rely on the
print function to handle this logic for us.
Bug: https://crbug.com/gerrit/14568
Change-Id: I8cd47e82329797ffc42534418a3dfbd8429205be
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/307222
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We changed sync to use multiprocessing for parallel work. This broke
the ssh proxy code as it's all based on threads. Rewrite the logic to
be multiprocessing safe.
Now instead of the module acting as a stateful object, callers have to
instantiate a new ProxyManager class that holds all the state, an pass
that down to any users.
Bug: https://crbug.com/gerrit/12389
Change-Id: I4b1af116f7306b91e825d3c56fb4274c9b033562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305486
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
The only time we really need ssh proxies is when we want to run many
connections and reuse them. That only happens when running sync.
Every other command makes at most two connections, and even then it's
only one or none. So the effort of setting up & tearing down ssh
proxies isn't worth it most of the time.
The big reason we want to move this logic to sync is that it's now
using multiprocessing for parallel work. The current ssh proxy code
is all based on threads, which means none of the logic is working
correctly. The current ssh design makes it hard to fix when all of
the state lives in the global/module scope.
So the first step to fixing this is top move the setup & teardown to
the one place that really needs it: sync. No other commands will use
proxies anymore, just direct connections.
Bug: https://crbug.com/gerrit/12389
Change-Id: Ibd351acdec39a87562b3013637c5df4ea34e03c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305485
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When converting this logic from print() to the output buffer, this
error codepath should have dropped the use of the file= redirect.
Bug: https://crbug.com/gerrit/14482
Change-Id: Ib484924a2031ba3295c1c1a5b9a2d816b9912279
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
While this provides a way to undo earlier command line options (e.g.
`repo sync --tags --no-tags`) which can be helpful for scripting &
automation, this more importantly allows the user to override the
manifest settings for syncing tags from a project.
Bug: https://crbug.com/gerrit/12401
Change-Id: Id4c36cd82e6ca7cb073b5d63a09f6c7ccdebba83
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304904
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
For most commands, this is more about providing a way to undo earlier
command line options (e.g. `repo info -c --no-current-branch`) which
can be helpful for scripting & automation. But for the sync command,
this is helpful to undo the setting that exists in the manifest itself.
With this in place, tweak the sync current_branch_only logic to only
apply the manifest settings when the user hasn't specified a command
line option.
Bug: https://crbug.com/gerrit/12401
Change-Id: I21e2384624680cc740d1b5d1e49c50589d2fe6a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304903
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We were updating the per-checkout m/ pseudo ref when syncing, but we
only created the common m/ redirect when initializing a project for
the first time. This is fine unless the user switches the manifest
branch in an existing project, then we never create that redirect.
Bug: https://crbug.com/gerrit/14468
Change-Id: I5325e7e602dcb4ce150bef258901ba5e9fdea461
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304822
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When depth is used, we would fetch only SHA1 when superproject is
used, as the result, only the manifest branch is being recorded,
and commands like repo start would fail.
Fix this by saving the upstream branch value in the overlay
manifest and add the upstream branch to fetch list.
Bug: [google internal] b/185951360
Change-Id: Ib36f56067723f2572ed817785b31cc928ddfec0a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304562
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Xin Li <delphij@google.com>
Default the partial_clone_exclude argument to an empty set.
Fixes the following report by Emil Medve.
With this change (up to v2.14.1), on an existing "normal" clone (without partial-clone options) I'm seeing this traceback during `repo selfupdate`:
Traceback (most recent call last):
File ".../.repo/repo/main.py", line 630, in <module>
_Main(sys.argv[1:])
File ".../.repo/repo/main.py", line 604, in _Main
result = run()
File ".../.repo/repo/main.py", line 597, in <lambda>
run = lambda: repo._Run(name, gopts, argv) or 0
File ".../.repo/repo/main.py", line 261, in _Run
result = cmd.Execute(copts, cargs)
File ".../.repo/repo/subcmds/selfupdate.py", line 54, in Execute
if not rp.Sync_NetworkHalf():
File ".../.repo/repo/project.py", line 1091, in Sync_NetworkHalf
if self.name in partial_clone_exclude:
TypeError: argument of type 'NoneType' is not iterable
$ ./run_tests -v
Change-Id: I71e744e4ef2a37b13aa9ba42eba3935e78c4e40a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304082
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Make sure we print a message whenever we retry so it's clear to the
user why repo is pausing for a long time, and why repo might have
passed even though it displayed some errors earlier.
Also unify the sleep logic so we don't have two independent methods.
This makes it easier to reason about.
Also don't sleep if we're in the last iteration of the for loop. It
doesn't make sense to and needlessly slows things down when there are
real errors.
Bug: https://crbug.com/gerrit/12494
Change-Id: Ifceace5b2dde75c2dac39ea5388527dd37376336
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303402
Reviewed-by: Sam Saccone 🐐 <samccone@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
partial-clone-exclude option excludes projects during
partial clone. This is a comma-delimited project names
(from manifest.xml). This option is persisted and it
is used by the sync command.
A project that has been unparital'ed will remain unpartial if
that project's name is specified in the --partial-clone-exclude
option. The project name should match exactly.
Added
$ ./run_tests -v
Bug: [google internal] b/175712967
"I can't "unpartial" my androidx-main checkout"
$ rm -rf androidx-main/
$ mkdir androidx-main/
$ cd androidx-main/
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml
$ repo_dev sync -c -j8
+ Verify a project is partial
$ cd frameworks/support/
$ git config -l | grep 'partial'
+ Unpartial a project.
$ /google/bin/releases/android/git_repack/git_unpartial
+ Verify project is unpartial
$ git config -l | grep 'partial'
$ cd ../..
+ Exclude the project from being unparial'ed after init and sync.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml
+ Verify project is unpartial
$ cd frameworks/support/
$ git config -l | grep 'partial'
$ cd ../..
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep 'partial'
$ cd ../..
+ Remove the project from exclude list and verify that project is partially cloned.
$ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml
$ repo_dev sync -c -j8
$ cd frameworks/support/
$ git config -l | grep 'partial'
Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Re-allow checking out projects to the top of the repo client checkout.
We add checks to prevent checking out files under .repo/ as that path
is only managed by us, and projects cannot inject content or settings
into it.
Bug: https://crbug.com/gerrit/14156
Bug: https://crbug.com/gerrit/14200
Change-Id: Id6bf9e882f5be748442b2c35bbeaee3549410b25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299623
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the current project doesn't have any local branches, then there's
nothing to prune, so return right away. This avoids running a few
git commands when we aren't actually going to use the results, and
it avoids checking repository validity. Since we aren't going to do
anything in here, no need to check it.
Change-Id: Ie9d5c75a954e42807477299f3e5a63a92fac138b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299742
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The recent commit 84230009ee ("project:
make diff tools synchronous") broke repo diff if it includes % formats.
Add an explicit format string to fix.
Bug: https://crbug.com/gerrit/14208
Change-Id: Ie255a43c5b767488616b2b3dd15abc18f93bfab2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299402
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The clone bundle logic assumes there is a one-to-one mapping between the
projects/ and project-objects/ trees. When using shared projects (where
we checkout different branches from the same project), this would lead us
to fetching the same clone bundle multiple times. Automatically skip the
clone bundle logic if the project-objects/ dir already exists.
Bug: https://crbug.com/gerrit/10993
Change-Id: I82c6fa1faf8605fd56c104fcea2a43dd4eecbce4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The default sync output should show a progress bar only for successful
commands, and the error output for any commands that fail. Implement
that policy here.
Bug: https://crbug.com/gerrit/11293
Change-Id: I85716032201b6e2b45df876b07dd79cb2c1447a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297905
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When sync moved to consume clone output, it merged stdout & stderr,
but the retry logic in this function is based on stderr only. Move
it over to checking stdout.
Change-Id: I71bdc18ed25c978055952721e3a768289d7a3bd2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297902
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Use multiprocessing to run diff in parallel.
Change-Id: I61e973d9c2cde039d5eebe8d0fe8bb63171ef447
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297483
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Python 3 has a simpler super() style so switch to it to make the
code a little simpler and to stop pylint warnings.
Change-Id: I1b3ccf57ae968d56a9a0bcfc1258fbd8bfa3afee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297383
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
These are the only users in the tree that process the output as it's
produced. All others capture all the output first and then process
the results. However, these functions still don't fully return until
it's finished processing, and these funcs are in turn used in other
synchronous code paths. So it's unclear whether anyone will notice
that it's slightly slower or less interactive. Let's try it out and
see if users report issues.
This will allow us to simplify our custom GitCommand code and move it
over to Python's subprocess.run, and will help fix interleaved output
when running multiple commands in parallel (e.g. `repo diff -j8`).
Change-Id: Ida16fafc47119d30a629a8783babeba890515de0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297144
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
A little sugar simplifies the code a bit.
Change-Id: Ie2b8a965faa9f9ca05c7be479d03e8e073cd816d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296522
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
_CheckForImmutableRevision is used to see if repo can
skip fetching a project, but 'git rev-parse' with partial
clone does a data fetch to accomplish this.
Changed to use: 'git rev-list -1 --missing=allow-any <SHA>^0' which
checks the local ref without fetching from the server first.
Bug: [google internal] b/179477822
Testing:
- Unit tests
- Verified init/sync working on aosp-master
- Verified wwith a pinned manifest that local ref check works (no fetch)
Change-Id: If327b893c6658421f41df1f58c337f53b4c60ce6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296142
Reviewed-by: Dan Willemsen <dwillemsen@google.com>
Tested-by: Ian Kasprzak <iankaz@google.com>
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.
Tested the code with the following commands.
$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,
.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []
Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
We're committed to Python 3 at this point, so purge all the
is_python3 related dynamic checks.
Bug: https://crbug.com/gerrit/10418
Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When checkout is done with Git worktrees then the HEAD in the
bare-git repositories point to the initialized default (e.g.
'refs/heads/master'). This default branch does not exist
locally and is not automatically created.
When a user now creates a branch in any git repository named
'master' then it is no longer possible to get rid of this branch,
neither is it possible to switch to another branch and switch
back to this master branch. Git concludes the 'master' branch is
already checked out (in the bare Git) and that results in a
lockdown of this master branch.
To repoduce this issue, run these commands in a repo tree
checked out with --worktree:
- git checkout master # assuming the remote repo has a master branch,
# a local tracking branch master is created here
- git checkout -b temp
- git checkout master # This one now fails
- git branch -d master # fails too
The failure is caused by Git assuming the master branch is checked out
by the bare git repository since HEAD is pointing towards it.
To workaround this, we always detach HEAD in the bare-git when
syncing. We don't need it to point to a ref in general, but we
would like it to be valid so git tools "just work" if they're run
in here.
Signed-off-by: Remy Bohmer <oss@bohmer.net>
Change-Id: I15c96604363c41f0d01c42f533174393097daeb5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290985
Reviewed-by: Mike Frysinger <vapier@google.com>
When intializing a new repo with the --reference option on Windows 10
the objects/info/alternates in each git repository is created with
Windows line endings (\r\n), leading to the following error:
error: object directory C:/<PATH_TO_MIRROR>/<REPO_NAME>.git/objects?
does not exist; check .git/objects/info/alternates
This can be fixed by simply using unix line endings on both
Windows and unix platforms.
Reported-by: Francisco Javier Alvarez Garcia <javier.alvarez.garcia.17@gmail.com>
Follow-up-from: I268fe029ede68802c21037b0f2ae8a95afb85e48
Bug: https://crbug.com/gerrit/13208
Change-Id: I6da60c4ca957778b3c42ab6b9ad85c40483f0042
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289431
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <oss@bohmer.net>
Worktree .git and gitdir reference files are written by Git with
Unix line ending, even on Windows & macOS. The conversion to
relative paths makes these files end with DOS line endings in
Windows. The Git integration in Visual Studio 2019 cannot deal
with these DOS line endings and considers these worktrees invalid.
Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: I088cfd994f3cc31db4e0ca7791fa0a4ee3ac222f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289310
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <linux@bohmer.net>
We conflate the manifest & parsing logic with the management of the
repo client checkout in a single class. This makes testing just one
part (the manifest parsing) hard as it requires a full checkout too.
Start splitting the two apart into separate classes to make it easy
to reason about & test.
Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The project.py file is huge and contains multiple
classes. By moving it to seperate class files the code
becomes more readable and maintainable.
Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: Ida9d99d31751d627ae1ea0373418080696d2e14b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/281293
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <linux@bohmer.net>
Instead of hardcoding "master" as our default, use the remote server's
default branch instead. For most people, this should be the same as
"master" already. For projects moving to "main", it means we'll use
the new name automatically rather than forcing people to use -b main.
For repositories that never set up a default HEAD, we should still use
the historical "master" default.
Bug: https://crbug.com/gerrit/13339
Change-Id: I4117c81a760c9495f98dbb1111a3e6c127f45eba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280799
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This change increases the speed of the command with parallelization with
processes. The parallelization with threads doesn't work well, and
increasing the number of jobs to many (8 threads ~) didn't increase the speed.
Possibly, the global interpreter lock of Python affects.
Bug: https://crbug.com/gerrit/12389
Change-Id: Icbe5df8ba037dd91422b96f4e43708068d7be924
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279936
Tested-by: Kimiyuki Onaka <kimiyuki@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
worktreeconfig extension only appears with version Git 2.20.0
Change-Id: I3ea8b7d9f8a1f7953e536edd77b09cbc4f8f3158
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/276700
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Adrien Bioteau <adrien.bioteau@gmail.com>
The intention of the check is to verify whether the target
file name contains a wild card. The code, however, assumes
that if the file is non-existent - it contains a wild card.
This has the side effect that a target file that does not
exist at the moment of the check is considered to contain a
wild card, this leads itself to softlink not being created.
Change-Id: I4e4cd7b5e1b8ce2e4b2edc9abf5a1147cd86242f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/265736
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Angel Petkov <apetkov86@gmail.com>
Add retries with exponential backoff and jitter to the fetch
operations. By default don't change behavior and enable
behind the new flag '--fetch-retries'.
Bug: https://crbug.com/1061473
Change-Id: I492710843985d00f81cbe3402dc56f2d21a45b35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/261576
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: George Engelbrecht <engeg@google.com>
This is a pretty common option for people to want too use, so include
it as a pass-thru option when cherry-picking.
Bug: https://crbug.com/gerrit/9418
Change-Id: I2a24c1ed7544541719caa4d3c0574347a151a1b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259853
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The git cherry-pick already supports this, so plumb the existing repo
option down. Otherwise it's confusing when people use -c --ff and it
doesn't use that behavior.
Change-Id: Id68932ffa09204bb30b92a21aff185c00394a520
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259852
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When we download git sources, we get a progress bar (good) and we get
a dump of all the refs we downloaded (bad) as it can easily be 100+ if
not 1000+ depending on the project (for each git repo!). Lets rework
the output behavior so that:
* quiet: Only errors.
* default: Progress bars (if on a tty).
* verbose: Full output (progress bars & downloaded refs).
Bug: https://crbug.com/gerrit/11293
Change-Id: I87a380075e79de6805f91095876dd1b37d32873a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256456
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Since most ref namespaces are shared among all worktrees, trying to
set the pseudo m/<branch> in the common git repo ends up clobbering
each other when using shared checkouts. For example, in CrOS:
<project path="src/third_party/kernel/v3.8"
name="chromiumos/third_party/kernel"
revision="refs/heads/chromeos-3.8" />
<project path="src/third_party/kernel/v3.10"
name="chromiumos/third_party/kernel"
revision="refs/heads/chromeos-3.10" />
Trying to set m/master in chromiumos/third_party/kernel.git/ will
keep clobbering the other.
Instead, when using git worktrees, lets set the m/ pseudo ref to
point into the refs/worktree/ namespace which is unique to each
git worktree. So we have in the common dir:
chromiumos/third_party/kernel.git/:
refs/remotes/m/master:
ref: refs/worktree/m/master
And then in each worktree we point refs/worktree/m/master to the
respective manifest revision expression. Now people can use the
m/master in each git worktree and have it resolve to the right
commit for that worktree.
Bug: https://crbug.com/gerrit/12404
Change-Id: I78814bdd5dd67bb13218c4c6ccd64f8a15dd0a52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256952
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Draft CLs were replaced by private/wip CLs in Gerrit instead years ago.
Change-Id: If4f3d6606aad40a6f1617a49681dfd45c64d3d37
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256673
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Add a helper to our git wrapper to find the .git subdir,
and then use that to detect internal rebase state.
Change-Id: I3b3b6ed4c1f45cc8c3c98dc19c7ca3aabdc46905
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256532
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Git likes to create .git files with read-only permissions which makes
it difficult to open+truncate+write in situ under Windows. Delete it
before we write the file content to workaround.
Change-Id: I3effd96525f0dfe0b90e298b6bf0b856ea26aa03
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256412
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Windows requires Administrator access to create symlinks. We can
mitigate this a bit by falling back to hardlinks as those may be
created by any user on the system. Do this with the git hooks as
these are supposed to be internal only and people shouldn't be
modifying them. If they do, they'll have to delink first. This
seems worth it to allow repo usage without extra privileges.
Change-Id: I996ea9c9238f7bd7d27d1d9b1f2786593bf75ef7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256312
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We also need to check more things in the manifest/project handlers,
and use platform_utils in a few places to address Windows behavior.
Drop Python 2.7 from Windows testing as it definitely doesn't work
and we won't be fixing it.
Change-Id: I83d00ee9f1612312bb3f7147cb9535fc61268245
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256113
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We're using this for git worktrees because it handles the .git file
format, but it should work for all flows. Unify to simplify. This
also fixes the worktree logic which duplicated .git/config settings.
Change-Id: Ie3af2e206710859dccfc376b3593f415d6830738
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256034
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Since deleting a source checkout involves a good bit of internal
knowledge of .repo/, move the DeleteProject helper out of the sync
code and into the Project class itself. This allows us to add git
worktree support to it so we can unlock/unlink project checkouts.
Change-Id: If9af8bd4a9c7e29743827d8166bc3db81547ca50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256072
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When using extensions, make sure we set the git repo format version
so git knows to check the extension compatibility. We can add a
helper to the Project API to simplify this and make it foolproof.
Change-Id: I9ab6c32d92fe2b8e5df6e2b080ca71556332e909
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256035
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
This provides initial support for using git worktrees internally
instead of our own ad-hoc symlink tree. It's been lightly tested
which is why it's not currently exposed via --help.
When people opt-in to worktrees in an existing repo client checkout,
no projects are migrated. Instead, only new projects will use the
worktree method. This allows for limited testing/opting in without
having to completely blow things away or get a second checkout.
Bug: https://crbug.com/gerrit/11486
Change-Id: Ic3ff891b30940a6ba497b406b2a387e0a8517ed8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254075
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
This allows users to specify custom hashtags when uploading, both via
the CLI and via the same gitconfig settings as other upload options.
Bug: https://crbug.com/gerrit/11174
Change-Id: Ia0959e25b463e5f29d704e4d06e0de793d4fc77c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255855
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If we're not in --verbose mode with repo sync, then omit the
per-project clone bundle progress bar.
Bug: https://crbug.com/gerrit/11293
Change-Id: Ibdf3be86d35fcbccbf6788c192189f38c577e6e9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255854
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Trying to use booleans with names like "no_xxx" are hard to follow due
to the double negatives. Invert all of them so we only have positive
meanings to follow.
Change-Id: Ifd37d0368f97034d94aa2cf38db52c723ac0c6ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255493
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Syncing projects works fine the majority of the time. So rather than
dump all of that noisy output to stdout, lets capture it and only show
when things fail or in verbose mode. This tidies up the default `repo
sync` output.
Bug: https://crbug.com/gerrit/11293
Change-Id: I8314dd92e1e6aadeb26e36a8c92610da419684e6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255413
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This allows us to control sync output better by having three levels
of output: quiet (only errors), default (progress bars), verbose (all
the things). For now, we just put the chatty "already have persistent
ref" message behind the verbose level.
Bug: https://crbug.com/gerrit/11293
Change-Id: Ia61333fd8085719f3e99edb7b466cdb04031b67f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255414
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Our sync output is pretty chatty, and the stat output on fast forward
merges doesn't really help. Suppress it to tighten up the output.
Change-Id: I91e50639b3cd8db9df3d13a7da6d1aaa70d7932f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255412
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
All of the instances of this are related to Python 2 names that
don't exist in Python 3, and the warnings are raised when running
flake8 on Python 3.
All of these will go away once we completely remove support for
Python 2, so just suppress them inline. We don't globally suppress
the check so that we will still see legitimate errors if/when they
occur in new code.
Change-Id: Iccf955f50abfc9f83b371fc0af6cceb51037456f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255039
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
We've been requiring git-1.7.2 since Oct 2012, so we can safely drop
the individual checks sprinkled throughout the code base for older.
Change-Id: I1737fff7b3f27f475960b0bff9cb300aefd5d108
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253135
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
flake8 reports:
E722 do not use bare 'except'
Replace them with 'except Exception' per [1] which says:
Bare except will catch exceptions you almost certainly don't want
to catch, including KeyboardInterrupt (the user hitting Ctrl+C) and
Python-raised errors like SystemExit
If you don't have a specific exception you're expecting, at least
except Exception, which is the base type for all "Regular" exceptions.
[1] https://stackoverflow.com/a/54948581
Change-Id: Ic555ea9482645899f5b04040ddb6b24eadbf9062
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254606
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E301,E302,E303,E305,E306
Manually fix issues in project.py caused by misuse of block comments.
Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
- E121 continuation line under-indented for hanging indent
- E122 continuation line missing indentation or outdented
- E125 continuation line with same indent as next logical line
- E126 continuation line over-indented for hanging indent
- E127 continuation line over-indented for visual indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E121,E122,E125,E126,E127,E128,E129,E131
Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
We have a few files that we optionally symlink from the work tree
.git/ to the .repo/projects/ path. If they don't exist when we
first initialize, then we skip creating symlinks. If the files
are created later on under the work tree .git/, repo gets upset.
This can happen with the packed-refs file: if we don't have any
packed refs initially, we don't symlink it. But if git tries to
pack refs later on and creates the file, the project gets wedged.
We could create an empty file initially and then symlink it, but
for some files, it's not clear we want to always do that (e.g.
the .git/shallow setting). Instead, lets make handling of these
paths more dynamic. If they show up later on in the work tree
.git/ only, we'll take care of relocating & symlinking. This
also makes repo a little more robust and autorecovers incase a
path goes missing in one of the dirs.
Ideally we wouldn't monkey around at all here, but considering
the only option we give to users currently is to blow things
away with --force-sync, this seems a bit better.
Bug: https://crbug.com/gerrit/12324
Change-Id: Ia6960f1896ac6d890c762d7d053684a1c6ab2c87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254632
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This reverts commit 4abf8e6ef8.
The curl process for updating the cookie file is not atomic. When
fetching many bundles in parallel, we can sometimes corrupt the file
causing it to be cleared. Since users should manage gitcookies on
their own, leave it read-only.
Bug: https://crbug.com/gerrit/12300
Change-Id: Id472c99b197bc4cf8533c649f8881509f38643c1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254092
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Since tempfile.mkstemp() returns a file handle in binary mode,
make sure we turn our strings into bytes before writing.
Bug: https://crbug.com/gerrit/12043
Change-Id: I3e84d595e84b8bc12a1fbc7fd0bb3ea0ba2832b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254393
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Some Android/Nest manifests are using <linkfile> with src="." to
create stable paths to specific projects. Allow that specific
use case as it seems reasonable to support.
Bug: https://crbug.com/gerrit/11218
Change-Id: I16dbe8d9fe42ea45440afcb61404c753bff1930d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254330
Reviewed-by: Chanho Park <parkch98@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Add a helper function to unify the duplication of finding the full
path to the symbolic HEAD ref. This makes it easy to handle git
worktrees where .git is a file rather than a dir/symlink.
Bug: https://crbug.com/gerrit/11486
Change-Id: I9f794f1295ad0d98c7c13622f01ded51e4ba7846
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254074
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
These files are not directly executable, so drop the +x bits.
Change-Id: Iaf19a03a497686cc21103e7ddf08073173440dd1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254076
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* Mirrors may contain multiple projects, some of which may not
always contain the default revision.
* Only fetch the default revision explicitly if
'--current-branch' is set.
* Fixes breakage casued by
commit 6856f98467
"Fix repo mirror with --current-branch"
Bug: https://crbug.com/gerrit/12274
Change-Id: Iaafabe2992f76f3644b841f24245d3e19c9515a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253093
Reviewed-by: Kuang-che Wu <kcwu@chromium.org>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Chirayu Desai <chirayudesai1@gmail.com>
Reject paths in <copyfile> & <linkfile> that try to use symlinks or
non-file or non-dirs.
We don't fully validate <linkfile> when src is a glob as it's a bit
complicated -- any component in the src could be the glob. We make
sure the destination is a directory, and that any paths in that dir
are created as symlinks. So while this can be used to read any path,
it can't be abused to write to any paths.
Bug: https://crbug.com/gerrit/11218
Change-Id: I68b6d789b5ca4e43f569e75e8b293b3e13d3224b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233074
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Before a2cd6aeae8, "repo mirror with --current-branch" fetches git data
using command
git fetch --progress --update-head-ok cros --tags
No refspec is specified, thus it fetches default refspec, which is +refs/heads/*:refs/heads/*
After a2cd6aeae8, the fetch command became
git fetch --progress --update-head-ok cros --tags +refs/tags/*:refs/tags/*
It did not only add tags refspec, but also suppressed the fetching of default refspec.
In other words, repo mirrors doesn't work if current_branch_only=True.
This CL explicitly adds the default refspec to command line if none is
specified.
Bug: https://crbug.com/gerrit/11990
Change-Id: Iadcf7b9aa50f53c47132cfe6c53b3fb2076ebca2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/246632
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Kuang-che Wu <kcwu@chromium.org>
Series of steps:
* Create a local "b1" branch with `repo start b1` that tracks a remote
branch (totally fine)
* Manually create a local "b2" branch with `git branch --track b1 b2`
that tracks the local "b1" (uh-oh...)
* Delete the local "b1" branch manually or via `repo prune` (....)
* Try to process the "b2" branch with `repo prune`
Since b2 tracks a branch that no longer exists, everything blows up
at this point as we try to probe the non-existent ref. Instead, we
should flag this as unknown and leave it up to the user to resolve.
This probably could come up if a local branch was tracking a remote
branch that was deleted from the server, and users ran something like
`repo sync --prune` which cleaned up the remote refs.
Bug: https://crbug.com/gerrit/11485
Change-Id: I6b6b6041943944b8efa6e2ad0b8b10f13a75a5c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236793
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Kirtika Ruchandani <kirtika@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the local branch changed state since its last upload, the data
cached in .git/config related to the last uploaded CL might not be
that relevant. If we're able to fast forward merge to the latest
tree state, then let's do that. This would be akin to checking
out a detached head before syncing where we already switch state.
If we aren't able to fast forward merge, then it's not a big deal
as we'll continue on to the existing branch checking logic.
This would be easy to reproduce by doing something like:
$ repo start foo .
$ git revert HEAD
$ repo upload --cbr .
$ git reset --hard HEAD^
<CL is merged>
$ repo sync .
<we can fast forward>
Change-Id: I7d62f3d1ba5314a349d85b4dbb0ec8352eca18bb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238552
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If you run `repo sync -l foo` without first `repo sync -n foo`,
repo sets up an invalid gitdir tree that gets wedged and requires
manual recovery. Add a sanity check to abort cleanly first.
Change-Id: Iad865ea860a3f1fd2f39ce683fe66bd4380745a5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244732
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Use open() as a context manager to simplify the close logic and make
the code easier to read & understand. This is also more Pythonic.
Change-Id: I579d03cca86f99b2c6c6a1f557f6e5704e2515a7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244734
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Hitting Ctrl-C in the middle of this func will leave the .git in a
bad state that requires manual recovery. The code tries to catch
all exceptions and recover by deleting the incomplete .git dir, but
it omits KeyboardInterrupt which Exception misses.
We could add that to the recovery path, but we can make this more
robust with a different approach: set up everything in .git.tmp/
and only move it to .git/ once we've fully initialized it.
Change-Id: I0f5b97f2e19fc39cffc3e5e23993a2da7220f4e3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244733
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If the repo client checkout is in an incomplete sync state, the work
git repo might be in a bad way. Turn errors parsing HEAD into None
since callers of CurrentBranch already need to account for it.
Change-Id: Ia7682e29ef4182006b1fb5f5e57800f8ab67a9f4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239239
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
If `git diff` fails in any project checkout (e.g. an incomplete
sync), make sure we print that error clearly rather than blowing
up, and exit non-zero in the process.
Bug: https://crbug.com/gerrit/11613
Change-Id: I12f278427cced20f23f8047e7e3dba8f442ee25e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239236
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
When displaying progress bars, we use \r to reset the cursor to the
start of the line before showing the new update. This assumes the
new line will fully erase whatever was displayed there previously.
The "done" codepath tries to handle this by including a few extra
spaces at the end of the message to "white out" what was there.
Lets replace that hack with the standard ECMA escape sequence that
clears the current line completely. This is the CSI "erase in line"
sequence that the terminal will use to delete all content. The \r
is still needed to move the cursor to the start of the line. Using
this sequence should be OK since we're already assuming the terminal
is ECMA compliant with our use of coloring sequences. We also put
the \r after the CSI sequence on the off chance the terminal can't
process it and displays a few bytes of garbage.
The other improvement is to the syncbuffer API. When it dumps its
status information, it almost always comes after a progress bar
update which leads to confusing comingled output. Something like:
Fetching projects: 100% (2/2) error: src/platform2/: branch ...
Since the progress bar is "throw away", have the syncbuffer reset
the current output to the start of the line before showing whatever
messages it has queued.
Bug: https://crbug.com/gerrit/11293
Change-Id: I6544d073fe993d98ee7e91fca5e501ba5fecfe4c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236615
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
This makes it way easier to recover from forgetting to run repo start
before committing: just run `repo start -b new-branch`, instead of
all that tedious mucking around with reflogs.
Change-Id: I56d49dce5d027e28fbba0507ac10cd763ccfc36d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232712
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
There is a standard Python "trace" module, so having a local trace.py
prevents us being able to import that. Rename the module to avoid.
Change-Id: I23e29ec95a2204bb168a641323d05e76968d9b57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234832
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The GitCommand Wait helper takes care of decoding bytes to strings
for us. That means we don't have to decode stdout ourselves which
is what our local rev list, ls-remote, and generic get_attr helpers
were doing.
If we don't use Wait though to capture the output but instead go
directly to the subprocess stdout, we do have to handle decoding
ourselves. This is what the diff helpers were doing.
Bug: https://crbug.com/gerrit/10418
Change-Id: I057ca245af3ff18d6b4a074e3900887f06a5617d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233076
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
As we convert repo to support Python 3, the version of Python that we
use might not be the version that repo hooks users have written for.
Since repo upgrades are not immediate, and not easily under direct
control of end users (relative to the projects maintaining the hook
code), allow hook authors to declare the version of Python that they
want to use.
Now repo will read the shebang from the hook script and compare it
against the version of Python repo itself is running under. If they
differ, we'll try to execute a separate instance of Python and have
it load & execute the hook. If things are compatible, then we still
use the inprocess execution logic that we have today.
This allows repo hook users to upgrade on their own schedule (they
could even upgrade to Python 3 ahead of us) without having to worry
about their supported version being exactly in sync with repo's.
Bug: https://crbug.com/gerrit/10418
Change-Id: I97c7c96b64fb2ee465c39b90e9bdcc76394a146a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228432
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
A new option, --partial-clone is added to 'repo init' which tells repo
to utilize git's partial clone functionality, which reduces disk and
bandwidth usage when downloading by omitting blob downloads initially.
Different from restricting clone-depth, the user will have full access
to change history, etc., as the objects are downloaded on demand.
Change-Id: I60326744875eac16521a007bd7d5481112a98749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
This fix exception with python3 with stack-trace:
error: Cannot fetch platform_external_grpc-grpc-java.git (UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 640: invalid start byte)
[...]
File "[...]project.py", line 2255, in _IsValidBundle
if f.read(16) == '# v2 git bundle\n':
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
Even if we ask 16 characters, python buffered decoder will try to decode more in the buffer
The patch works for python2 and python3, and open the file in byte mode so that decoding is not attemped
Signed-off-by: Pierre Tardy <tardyp@gmail.com>
Change-Id: I837ae3c5cd724b34670fc2a84e853431f482b20d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/224642
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
Currently we read the binary stream from the subprocess code directly
before waiting for it to finish, but there's no need to do so as we
aren't streaming the output to the user. This also means we pass up
binary data to the caller as we don't go through GitCommand's internal
logic which decodes the stream as utf-8.
Simplify the code by calling Wait first, then splitting the entire
captured output in one line.
Bug: https://crbug.com/gerrit/10418
Change-Id: I7a57904be8cb546a229980fb79c829fc3df31e7d
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.
Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
During sync, repo runs `git read-tree --reset -u -v HEAD` which causes
git-lfs's smudge filter to run, which fails because git-lfs does not
work with bare repositories.
This was fixed in I091ff37998131e2e6bbc59aa37ee352fe12d7fcd to
automatically disable this smudge filter. However, later versions of
Git (2.11.0) introduced a new filter protocol [1], to avoid spawning
a new command for each filtered file. This was implemented in Git-LFS
1.5.0 [2].
This patch fixes the issue by setting the git lfs process filter, in
addition to the smudge filter. For any projects that have LFS objects,
`git lfs pull` must still be executed manually afterwards.
[1] edcc85814c
[2] https://github.com/git-lfs/git-lfs/pull/1617
Bug: https://crbug.com/gerrit/10911
Change-Id: I277fc68fdefc91514a2412b3887e3be9106cab48
When syncing a lot of projects in parallel, it is not otherwise
clear which one of them has failed to init work tree.
Change-Id: I8edfb4955023389a499e99cfa511bdc0d2850ba2
This allows projects to include submodules inside of
projects that use repo without repo incorrectly believing
the area is dirty just because a submodule has updates.
This is in line with git porcelain commands which generally
require a commandline flag to include submodules (git add,
git rebase).
Change-Id: Ide8a292162a42ab35145b5c4ca8ca0d020cdfe81
The newly introduced "Already have persistent ref" message prevents
repo from overwriting the last line when syncing quietly. Omit the
message when syncing quietly to clean up the output and to restore
the previous behaviour.
Change-Id: Idf42751c67f95924d6de50092ba54d4c6fe93096
If a tag is rewritten on the server (which is bad), trying to recover
locally with `repo sync --force-sync` doesn't actually work. People
have to manually delete things themselves to fix syncing. While tags
should never be rewritten in practice, allow users to easily recover
from broken servers.
We updated some of these code paths already (see commit 6e53844f1e
"Allow clobbering of existing tags from remote."), but the incremental
update flow was missed.
Bug: b/120778183
Bug: chromium:932651
Test: delete local tag & recreate to diff commit, then check
`repo sync` & `repo sync --force-sync` behavior
Change-Id: I3648f7d2526732c06016b691a9a36c003157618d
Even if dots are used as separators for Git config keys, they are not
forbidden as part of submodule names. This fixes the issue of submodules
with a name like e.g. "long.js" to be skipped from checkout.
Change-Id: I77da07925ad207fa3d043067dfbbcb4a1ebdac4d
When --ne/--no-emails is added to 'repo upload' command line, gerrit
server will not generate notification emails.
project.py:Project.UploadForReview method is modified to accept a
string recognizable by gerrit to indicate different sets of destination
email addressees, but the upload command line allows only one option -
disable sending emails completely.
Default repo upload behavior is not being changed.
TEST=tried in the Chrome OS repo, observed that patches uploaded with
--ne or --no-emails indeed do not trigger any emails, while
patches uploaded without these command line options still trigger
email notifications.
Change-Id: I0301edec984907aedac277d883bd0e6d3099aedc
Pass options through the refspec for all transports, including ssh.
This means the behavior will be more consistent between the ssh and
https cases.
A downside is that this prevents passing special characters in
reviewer options. That already didn't work over https, so it seems
okay. It could be fixed by using push options instead.
Change-Id: Ia38d16e350cb8cb0de14463bfb3d9724e13bc4bf
* Add more file i/o wrappers in platform_utils to allow using
long paths (length > MAX_PATH) on Windows.
* Paths using the long path syntax ("\\?\" prefix) should never
escape the platform_utils API surface area, so that this
specific syntax is not visible to the rest of the repo code base.
* Forward many calls from os.xxx to platform_utils.xxx in various place
to ensure long paths support, specifically when repo decides to delete
obsolete directories.
* There are more places that need to be converted to support long paths,
this commit is an initial effort to unblock a few common use cases.
* Also, fix remove function to handle directory symlinks
Change-Id: If82ccc408e516e96ff7260be25f8fd2fe3f9571a
"repo init --reference" has two purposes: to decrease bandwidth used
at clone time, and to decrease disk usage afterward, by using the
reference repositories as an alternate store of objects even after
the clone. The downside is that it makes the borrowing repositories
dependent on the reference repositories, so it is easy to end up
with missing objects by mistake after a cleanup operation like "git
gc".
To prevent that, v2.3.0-rc0~30^2 (clone: --dissociate option to mark
that reference is only temporary, 2014-10-14), "git clone" gained a
--dissociate option that makes --reference reuse objects from the
reference repository at clone time but copy them over instead of
using the reference as an alternate. This is more straightforward to
use than plain --reference, at the cost of higher disk usage.
Introduce a --dissociate to "repo init" that brings the same benefits
to repo. The option is simply passed on to "git clone".
Change-Id: Ib50a549eb71e0a2b3e234aea57537923962a80d4
pylint is not used since bb5b1a0. The pyflakes cleanup mentioned in that
commit has not been done, but given that this project is no longer being
actively developed I don't think it's worth spending time doing it.
Leaving the pylint suppressions causes confusion because it leads people
to think that we are still using pylint.
Change-Id: If7d9f280a0f408c780f15915ffdb80579ae21f69
When someone does "repo download -c <project> <change>"
without specifying a patch number, by default patch 1 is
downloaded. An alternative is to look for the latest patch
and download the same when no explicit patch is given.
This commit does the same by identifying the latest patch
using "git ls-remote".
Change-Id: Ia5fa7364415f53a3d9436df4643e38f3c90ded58
Allow the 'default' and 'project' element in the manifest
file to apply "--no-tags" option equivalent.
Change-Id: I7e0f8c17a0e25cca744d45df049076d203c52ff5
Signed-off-by: YOUNG HO CHA <ganadist@gmail.com>
Put the correctly-expanded relative paths in objects/info/alternates.
From gitrepository-layout(5), this path should be "relative to the
object database, not to the repository".
Change-Id: I7b2027ae23cf7d367b80f5a187603c4cbacdb2de
Issue: when subproject url is a relative in .gitmodules
repo tool cannot handle this and cause:
"fatal: '***' does not appear to be a git repository
fatal: Could not read from remote repository."
issue.
Signed-off-by: Shouheng Zhang <shouheng.zhang@intel.com>
Change-Id: I2a24c291ea0074ba13a740b32a11c0c25975e72b
os.remove raises an exception when deleting read-only files on
Windows. Replace all calls with calls to platform_utils.remove,
which deals with deals with that issue.
Change-Id: I4dc9e0c9a36b4238880520c69f5075eca40f3e66
Without this change, '.git\HEAD' files, for examples, are sometime
read incorrectly resulting in the current branch to be reset to
"master" when running a "repo init -b xxx" on an already initialized
repository.
Change-Id: I48c7ef85ff81626edf156914329a560e14252f2a
* changes:
Port os.rename calls to work on Windows
Workaround shutil.rmtree limitation on Windows
Add support for creating symbolic links on Windows
Make "git command" and "forall" work on Windows
The output indicates that fetching happens even when it is skipped.
To avoid confusion, print a message when fetching is skipped for
an immutable ref so that the user knows when and why a fetch is skipped.
Change-Id: Id6e4812cebc5e57d379feb76a9d034af0b93043b
This option allow to bypass verification ssl certification while
establishing connection with Gerrit to upload review.
Change-Id: If2e15f5a273c18a700eb5093ca8a4d5a4cbf80cd
This reverts commit d88f53e2b9. I merged
it too hastily without paying enough attention to compatibility with
released Gerrit versions.
Change-Id: I4028d4737df1255f11e217da183a19a010597d5b
Considering that some users might expect changes created with
'-d' option are not public. Private changes may be a better
choice here than work-in-progress changes.
Change-Id: I46a8fb9ae38beb41cf96d6abe82bea6db2439669
This change adds options for git-repo tool to support private
changes and work-in-progress changes.
Change-Id: I343491f5949f06f1580d53f9cc0dee2dca09130f
The submodule argument to Sync_LocalHalf was missing in
MetaBranchSwitch, causing submodules not to get synced when the
-b/--manifest-branch argument to init is used.
Change-Id: Ie86d271abac2020725770be36ead83be3326e64b
Signed-off-by: Martin Kelly <mkelly@xevo.com>
os.rename fails on Windows if the destination exists, so replace
os.rename to platform_utils.rename which handles the platform
differences.
Change-Id: I15a86f10f65eedee5b003b80f88a0c28a3e1aa48
By default, shutil.rmtree raises an exception when deleting readonly
files on Windows.
Replace all shutil.rmtree with platform_utils.rmtree, which adds an
error handler to make files read-write when they can't be deleted.
Change-Id: I9cfea9a7b3703fb16a82cf69331540c2c179ed53
Replace all calls to os.symlink with platform_utils.symlink.
The Windows implementation calls into the CreateSymbolicLinkW Win32
API, as os.symlink is not supported.
Separate the Win32 API definitions into a separate module
platform_utils_win32 for clarity.
Change-Id: I0714c598664c2df93383734e609d948692c17ec5
Project names are stored as path using the '/' file separator, and
stored in a dictionary as keys.
Change-Id: Ide40dfe840958ac0d46caae5f77f1a49d71c9d90
repo sync can sync submodules via the --fetch-submodules option.
However, if the manifest repo has submodules, those will not be synced.
Having submodules in the manifest repo -- while not commonly done -- can
be useful for inheriting a manifest from another project using <include>
and layering changes on top of it. In this way, you can avoid having to
deal with merge conflicts between your own manifests and the other
project's manifests (for example, if you're managing an Android fork).
Add a --submodule option to init that automatically syncs the submodules
in the manifest repo whenever the manifest repo changes.
Change-Id: I45d34f04517774c1462d7f233f482d1d81a332a8
Signed-off-by: Martin Kelly <mkelly@xevo.com>
Currently, if direct fetch of a sha1 is not supported by git server and
depth option is used, we fallback on syncing the upstream branch by
ignoring depth option.
This fallback doesn't work in next 2 cases:
(1) upstream attribute is not specified in manifest
(2) depth option is passed to repo init command
(not with clone-depth attribute in manifest)
This commit do the following:
- fixes (1) by updating condition used to apply fallback
first we retry with depth set to None, then by syncing all branches
- fixes (2) by passing depth as argument of _RemoteFetch() method
thus, its value is not set again to depth value passed to repo init
command when applying fallback
Change-Id: Ifd6fffafc49ba229df624b0d7b64c83d47619d17
A recent backward incompatible change created confusion and loss of
productivity and highlighted the very limited amount of information
provided when repo sync fails; merely recommending to --force-sync
and blow-up git repos without any hint as to why. The addition of
this basic _error(...) call would have provided a clue and will in
the future.
BUG=Issue 232
TEST=simulate a breakage similar to the ones reported at
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-dev/2-0oCy_CX5s
cd .repo/projects/src/third_party/libapps.git/
file info; rm info; ln -s wronglink info
cd -
repo sync src/third_party/libapps/
# error message now shows the failure
Change-Id: Idd2f177a096f1ad686caa8c67cb361d594ccaa57
The --quiet option reduces the output to just
a list of projects with modified workspaces (and
orphans if -o is specified)
A common use case is when performing a full-workspace
merge. The integrator will kick-off a merge via:
repo forall -c git merge <some tag>
And then produce a short list of conflicted projects via:
repo status -q
The integrator can then iteratively fix and clean up all conficted
components. The merge is complete when:
repo status -q
returns no output.
Change-Id: Ibbba8713eac35befd8287c95948874e23fd5c7e2
