Commit Graph

682 Commits

Author SHA1 Message Date
Mike Frysinger
401c6f0725 init: make --manifest-url flag optional
Since the --manifest-url flag is always required when creating a new
checkout, allow the url to be specified via a positional argument.
This brings it a little closer to the `git clone` UI.

Change-Id: Iaf18e794ae2fa38b20579243d067205cae5fae2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297322
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2021-02-18 20:38:47 +00:00
Mike Frysinger
f37b9827a9 git_command: rework stdin handling
We only provide input to GitCommand in one place, so inline the logic
to be more synchronous and similar to subprocess.run.  This makes the
code simpler and easier to understand.

Change-Id: Ibe498fedf608774bae1f807fc301eb67841c468b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297142
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-17 15:15:16 +00:00
Mike Frysinger
c47a235bc5 trim redundant pass statements
Clean up a few linter warnings.

Change-Id: I531d0263a202435d32d83d87ec24998f4051639c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297062
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-16 19:23:00 +00:00
Mike Frysinger
fb21d6ab64 sync: use subprocess.run to verify tags
The code is a bit simpler & easier to reason about.

Change-Id: I149729c7d01434b08b58cc9715dcf0f0d11201c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297022
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-16 16:26:41 +00:00
Raman Tenneti
21dce3d8b3 init: added --use-superproject option to clone superproject.
Added --no-use-superproject to repo and init.py to disable use of
manifest superprojects.

Replaced the term "sha" with "commit id".

Added _GetBranch method to Superproject object.

Moved shared code between init and sync into SyncSuperproject function.
This function either does git clone or git fetch. If git fetch fails
it does git clone.

Changed Superproject constructor to accept manifest, repodir and branch
to avoid passing them to multiple functions as argument.

Changed functions that were raising exceptions to return either True
or False.

Saved the --use-superproject option in config as repo.superproject.
Updated internal-fs-layout.md document.

Updated the tests to work with the new API changes in Superproject.

Performance for the first time sync has improved from 20 minutes to
around 15 minutes.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL.

$ repo init took around 20 seconds longer because of cloning of superproject.

$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real	0m35.919s
user	0m21.947s
sys	0m8.977s

First run
$ time repo sync --use-superproject
...
real	16m41.982s
user	100m6.916s
sys	19m18.753s

No difference in repo sync time after the first run.

Bug: [google internal] b/179090734
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707

Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-11 18:59:29 +00:00
Mike Frysinger
e3315bb49a diffmanifests/sync: simplify repodir lookup
We have access to repodir on the command object itself, so we don't
need to pull it indirectly out of the manifest object.

Change-Id: I8688fb1c84979825efa966dc787e78c6f7ba3823
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296542
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-11 02:38:53 +00:00
Mike Frysinger
38867fb6d3 git_config: add SetBoolean helper
A little sugar simplifies the code a bit.

Change-Id: Ie2b8a965faa9f9ca05c7be479d03e8e073cd816d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296522
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-11 01:48:12 +00:00
Raman Tenneti
8d43dea6ea sync: pass --bare option when doing git clone of superproject.
Changed "git pull" to "git fetch" as we are using --bare option. Used the
following command to fetch:
  git fetch origin +refs/heads/*:refs/heads/* --prune

Pass --branch argument to Superproject's UpdateProjectsRevisionId function.

Returned False/None when directories don't exist instead of raise
GitError exception from _Fetch and _LsTree functions. The caller of Fetch
does Clone if Fetch fails.

Tested the code with the following commands.

$ ./run_tests -v

Tested the init and sync code by copying all the repo changes into my Android
AOSP checkout and running repo sync with --use-superproject option.

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-02-08 17:34:55 +00:00
Raman Tenneti
1fd7bc2438 sync: superproject performance changes.
After updating all project’s revsionIds with the SHAs from superproject,
write the updated manifest into superproject_override.xml file. Reload
that file for future Reloads. This file is created in exp-superproject
directory.

Moved most of the code that is superproject specific into
git_superproject.py and wrote test code.

If git pull fails, did a git clone of the superproject.

We saw performance gains for consecutive repo sync's. The time to sync
went down from around 120 secs to 40 secs when repo sync is executed
consecutively.

Tested the code with the following commands.

$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.

First run
$ time repo sync --use-superproject
...
real	21m3.745s
user	97m59.380s
sys	19m11.286s

After two consecutive sync runs
$ time repo sync -c -j8 --use-superproject
real	0m39.626s
user	0m29.937s
sys	0m38.155s

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>

Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-07 22:25:38 +00:00
Anders Björklund
2a2da80ba6 sync: Disable info about disabling pruning when quiet
If you have a lot of shared projects, it spams.

Bug: https://crbug.com/gerrit/13961
Change-Id: If3f5baef65930830af9a2cd01a1b593dd518ab09
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294049
Tested-by: Anders Björklund <anders.bjorklund.2@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-22 11:43:13 +00:00
Raman Tenneti
6a872c9dae sync: Added --use-superproject option and support for superproject.
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,

.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-21 19:41:52 +00:00
Mike Frysinger
0588f3dc52 version: add remote tracking information
This tells us what --repo-rev the user is using.

Bug: https://crbug.com/1164415
Change-Id: Idb6c48e6ca5a4783c529717e6be38266bf7038b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293143
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-08 20:41:27 +00:00
Mike Frysinger
acf63b2892 drop pyversion & is_python3 checking
We're committed to Python 3 at this point, so purge all the
is_python3 related dynamic checks.

Bug: https://crbug.com/gerrit/10418
Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:58 +00:00
Mike Frysinger
784ccfc040 strip python2-only coding:utf-8 & print_function settings
We're committed to Python 3 at this point, so clean up boilerplate.

Bug: https://crbug.com/gerrit/10418
Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:05 +00:00
Chris McDonald
8add62325d Add parallelism to 'branches' command
Spread the operation of querying which local branches exist across a
pool of processes and build the name map of projects -> branches as
these tasks finish rather than blocking on the entire query. The search
operations are submitted in batches to reduce the overhead of interprocess
communication. The `chunksize` argument used to control this batch size
was selected by incrementing through powers of two until it stopped being
faster.

Change-Id: Ie3d7f799ee8e83e5058536caf53e2979175408b7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291342
Tested-by: Chris Mcdonald <cjmcdonald@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-12-14 23:35:12 +00:00
Mike Frysinger
5e2f32fe13 init: reset hard to --repo-rev
When updating the tracking ref to whatever the user requested,
make sure we reset state completely rather than trying to update
the ref to it.  This avoids confusing git as to the current state
of the tree, and is more inline with user intentions: if they made
a local change to the checkout, but ran repo init with a specific
rev, we shouldn't stay wedged forever until they manually clean it
all up.

Bug: https://crbug.com/gerrit/12801
Change-Id: Ieba8d9c15781b4d0649bf01c7460694da63387b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290923
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-12-06 21:46:30 +00:00
Remy Bohmer
7f7acfe9fd Concentrate the RepoHook knowledge in the RepoHook class
The knowledge about running hooks and all its exception handling
is scattered over multiple files. This makes the code harder
to read, but also it requires duplication of logic in case
other RepoHooks are added to different commands.
This refactoring also creates uniform behavior of the hooks
across multiple commands and it guarantees the re-use of the same
arguments on all of them.

Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: Ia4d90eab429e4af00943306e89faec8db35ba29d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/277562
Tested-by: Remy Bohmer <oss@bohmer.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-11-23 09:59:16 +00:00
Mike Frysinger
8c1e9cbef1 manifest_xml: refactor manifest parsing from client management
We conflate the manifest & parsing logic with the management of the
repo client checkout in a single class.  This makes testing just one
part (the manifest parsing) hard as it requires a full checkout too.

Start splitting the two apart into separate classes to make it easy
to reason about & test.

Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-18 19:10:57 +00:00
Erwan Yvin
dc5c4d1d11 sync: respect --force-sync when fetching manifest project updates
The --force-sync option was being passed down for all updates except
for the manifest project, so add that there too.

Bug: https://crbug.com/gerrit/11034
Change-Id: I33818b652f828c6b847dbc70f1fedfac5ac17bbe
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228146
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-11-17 03:06:06 +00:00
Mike Frysinger
23411d3f9c manifest: add a --json output option
Sometimes parsing JSON is easier than parsing XML, especially when
the XML format is limited (which ours is).  Add a --json option to
the manifest command to quickly emit that form.

Bug: https://crbug.com/gerrit/11743
Change-Id: Ia2bb254a78ae2b70a851638b4545fcafe8c1a76b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280436
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-17 01:38:00 +00:00
Mike Frysinger
160748f828 upload: improve tip for fixing upload remote
Instead of assuming the repo client is tracking the "master" branch
of the manifest repo, use the existing info we have to display the
right info to the user.

Bug: https://crbug.com/gerrit/13339
Change-Id: I8b265f4b2e075fdc41909b1f3dff9aee87384353
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/287279
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-16 23:13:02 +00:00
Remy Bohmer
16c1328fec Move RepoHook class from project.py file to dedicated file
The project.py file is huge and contains multiple
classes. By moving it to seperate class files the code
becomes more readable and maintainable.

Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: Ida9d99d31751d627ae1ea0373418080696d2e14b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/281293
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <linux@bohmer.net>
2020-11-03 22:08:08 +00:00
Mike Frysinger
50a81de2bc init: use the remote default manifest branch
Instead of hardcoding "master" as our default, use the remote server's
default branch instead.  For most people, this should be the same as
"master" already.  For projects moving to "main", it means we'll use
the new name automatically rather than forcing people to use -b main.

For repositories that never set up a default HEAD, we should still use
the historical "master" default.

Bug: https://crbug.com/gerrit/13339
Change-Id: I4117c81a760c9495f98dbb1111a3e6c127f45eba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280799
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-09-09 05:46:07 +00:00
Kimiyuki Onaka
0501b29e7a status: Use multiprocessing for repo status -j<num> instead of threading
This change increases the speed of the command with parallelization with
processes.  The parallelization with threads doesn't work well, and
increasing the number of jobs to many (8 threads ~) didn't increase the speed.
Possibly, the global interpreter lock of Python affects.

Bug: https://crbug.com/gerrit/12389
Change-Id: Icbe5df8ba037dd91422b96f4e43708068d7be924
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279936
Tested-by: Kimiyuki Onaka <kimiyuki@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-09-09 03:52:24 +00:00
Mike Frysinger
4e1fc1013c manifest: drop support for local_manifest.xml
We deprecated this 8 years ago.  Time to drop it to simplify the code
as it'll help with refactoring in this module to not migrate it.

Change-Id: I2deae5496d1f66a4491408fcdc95cd527062f8b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280798
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
2020-09-08 17:00:06 +00:00
Mike Frysinger
0578ebf61a init: reject unknown args
If you pass args to `repo init` when first creating a checkout, the
repo launcher throws an error.  But the init subcommand that runs in
an existing checkout silently ignores them.  Throw a proper error.

Change-Id: I433bfcc73902d25f6b6a2974e77f6a977a75ed16
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279696
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-09-02 07:53:16 +00:00
Mike Frysinger
80944b538d upload: exit non-zero when preupload hooks fail
Bug: https://crbug.com/gerrit/13159
Change-Id: Id140b619242c841223c6bc5d4aa0c37a7ce0219d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/276294
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-07-25 08:31:52 +00:00
Roger Shimizu
ac29ac397f subcmds/sync.py: Fix typo in help
Change-Id: I70b63477241284249e395b8b0a220cb6f44f836b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/270183
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
Tested-by: David Pursehouse <dpursehouse@digital.ai>
2020-06-06 23:46:00 +00:00
Mike Frysinger
7ae210a15b sync: fix duplicate word in description
Bug: https://crbug.com/gerrit/12814
Change-Id: Id722eec9a59dded588f13bc605ce2c94b4047265
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268739
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
2020-05-24 23:51:28 +00:00
Xin Li
d79a4bc51b Make partial clone imply no-clone-bundle by default.
For large projects, clone bundle is useful because it provided a way to
efficiently transfer a large portion of git objects through CDN, without
needing to interact with git server. However, with partial clones, the
intention is to not download most of the objects, so the use of clone
bundles would defeat the space savings normally seen with partial
clones, as they are downloaded before the first fetch.

A new option, --clone-bundle is added to override this behavior.
Add a new repo.clonebundle variable which remembers the choice if
explicitly given from command line at repo init.

Change-Id: I03638474af303a82af34579e16cd4700690b5f43
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268452
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-05-21 19:47:36 +00:00
Sean McAllister
682f0b6426 Fix how we format the full destination branch when uploading.
If the dest-branch attribute is set in the project manifest, then
we need to push to that branch.  Previously, we would unconditionally
pre-pend the refs/heads prefix to it.  The dest-branch attribute is
allowed to be a ref expression though, so it may already have it.

Simple fix is to check if it already has the prefix before adding it.

Bug: crbug.com/gerrit/12770

Change-Id: I45d6107ed6cf305cf223023b0ddad4278f7f4146
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268152
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
2020-05-19 15:25:42 +00:00
Daniel Kutik
e7082ccb54 repo info findRemoteLocalDiff use short branch
When running repo info -d an error would be thrown saying:
  fatal: bad revision 'refs/remotes/m/refs/heads/master..'

Using the short branch name here instead, like 'refs/remotes/m/master..'
resolves this issue.

Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: I50ea92c45c011b2c3e3a63803decb88e7837a380
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/266578
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-05-12 16:15:01 +00:00
Sean McAllister
af908cb543 When writing manifest, set the dest-branch attribute for projects
When generating a revision locked manifest, we need to know what
ref to push changes to when doing 'repo upload'.  This information
is lost when we lock the revision attribute to a particular commit
hash, so we need to expose it through the dest-branch attribute.

Bug: https://crbug.com/1005103
Test: manual execution
Change-Id: Ib31fd77ad8c9379759c4181dac1ea97de43eec35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263572
Tested-by: Sean McAllister <smcallis@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-04-20 16:35:02 +00:00
Sean McAllister
74e8ed4bde Expose upstream and dest-branch attributes through environment
Recent changes in ChromeOS Infra to ensure we're reading from
snapshot manifests properly have exposed several bugs in our
assumptions about manifest files.  Mainly that the revision field
for a project does _not_ have to refer to a ref, it can just be
a commit hash.

Several places assume that the revision field can be parsed as a
ref to get the branch the project is on, which isn't true.  To fix
this we need to be able to look at the upstream and dest-branch
attributes of the repo, so we expose them through the environment
variables set in `repo forall`.

Test: manual 'repo forall' run
Bug: https://crbug.com/1032441

Change-Id: I2c039e0f4b2e0f430602932e91b782edb6f9b1ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263132
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
2020-04-16 18:42:53 +00:00
Karsten Pfeiffer-Raddatz
d086467012 forall.py: Close file after removing the stream
In order to remove the stream fileno() will be called on the filedescriptor.
If the file is already closed fileno() will raise an error and forall
will fail.

Bug: https://crbug.com/gerrit/12563
Change-Id: Ib7b895fe881c844e3eb3672b011fdcdbdae63024
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/262838
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Karsten Pfeiffer-Raddatz <raddatz.karsten@gmail.com>
2020-04-14 06:49:31 +00:00
George Engelbrecht
9bc283e49b sync: add retry to fetch operations
Add retries with exponential backoff and jitter to the fetch
operations. By default don't change behavior and enable
behind the new flag '--fetch-retries'.

Bug: https://crbug.com/1061473

Change-Id: I492710843985d00f81cbe3402dc56f2d21a45b35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/261576
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: George Engelbrecht <engeg@google.com>
2020-04-02 21:17:54 +00:00
Mike Frysinger
b4a6f6d798 version: include tag commit date for easy reference
This is more for users trying to get a sense of how old/new their
current version of repo is when debugging issues.

Change-Id: Ifb413c679bb8c8dbf4f9334137adf086bb000a68
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/261192
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-03-31 03:27:57 +00:00
Mike Frysinger
e257d56665 version: fix running under Python 2
This gets the unittests passing again for now.

Change-Id: Ibed430a305bc26b907ad0ea424c7eec7de37e942
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259994
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-03-25 04:56:07 +00:00
Mike Frysinger
3599cc3975 init: respect --repo-rev changes
We respect this option when running the first `repo init`, but then
silently ignore it once the initial sync is done.  Make sure users
are able to change things on the fly.

We refactor the wrapper API to allow reuse between the two init's.

Bug: https://crbug.com/gerrit/11045
Change-Id: Icb89a8cddca32f39a760a6283152457810b2392d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260032
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-03-25 04:55:50 +00:00
Mike Frysinger
78964472ad download: add a --branch option
This allows people to quickly create new branches when pulling down
changes rather than having to juggle the git calls themselves.

Bug: https://crbug.com/gerrit/11609
Change-Id: Ie6a4d05e9f4e9347fe7f7119c768e6446563ae65
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259855
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-03-23 00:31:10 +00:00
Mike Frysinger
05097c6222 download: unify error handling with sub git calls
We gracefully handle cherry-pick errors, but none of the others
which means people get confusing Python tracebacks.  Move the
main logic in a single GitError try block so we can show pretty
error messages for all of them.

Change-Id: I52cdf6468d21a98de7f65b86d5267b3caabd5af8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259854
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-03-23 00:28:54 +00:00
Mike Frysinger
915fda130e download: support -x when cherry-picking
This is a pretty common option for people to want too use, so include
it as a pass-thru option when cherry-picking.

Bug: https://crbug.com/gerrit/9418
Change-Id: I2a24c1ed7544541719caa4d3c0574347a151a1b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259853
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-03-23 00:27:52 +00:00
Mike Frysinger
ea43176de0 download: support --ff when cherry-picking
The git cherry-pick already supports this, so plumb the existing repo
option down.  Otherwise it's confusing when people use -c --ff and it
doesn't use that behavior.

Change-Id: Id68932ffa09204bb30b92a21aff185c00394a520
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259852
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-03-23 00:26:26 +00:00
Mike Frysinger
58ac1678e8 init: rename --repo-branch to --repo-rev
We refer to this as "revision" in help text, and in REPO_REV envvar,
so rename to --repo-rev to be consistent.  We keep --repo-branch for
backwards compatibility, but as a hidden option.

Bug: https://crbug.com/gerrit/11045
Change-Id: I1ecc282fba32917ed78a63850360c08469db849a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259352
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-03-18 00:24:43 +00:00
Mike Frysinger
7936ce8677 init: respect --repo-url changes
We respect this option when running the first `repo init`, but then
silently ignore it once the initial sync is done.  Make sure users
are able to change things on the fly.

Bug: https://crbug.com/gerrit/11045
Change-Id: I129ec5fec43a85067d555bb60c0d1ae02465f139
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258893
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-03-17 05:39:17 +00:00
Mike Frysinger
23c900f105 sync: warn if not tracking a branch
Since tracking a branch prevents repo from updating, make sure we
warn people about the situation when using `repo sync`.

Bug: https://crbug.com/gerrit/11045
Change-Id: I966513f510827cc93194f8df176c6745946bd739
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258892
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-03-17 05:38:19 +00:00
Mike Frysinger
bb930461ce subcmds: stop instantiating at import time
The current subcmds design has singletons in all_commands.  This isn't
exactly unusual, but the fact that our main & help subcommand will then
attach members to the classes before invoking them is.  This makes it
hard to keep track of what members a command has access to, and the two
code paths (main & help) attach different members depending on what APIs
they then invoke.

Lets pull this back a step by storing classes in all_commands and leave
the instantiation step to when they're used.  This doesn't fully clean
up the confusion, but gets us closer.

Change-Id: I6a768ff97fe541e6f3228358dba04ed66c4b070a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259154
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-03-17 00:08:52 +00:00
Mike Frysinger
d3639c53d5 subcmds: centralize all_commands logic
The branch->branches alias is setup in the main module when that
really belongs in the existing all_commands setup.

For help, rather than monkey patching all_commands to the class,
switch it to use the state directly from the module.  This makes
it a bit more obvious where it's coming from rather than this one
subcommand having a |commands| member added externally to it.

Change-Id: I0200def09bf4774cad8012af0f4ae60ea3089dc0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259153
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-03-17 00:08:52 +00:00
Mike Frysinger
f725e548db upload: add config setting for upload notifications
This allows people to set default e-mail notifications via gitconfig.

Bug: https://crbug.com/gerrit/12451
Change-Id: Ic04ea3b7df0c5603c491961112c5be8cabb9dddd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259014
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-03-15 08:31:35 +00:00
Connor Newton
8b40c00eab diffmanifests: honour --pretty-format when printing --raw
Enable using --pretty-format to build a custom subject line
even when using the --raw option.

Change-Id: I0c1e682d984e56698fe65939aa6de12a653cd0f1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258565
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Connor Newton <connor@ifthenelse.io>
2020-03-13 09:50:46 +00:00
Mike Frysinger
e20da3eeed sync: fix os.environ logic errors
This is a dict to index, not a function to call.

Change-Id: I0117eeaaa8b2ef4762ab6f0d22f9ffdaee961f52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258132
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-03-07 13:10:14 +00:00
Mike Frysinger
5f11eac147 launcher/version: include OS/CPU info in output
We often ask users what OS/version they're running when debugging.
Include that in the version output to simplify the process.

Change-Id: Ie480b6d1c874e6f4c6f4996a03795077b844f858
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256732
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-25 23:31:47 +00:00
Mike Frysinger
b0fbc7fb58 upload: drop support for drafts
Draft CLs were replaced by private/wip CLs in Gerrit instead years ago.

Change-Id: If4f3d6606aad40a6f1617a49681dfd45c64d3d37
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256673
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-25 20:58:09 +00:00
Mike Frysinger
fc1b18ae9e upload: allow users to set labels when uploading
Bug: https://crbug.com/gerrit/11801
Change-Id: I060465105b4e68ddfc815e572f62bf5dac2c1ffd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256614
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jack Rosenthal <jrosenth@chromium.org>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-24 23:35:47 +00:00
Sean McAllister
d38300c756 manifest: support optional --manifest-name
Still use the repo manifest by default as before, but gives us
the option of overriding it to support e.g.: using a subset of
the full manifest.

Change-Id: Ia42cd1cb3a0a58929d31bb01c9724e9d31f68730
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256372
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
2020-02-22 19:17:40 +00:00
Mike Frysinger
dcbfadf814 repo/init: improve basic progress messages
We produce some simple "Get" messages that aren't super clear as to
what they're doing, especially for people not familiar with repo.
Rephrase these to explicitly state the thing we're doing so it's
clear why we're downloading a particular source.

Bug: https://crbug.com/gerrit/11293
Change-Id: I0749504f17c5385c6c65274a274e0ae25b117413
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256455
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-22 08:23:51 +00:00
Mike Frysinger
edd3d45b35 repo/init: add --verbose flags
We don't use these for much yet, but init passes it down to the
project sync layers which already has support for verbose mode.

Change-Id: I651794f1b300be1688eeccf3941ba92c776812b5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256454
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-02-22 06:31:22 +00:00
Mike Frysinger
0b888912cb init: hide summary output when using --quiet
Change-Id: I5e30a6d6a1c95fb8d75d8b0f4d63b497e9aac526
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256452
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-22 04:54:28 +00:00
Mike Frysinger
e6e27b338b abandon: add support for --quiet
Also fix the normal output to write to stdout.

Change-Id: I6c117eea9cec08f5be9a44b90dbe9bf1f824ec95
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256114
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-20 06:14:00 +00:00
David Pursehouse
aa611a2ca2 sync: Fix flake8 E125 and E129 warnings
- E129 visually indented line with same indent as next logical line
- E125 continuation line with same indent as next logical line

Fixed automatically by:

 autopep8 --in-place --select E125,E129 subcmds/sync.py

Change-Id: Ia2f82f443e1e6a23ba22c6f9849c8485405aed0e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256092
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-20 02:17:08 +00:00
Mike Frysinger
c0d1866b35 project/sync: move DeleteProject helper to Project
Since deleting a source checkout involves a good bit of internal
knowledge of .repo/, move the DeleteProject helper out of the sync
code and into the Project class itself.  This allows us to add git
worktree support to it so we can unlock/unlink project checkouts.

Change-Id: If9af8bd4a9c7e29743827d8166bc3db81547ca50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256072
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-20 00:51:42 +00:00
Mike Frysinger
f81c72ed77 project: set core.repositoryFormatVersion=1 when using extensions
When using extensions, make sure we set the git repo format version
so git knows to check the extension compatibility.  We can add a
helper to the Project API to simplify this and make it foolproof.

Change-Id: I9ab6c32d92fe2b8e5df6e2b080ca71556332e909
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256035
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-02-19 23:44:10 +00:00
Mike Frysinger
7ff80afdf6 upload: add a --hashtag-branch option akin to -t
This will automatically add the current local branch name as a hashtag.

Bug: https://crbug.com/gerrit/10477
Change-Id: I888f8be8419c801f2d98b7a2ad2486799e94f32c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255893
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-19 18:12:28 +00:00
Mike Frysinger
979d5bdc3e add experimental git worktree support
This provides initial support for using git worktrees internally
instead of our own ad-hoc symlink tree.  It's been lightly tested
which is why it's not currently exposed via --help.

When people opt-in to worktrees in an existing repo client checkout,
no projects are migrated.  Instead, only new projects will use the
worktree method.  This allows for limited testing/opting in without
having to completely blow things away or get a second checkout.

Bug: https://crbug.com/gerrit/11486
Change-Id: Ic3ff891b30940a6ba497b406b2a387e0a8517ed8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254075
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-19 18:11:33 +00:00
Mike Frysinger
56ce3468b4 assume environment always accepts strings
Different Python & OS versions have different environ behavior wrt
accepted types & encoding.  Since we're migrating to be Python 3 only,
lets change our code to assume strings always work as that's what the
newer Python 3 does.  This will fail under Python 2 for some env vars,
mostly on Windows, but the effort of maintaining shim layers that can
handle these edge cases isn't worth it when we're dropping that code.

We leave the logic in the `repo` launcher for now as it is simple, and
we want it to be able to switch versions a bit longer than the rest of
the tree.

Here's the support table:
          |    *NIX      |         Windows           |
 Python 2 | ASCII string | str or bytes, not unicode |
 Python 3 | str or bytes | str only                  |

Windows uses strings natively in its environment all the time.  But it
doesn't allow unicode strings under Python 2, so we have to encode.

Python 2 on *NIX is funky in that it always lowers to ASCII, so we had
to manually encode to avoid errors regardless of unicode or str.

Python 3 on Windows & *NIX will accept strings.  *NIX will also accept
bytes but Windows will not.

Bug: https://crbug.com/gerrit/12145
Change-Id: I3cf8f95a06902754ea1f08ad4b28503f7063531b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248972
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-19 18:03:46 +00:00
Mike Frysinger
02aa889ecd upload: add support for --yes
This adds a CLI option to the existing autoupload gitconfig knob that
allows people to automatically answer "yes" to the various prompts.

Bug: https://crbug.com/gerrit/12368
Change-Id: I819ebca01b9a40240b33866ae05907c7469703e3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255892
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-19 16:04:14 +00:00
Mike Frysinger
819cc81c57 upload: add support for standard --dry-run
Change-Id: I69ea2f3170ba17bfb9e0e3771db4ecc66a736797
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255856
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-19 08:32:12 +00:00
Mike Frysinger
84685ba187 upload: add support for setting hashtags
This allows users to specify custom hashtags when uploading, both via
the CLI and via the same gitconfig settings as other upload options.

Bug: https://crbug.com/gerrit/11174
Change-Id: Ia0959e25b463e5f29d704e4d06e0de793d4fc77c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255855
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-19 08:31:18 +00:00
Mike Frysinger
7a753b8b18 upload: improve no pending CL error handling
Show clearer messages and exit non-zero if there's nothing to upload.

Change-Id: Icd9c13b9b1126610a409fc13d1d11bfc66f5e802
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255834
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-19 05:46:43 +00:00
Mike Frysinger
c58ec4dba1 avoid negative variables
Trying to use booleans with names like "no_xxx" are hard to follow due
to the double negatives.  Invert all of them so we only have positive
meanings to follow.

Change-Id: Ifd37d0368f97034d94aa2cf38db52c723ac0c6ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255493
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-19 00:24:43 +00:00
Collin Fijalkovich
e1191b3adb Open temporary cookie file as writable in sync.py
Named Temporary file defaults to mode 'w+b' which causes repo sync to
fail. By opening the tmpcookiefile in PersistentTransport.request as
writable, we are able to run sync successfully.

Bug: https://crbug.com/gerrit/12370
Test: Ran smartsync successfully
Change-Id: I01ddf915fc30eb3ff0e4d440a6f1aa261c63e88d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255692
Tested-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-02-18 19:20:53 +00:00
Mike Frysinger
af1e5dea35 resort a few module imports to follow PEP8
All the stdlib imports are supposed to come before any local imports.

Change-Id: I10c0335ba2ff715fd34c9eb91bfe6560e904df08
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255593
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-18 06:28:12 +00:00
Mike Frysinger
16f2fae16f diff: delete unused nested func
Change-Id: I43ab4bc944269e43a6cd7b2ac350c09b7c700a6c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255492
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-17 23:49:47 +00:00
Mike Frysinger
521d01b2e0 sync: introduce --verbose option
This allows us to control sync output better by having three levels
of output: quiet (only errors), default (progress bars), verbose (all
the things).  For now, we just put the chatty "already have persistent
ref" message behind the verbose level.

Bug: https://crbug.com/gerrit/11293
Change-Id: Ia61333fd8085719f3e99edb7b466cdb04031b67f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255414
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-17 17:02:27 +00:00
David Pursehouse
a46bf7dc2a flake8: Suppress "F821 undefined name" inline for Python 2 names
All of the instances of this are related to Python 2 names that
don't exist in Python 3, and the warnings are raised when running
flake8 on Python 3.

All of these will go away once we completely remove support for
Python 2, so just suppress them inline. We don't globally suppress
the check so that we will still see legitimate errors if/when they
occur in new code.

Change-Id: Iccf955f50abfc9f83b371fc0af6cceb51037456f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255039
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-15 04:45:16 +00:00
David Pursehouse
3cda50a41b pyflakes: Fix remaining "E501 line too long" warnings
We increased the max line length to 100 columns which got rid of
the majority of these warnings, but there were still a few lines
that exceeded 100 columns.

Change-Id: Ib3372868ca2297f83073a14f91c8ae3df9d0d0e6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254699
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-13 04:54:10 +00:00
Mike Frysinger
e19d9e1a65 sync: add a "finished" message
Some people find the existing output to be a bit confusing.  It spews
a lot of git output before exiting, but it's not exactly clear what
the final state is when things pass.  Add an explicit message.

Bug: https://crbug.com/gerrit/10501
Change-Id: I9de83b595d3185feb820005b8fc81c6adc55b357
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254732
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 20:54:57 +00:00
Mike Frysinger
6f1c626a9b drop old git_require checks
We've been requiring git-1.7.2 since Oct 2012, so we can safely drop
the individual checks sprinkled throughout the code base for older.

Change-Id: I1737fff7b3f27f475960b0bff9cb300aefd5d108
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253135
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 11:44:59 +00:00
David Pursehouse
145e35b805 Fix usage of bare 'except'
flake8 reports:

  E722 do not use bare 'except'

Replace them with 'except Exception' per [1] which says:

  Bare except will catch exceptions you almost certainly don't want
  to catch, including KeyboardInterrupt (the user hitting Ctrl+C) and
  Python-raised errors like SystemExit

  If you don't have a specific exception you're expecting, at least
  except Exception, which is the base type for all "Regular" exceptions.

[1] https://stackoverflow.com/a/54948581

Change-Id: Ic555ea9482645899f5b04040ddb6b24eadbf9062
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254606
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:49:25 +00:00
David Pursehouse
819827a42d Fix blank line issues reported by flake8
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E301,E302,E303,E305,E306

Manually fix issues in project.py caused by misuse of block comments.

Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:36:40 +00:00
David Pursehouse
abdf750061 Fix indentation issues reported by flake8
- E121 continuation line under-indented for hanging indent
- E122 continuation line missing indentation or outdented
- E125 continuation line with same indent as next logical line
- E126 continuation line over-indented for hanging indent
- E127 continuation line over-indented for visual indent
- E128 continuation line under-indented for visual indent
- E129 visually indented line with same indent as next logical line
- E131 continuation line unaligned for hanging indent

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E121,E122,E125,E126,E127,E128,E129,E131

Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:36:22 +00:00
David Pursehouse
54a4e6007a Fix various whitespace issues reported by pyflakes
- E201 whitespace after '['
- E202 whitespace before '}'
- E221 multiple spaces before operator
- E222 multiple spaces after operator
- E225 missing whitespace around operator
- E226 missing whitespace around arithmetic operator
- E231 missing whitespace after ','
- E261 at least two spaces before inline comment
- E271 multiple spaces after keyword

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E201,E202,E221,E222,E225,E226,E231,E261,E271

Change-Id: I367113eb8c847eb460532c7c2f8643f33040308c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254601
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:00:16 +00:00
David Pursehouse
42339d7e52 Remove redundant backslashes
fleka8 reports:

  E502 the backslash is redundant between brackets

Fixed automatically with autopep8:

  git-repo $ git ls-files | grep py$ | xargs autopep8 --in-place --select E502

Change-Id: I1486ae1d17206918474363daf518274c5be8daed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254602
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:45:44 +00:00
David Pursehouse
9090e804ab Remove unused imports
flake8 reports:

  F401 'name' imported but unused

Change-Id: Id45d6efa87ddf53f2c4a0f0c4136ea361ab1b746
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254592
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 05:40:10 +00:00
David Pursehouse
eeff3537de Fix tests for membership to use 'not in'
flake8 reports:

  E713 test for membership should be 'not in'

Change-Id: I4446be67c431b7267105b53478d2ceba2af758d7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254451
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:18:17 +00:00
David Pursehouse
8f78a83083 upload: Fix tests for object identity to use 'is not'
flake8 reports:

  E714 test for object identity should be 'is not'

Change-Id: Ib8c4100babaf952bbfe65fd56555ece8a958e4b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254450
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:17:49 +00:00
David Pursehouse
e5913ae410 Fix flake8 E251 unexpected spaces around keyword / parameter equals
Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place --select E251

Change-Id: I58009e1c8c91c39745d559ac919be331d4cd9e77
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254598
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:17:08 +00:00
David Pursehouse
086710465e upload: Fix flake8 E241 multiple spaces after ','
Change-Id: I3a65869f9d006027270a7826d7982950c0e6759a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254597
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 05:00:36 +00:00
David Pursehouse
719675bcec info: Fix formatting of block comment
flake8 reports:

  E265 block comment should start with '# '

While we're at it, add a period at the end of the comment sentence.

Change-Id: Icb7119079a1d64e6defafc3f6d24e99dbf16139d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254596
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 04:31:40 +00:00
Mike Frysinger
21c1575ee4 upload: add a --ignore-hooks option
When upload hooks fail, people are forced to use --no-verify to upload
CLs anyways.  When projects have flaky hooks, this trains people to
always use that option.  This is obviously bad: hooks might get fixed,
or some of the hooks are always good & people should review.

Lets add an --ignore-hooks option.  This still runs the hooks, but any
failures will be ignored and allow the user to upload anyways.

Bug: https://crbug.com/gerrit/12230
Change-Id: Ide2ac8a40a656bfcd6aae20c3ce8118e06bf909b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254452
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 04:18:49 +00:00
David Pursehouse
8f9e02231a Remove trailing blank lines
flake8 reports:

  W391 blank line at end of file

Change-Id: I5498b2de2d1268d4f1f4b9e1760f9fa93a6da4cd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254594
Tested-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-12 02:58:17 +00:00
Mike Frysinger
82caef67a1 repo: lower min version of git a bit
We were perhaps a bit too hasty to jump to git-2.10.  Existing LTS
releases of Ubuntu are quite old still: Trusty has 1.9 while Xenial
has 2.5.  While we plan on dropping support for those eventually as
we migrate to Python 3.6, we don't need to be so strict just yet on
the git versions.

We also want to disconnect the version the repo launcher requires
from the version the rest of the source tree requires.  The repo
launcher doesn't need as many features, and being flexible there
allows us more freedom to upgrade & rollback as needed.

So we'll allow git-1.7 again, but start warning on any users older
than git-1.9.  This aligns better with existing LTS releases, and
gives users a chance to start upgrading before we cut them off.

Change-Id: I140305dd8e42c9719c84e2aee0dc6a5c5b18da25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254573
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-12 00:28:03 +00:00
Mike Frysinger
5f2b045195 sync: change how we preserve objects in shared repos
Some automatic git operations will prune objects on us, and not just
the gc step.  Normally we don't care, but with shared projects, we
will have multiple git checkouts with refs that the others cannot
see, but with a shared object dir.  Any pruning of objects based on
refs in just one repo can easily break the others.

git-2.7.0 introduced a preciousObjects setting which tells git to
never prune objects for this exact scenario: there might be refs in
some location that git is unable to see.

Change-Id: I781de27c5bbe1d4c70f0187566141c9cce088bd8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254392
Reviewed-by: Nasser Grainawi <nasser@codeaurora.org>
Reviewed-by: David Riley <davidriley@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-11 23:58:43 +00:00
Jiri Tyr
83a3227b62 Fixing forall subcommand for Py3
Execution of 'repo forall -p -c' doesn't work with Py3 and ends up
with an error:

Got an error, terminating the pool: TypeError: can only concatenate
str (not "bytes") to str

That's fixed by using the decode() method.

Change-Id: Ice01aaa1822dde8d957b5bf096021dd5a2b7dd51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253659
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Jiri Tyr <jiri.tyr@gmail.com>
2020-02-10 10:52:27 +00:00
Mike Frysinger
e7c91889a6 remove spurious +x bits
These files are not directly executable, so drop the +x bits.

Change-Id: Iaf19a03a497686cc21103e7ddf08073173440dd1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254076
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2020-02-09 23:24:03 +00:00
Dan Willemsen
93293ca47f Fix inverted logic around [gitc-]init and -c
Instead of not using '-c' for '--current-branch' when using gitc, we
were only using '-c' when using gitc, so we still had the conflict with
the gitc option, and other users still couldn't use '-c'.

Test: repo init -u https://android.googlesource.com/platform/manifest; repo init -c
Test: repo gitc-init -u ... -b ... -c testing
Change-Id: I71e4950a49c281418249f0783c6a2ea34f0d3e2b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253795
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Dan Willemsen <dwillemsen@google.com>
2020-02-07 20:54:34 +00:00
Mike Frysinger
5a03308c5c sync: try to checkout repos across sync failures
Currently our default behavior is:
* Try to sync all repos
  * If any errors seen, exit
* Try to garbage collect all repos
  * If any errors seen, exit
* Try to update local project list
  * If any errors seen, exit
* Try to checkout out all local repos
  * If any errors seen, exit

Users find these incomplete syncs confusing, so lets try to complete
as much as possible by default and printing out summaries at the end.

Bug: https://crbug.com/gerrit/11293
Change-Id: Idd17cc9c3bbc574d8a0f08a30225dec7bfe414cb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238554
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-05 21:37:20 +00:00
Fredrik de Groot
cc960971f4 sync: add option to skip manifest update
The use case is any situation where your manifest does
not exist on server, but where you still want to do
full sync for the projects, without having your
workspace manifest switched to other branch or
forwarded to latest or similar.
This allows syncing to a historical manifest in git log,
that does not have a branch, as well as when integrating
something together that has not been pushed upstream yet.
Changes can also exist locally on a manifest that is
behind head, meaning not requiring rebase to latest.

Tested using:
  $ cd .repo/manifests/
  $ git checkout <any hash 1>
  $ <do local modifications>
  $ repo sync --no-manifest-update
  $ git checkout <any hash 2>
  $ repo sync --no-manifest-update

Change-Id: I0c9773aa8bc5876813a2e7d7fec697abcb2d9e94
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/246445
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-02-05 18:57:58 +00:00
Mike Frysinger
66098f707a init: handle -c conflicts with gitc-init
We keep getting requests for init to support -c.  This conflicts with
gitc-init which allocates -c for its own use.  Lets make this dynamic
so we keep it with "init" but omit it for "gitc-init".

Bug: https://crbug.com/gerrit/10200
Change-Id: Ibf69c2bbeff638e28e63cb08926fea0c622258db
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253252
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-05 16:00:10 +00:00
Diogo Ferreira
0eb2d3c8a0 init: Add '-c' as an alias to '--current-branch'
This makes it consistent with the short option for current-branch in
repo sync.

Change-Id: I2848e87f45a66ef8d829576d0c0c4c0f7a8636a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/241700
Tested-by: Diogo Ferreira <deovferreira@gmail.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-01-24 14:46:58 +00:00
Diogo Ferreira
e4d20372b2 info: Add the manifest revision
After Ib546f5ebbc8a23875fbd14bf166fbe95b7dd244e, repo info now displays
the current project revision in the 'Current revision' field.

While the output is more consistent, there are use cases for the
revision expression as shown in the manifest. This patch re-adds the
manifest revision as a new 'Manifest revision' field.

Change-Id: I50c1559dcb7ceb69af07352b956d78f85b8f592e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/240799
Tested-by: Diogo Ferreira <deovferreira@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-01-24 14:46:30 +00:00
Mike Frysinger
0b304c06ff help: unify command display
No functional changes, just unifying duplicate code paths.

Change-Id: I6afa797ca1e1eb90abdc0236325003ae070cbfb3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/247293
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-12-03 02:31:05 +00:00
Mike Frysinger
70c54dc255 upload/editor: fix bytes/string confusion
The upload module tries to turn the strings into bytes before passing
to EditString, but it combines bytes & strings causing an error.  The
return value might be bytes or string, but the caller only expects a
string.  Lets simplify this by sticking to strings everywhere and have
EditString take care of converting to/from bytes when reading/writing
the underlying files.  This also avoids possible locale confusion when
reading the file by forcing UTF-8 everywhere.

Bug: https://crbug.com/gerrit/11929
Change-Id: I07b146170c5e8b5b0500a2c79e4213cd12140a96
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/245621
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-11-16 23:55:30 +00:00
Mike Frysinger
6da17751ca prune: handle branches that track missing branches
Series of steps:
* Create a local "b1" branch with `repo start b1` that tracks a remote
  branch (totally fine)
* Manually create a local "b2" branch with `git branch --track b1 b2`
  that tracks the local "b1" (uh-oh...)
* Delete the local "b1" branch manually or via `repo prune` (....)
* Try to process the "b2" branch with `repo prune`

Since b2 tracks a branch that no longer exists, everything blows up
at this point as we try to probe the non-existent ref.  Instead, we
should flag this as unknown and leave it up to the user to resolve.

This probably could come up if a local branch was tracking a remote
branch that was deleted from the server, and users ran something like
`repo sync --prune` which cleaned up the remote refs.

Bug: https://crbug.com/gerrit/11485
Change-Id: I6b6b6041943944b8efa6e2ad0b8b10f13a75a5c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236793
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Kirtika Ruchandani <kirtika@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2019-11-16 19:55:02 +00:00
Mike Frysinger
3538dd224d sync: merge project updates with status bar
The current sync output displays "Fetching project" and "Checking out
project" messages and progress bar updates independently leading to a
lot of spam.  Lets merge these periodic outputs with the status bar to
get a little bit tighter output in the normal case.  This doesn't solve
all our problems, but gets us closer.

Bug: https://crbug.com/gerrit/11293
Change-Id: Icd627830af4dd934a9355b7ace754b56dc96cfef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244934
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-11-12 23:33:51 +00:00
Vadim Bendebury
dff919493a sync: report list of failing git trees
When repo sync fails because some git trees are not in clean state and
as such can not be rebased automatically, it is a pain to figure out
which trees are the culprits.

With this patch the list of offending trees is printed when repo sync
reports checkout errors.

TEST=ran 'repo sync' and observed the proper list of directories show
     up after the final error message

Bug: https://crbug.com/gerrit/11293
Change-Id: Icdf1a03e9014ecb184f331f513cc9a2efc7d11ed
Signed-off-by: Vadim Bendebury <vbendeb@google.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244053
Reviewed-by: Mike Frysinger <vapier@google.com>
2019-11-12 21:08:54 +00:00
Mike Frysinger
3164d40e22 use open context managers in more places
Use open() as a context manager to simplify the close logic and make
the code easier to read & understand.  This is also more Pythonic.

Change-Id: I579d03cca86f99b2c6c6a1f557f6e5704e2515a7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244734
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-11-12 03:44:39 +00:00
Mike Frysinger
b466854bed forall: add an --ignore-missing option
In CrOS, our infra has to deal with partial checkouts constantly
(for a variety of reasons).  To help reset back to a good state,
we run git commands via `repo forall`, but don't care about the
missing checkouts.  Add a flag so we can disambiguate between
missing repos and failing git subcommands.

Bug: https://crbug.com/1013377
Bug: https://crbug.com/1013623
Change-Id: Ie3498c6d111276c60d2ecedbba21bfa778588d50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/241935
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-10-22 14:10:34 +00:00
Mike Frysinger
f1c5dd8a0f info: fix "current" output
The "Current revision" field shows the revision as listed in the
manifest.  I think most users expect this to show the revision
that the git tree is checked out to instead.  Switch the output
to show that revision instead, and add a "Current branch" if it
matches a local branch.

Change-Id: Ib546f5ebbc8a23875fbd14bf166fbe95b7dd244e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239240
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-10-05 05:24:34 +00:00
Mike Frysinger
9775a3d5d2 info: allow NoSuchProjectError to bubble up
If the user passes in bad projects like `repo info asdf`, we currently
silently swallow those and do nothing.  Allow NoSuchProjectError to
bubble up to main which will handle & triage this correctly for us.

Change-Id: Ie04528e7b7a164293063a636813a73eaabdd5bc3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239238
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-10-01 05:53:09 +00:00
Mike Frysinger
9bfdfbe117 version: add source versions & User-Agents to the output
Depending on where/how repo is invoked, the active version might be
from a git tree, and it might be different from the .repo/repo.git/
version in the current repo client checkout.  Report both if they're
different so it's clearer.

Lets also include the two different User-Agent's that we set up when
talking to networked services.

Bug: https://crbug.com/gerrit/11144
Change-Id: I2ebb6e3ac30e374a8406cab3e4438087246a8c57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239234
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-10-01 05:47:35 +00:00
Mike Frysinger
72ab852ca5 grep: handle errors gracefully
If `git grep` fails in any project checkout (e.g. an incomplete
sync), make sure we print that error clearly rather than blowing
up, and exit non-zero in the process.

Bug: https://crbug.com/gerrit/11613
Change-Id: I31de1134fdcc7aaa9814cf2eb6a67d398eebf9cf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239237
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-10-01 05:45:58 +00:00
Mike Frysinger
0a9265e2d6 diff: handle errors gracefully
If `git diff` fails in any project checkout (e.g. an incomplete
sync), make sure we print that error clearly rather than blowing
up, and exit non-zero in the process.

Bug: https://crbug.com/gerrit/11613
Change-Id: I12f278427cced20f23f8047e7e3dba8f442ee25e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239236
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-10-01 05:44:09 +00:00
Mike Frysinger
dc1b59d2c0 forall: exit 1 if we skip any repos
If a repo doesn't exist (e.g. an incomplete sync), make sure we exit
non-zero when they get skipped.

Change-Id: Ifff711e374416b1e6b9b8da4fdc6f14b27ced450
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239235
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-10-01 05:43:51 +00:00
Mike Frysinger
e37aa5f331 rebase: add basic coloring output
This uses coloring style like we use in grep/forall already.

Change-Id: I317e2e47567a30c513083c48e7c7c40b091bb29a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238555
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-09-24 04:17:03 +00:00
Mike Frysinger
4a07798c82 rebase: add --fail-fast support
Lets switch the default rebase behavior to align with our new sync
behavior: we try to rebase all projects by default and exit/summarize
things at the very end if there were any errors.  Or if people want
to exit immediately, they can use the new --fail-fast option.

Change-Id: I436ac563f972b45de6ce9ad74da1e4870e584902
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/238553
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-09-24 02:06:26 +00:00
Mike Frysinger
fb527e3f52 sync: create dedicated manifest project update func
Cut out some more standalone code from Execute to make this func a
bit more manageable.  The manifest project update is pretty simple
and standalone, but still takes up a good chunk of what's left.

Change-Id: Idc2442d9def495eccd0a49cda203c44aef16f129
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236614
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2019-09-19 01:52:08 +00:00
Mike Frysinger
01d6c3c0c5 sync: create dedicated smart sync func
The smart sync logic takes up about 45% of the overall Execute func
and is about 100 lines of code.  The only effect it has on the rest
of the code is to set the manifest_name variable.  Since this func
is already quite huge, split the smart sync logic out.

Change-Id: Id861849b0011ab47387d74e92c2ac15afcc938ba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234835
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2019-09-11 23:24:11 +00:00
Theodore Dubois
60fdc5cad1 Add repo start option to create the branch based off HEAD
This makes it way easier to recover from forgetting to run repo start
before committing: just run `repo start -b new-branch`, instead of
all that tedious mucking around with reflogs.

Change-Id: I56d49dce5d027e28fbba0507ac10cd763ccfc36d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232712
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-09-04 04:34:50 +00:00
Stefan Müller-Klieser
46702eddc7 sync: fix deprecated command line option -f
In commit d9e5cf0e ("sync: invert --force-broken with --fail-fast") the
force-broken option has been deprecated. Accidentally the option has
been changed from Boolean to Value. This breaks all users of repo with:

  main.py: error: -f option requires an argument

This is easy to avoid by keeping the type.

Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Change-Id: Ia8b589cf41ac756d10c61e17ec8d76ba8f7031f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/235043
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-30 16:29:01 +00:00
Mike Frysinger
ae6cb08ae5 split out cli validation from execution
A common pattern in our subcommands is to verify the arguments &
options before executing things.  For some subcommands, that check
stage is quite long which makes the execution function even bigger.
Lets split that logic out of the execute phase so it's easier to
manage these.

This is most noticeable in the sync subcommand whose Execute func
is quite large, and the option checking makes up ~15% of it.

The manifest command's Execute can be simplified significantly as
the optparse configuration always sets output_file to a string.

Change-Id: I7097847ff040e831345e63de6b467ee17609990e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234834
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-28 03:54:11 +00:00
Mike Frysinger
898f4e6217 help: add a --help-all option to show all commands at once
This is useful when you want to scan all the possibilities of repo
at once.  Like when you're searching for different option names.

Change-Id: I225dfb94d2be78229905b744ecf57eb2829bb52d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/232894
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-27 07:07:24 +00:00
Mike Frysinger
d9e5cf0ee7 sync: invert --force-broken with --fail-fast
People seem to not expect the sync process to halt immediately if an
error is encountered.  It's also basically guaranteed to leave their
tree in an incomplete state.  Lets invert the default behavior so we
attempt to sync (both fetch & checkout) all projects.  If an error is
hit, we still exit(1) and show it at the end.

If people want the sync to abort quickly, they can use the new option
--fail-fast.

Bug: https://crbug.com/gerrit/11293
Change-Id: I49dd6c4dc8fd5cce8aa905ee169ff3cbe230eb3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234812
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-27 01:20:44 +00:00
Mike Frysinger
d5c306b404 rebase: pull out project-independent settings from the for loop
This makes the code a bit easier to read by doing all the project
independent settings first instead of repeating it for every for
loop iteration.

Change-Id: I4ff21296e444627beba2f4b86561069f5e9a0d73
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233554
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-08 17:41:40 +00:00
Mike Frysinger
a850ca2712 rebase/sync: use exit(1) for errors instead of exit(-1)
Callers don't actually see -1 (they'll usually see 255, but the exact
answer here is complicated).  Just switch to 1 as that's the standard
value tools use to indicate an error.

Change-Id: Ib712db1924bc3e5f7920bafd7bb5fb61f3bda44f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233553
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-08 05:07:31 +00:00
Mike Frysinger
a34186e481 sync: fix handling of -f and local checkouts
The partial clone rework (commit 745be2ede1
"Add support for partial clone") changed the behavior when a single repo
hit a failure: it would always call sys.exit() immediately.  This isn't
even necessary as we already pass down an error event object which the
workers set and the parent checks.  Just delete the exit entirely.

Change-Id: Id72d8642aefa2bde24e1a438dbe102c3e3cabf48
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233552
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-08-08 02:54:39 +00:00
Xin Li
745be2ede1 Add support for partial clone.
A new option, --partial-clone is added to 'repo init' which tells repo
to utilize git's partial clone functionality, which reduces disk and
bandwidth usage when downloading by omitting blob downloads initially.
Different from restricting clone-depth, the user will have full access
to change history, etc., as the objects are downloaded on demand.

Change-Id: I60326744875eac16521a007bd7d5481112a98749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
2019-07-16 00:23:16 +00:00
Mike Frysinger
ab85fe7c53 use print() instead of sys.stdout.write()
We're relying on sys.stdout.write() to flush its buffer which isn't
guaranteed, and is not the case in Python 3.  Change to use print()
everywhere to be standard, and utilize the end= keyword to get the
EOL semantics we need.

We can't use print's flush= keyword as that's only in Python 3.
Leave behind a TODO to clean it up when we can drop Python 2.

Bug: https://crbug.com/gerrit/10418
Change-Id: I562128c7f1e6d154f4a6ecdf33a70fa2811dc2af
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/230392
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
2019-07-11 06:26:40 +00:00
Mike Frysinger
ca540aed19 git_command: drop custom version helper
Since ParseGitVersion can call `git --version` automatically, we don't
need this duplicate version() helper anymore.  The only other user is
the `repo version` code, so convert that to version_tuple().full.

Bug: https://crbug.com/gerrit/11144
Change-Id: I9d77822fc39f4ba28884d9183359169cabf5f17d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231055
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-07-11 01:30:18 +00:00
Solomon Kinard
490e16385d Remove double forall from "repo help forall" output
%prog represents the full subcommand ("repo" + subcommand name), not a
Windows-style environment variable for "repo". The current help output
shows

 repo forall% forall ...

Correct the variable usage so it shows "repo forall ..." instead.

Change-Id: I1fea55572428cc922ddf24ace1168a3d8f82dad0
2019-07-08 22:42:38 +00:00
Mike Frysinger
0c0e934b69 sync: use integer division with job counts
Neither of the fields here expect floats so make sure we use integer
division when calculating things.

Bug: https://crbug.com/gerrit/10418
Change-Id: Ibda068b16a7bba7ff3efba442c4bbff4415caa6e
2019-06-14 14:47:01 +00:00
Mike Frysinger
9e71842fbf status: import print_function
This module uses print() so make sure we import the print function.
It doesn't really impact the current code due to the simple way it
is calling print, but we should be sane to avoid future issues.

Change-Id: I0b15344678c1dcb71207faa333c239b3fced1d62
2019-06-14 14:13:23 +00:00
Mike Frysinger
f601376e13 set default file encoding to utf-8
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.

Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
2019-06-13 14:30:52 +00:00
Mike Frysinger
31067c0ac5 tweak raise/dict syntax for Python 3 compat
Use the `raise` statement directly.

Switch to using .items() instead of .iteritems().  Python 3 doesn't
have .iteritems() as .items() is a generator, and these are small
enough that the Python 2 overhead should be negligible.

We have to run .keys() through list() in a few places as Python 3
uses a generator and we sometimes want to iterate more than once.
That's why we don't change all .keys() or .items() calls -- most
are in places where generators are fine.

Bug: https://crbug.com/gerrit/10418
Change-Id: I469899d9b77ffd77ccabb831bc4b217407fefe6f
2019-06-13 13:39:25 +00:00
Basil Gello
c745350ab9 diffmanifests: honor user-supplied manifest paths
The current implementation ignores the user-specified paths to
manifests.  if the "repo diffmanifests" is invoked with absolute
file paths for one or both manifests, the command fails with message:

fatal: duplicate path ... in /tmp/manifest-old.xml

Also the current implementation fails to expand the absolute path to
manifest files if "repo diffmanifests" is invoked with relative
paths, i.e "repo diffmanifests manifest-old.xml manifest-new.xml".

fatal: manifest manifest-old.xml not found

This commit fixes the first issue by disabling the local manifest
discovery for diffmanifests command, and the second issue by
expanding paths to manifests within "diffmanifests" sub-command.

Test: repo manifest --revision-as-HEAD -o /tmp/manifest-old.xml
      repo sync
      repo manifest --revision-as-HEAD -o /tmp/manifest-new.xml
      repo diffmanifests /tmp/manifest-old.xml /tmp/manifest-new.xml
Change-Id: Ia125d769bfbea75adb9aba81abbd8c636f2168d4
Signed-off-by: Vasyl Gello <vasek.gello@gmail.com>
2019-06-06 07:36:10 +00:00
Nasser Grainawi
d92464e8ef Honor --depth during repo init
If a user is asking for a shallow clone of the repos, they probably
expect a shallow clone of the manifest repo too. For very large
manifest repos, this can be a huge space and time savings. For one real-world
repo, a 'repo init --no-tags --current-branch' used 350MB of disk space and
took 7 minutes. Adding --depth 1 and this change reduced it to 10MB and 2.5
minutes.

Change-Id: I6fa662e174e623ede8861efc862ce26d65d4958d
2019-05-21 10:47:21 -06:00
Sebastian Schuberth
266f74c888 info: Use the non-formatting printer for headtext
If "repo init" was run in a path containing "%", "repo info" would fail
printing the path with

    File ".repo/repo/color.py", line 173, in f
      return fmt % args
    TypeError: not enough arguments for format string

as the "%" in the path name is interpreted as the start of a formatting
specifier. Avoid that by using the non-formatting printer for headtext
which does not require any formatting so there is no need to try to
expand "%" sequences.

Change-Id: Ie193b912191fe7cdabdce5c97bb100f0714f6e76
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
2019-04-17 19:46:05 +02:00
Kuang-che Wu
0d9b16d1d8 sync: deleted unused repos in reversed order (children before parent)
Bug: chromium:950002
Test: repo sync chromeos using release manifest file
Change-Id: I613df6a1973eb36acd806a703e72f5172554bcc7
2019-04-06 00:49:47 +08:00
Oleksii Okolielov
d3c0f5914f sync: Add option '--force-remove-dirty'
Forcefully remove dirty projects if option '--force-remove-dirty' is given.
The '--force-remove-dirty' option can be used to remove previously used
projects with uncommitted changes. WARNING: This may cause data to be lost
since uncommitted changes may be removed with projects that no longer exist
in the manifest.

Change-Id: I844a6e943ded522fdc7b1b942c0a1269768054bc
2019-03-11 13:06:49 -04:00
Ereth McKnight-MacNeil
12ee5446e9 init: Remove -c short option for --current-branch
This option conflicts with the gitc-init -c short option.

Bug: https://crbug.com/gerrit/10200
Change-Id: I06f37564429ca0bd4c0bbea6066daae4f663c838
2018-12-20 19:55:02 +00:00
Vadim Bendebury
bd8f658823 Add option for git-repo to support 'silent' uploads
When --ne/--no-emails is added to 'repo upload' command line, gerrit
server will not generate notification emails.

project.py:Project.UploadForReview method is modified to accept a
string recognizable by gerrit to indicate different sets of destination
email addressees, but the upload command line allows only one option -
disable sending emails completely.

Default repo upload behavior is not being changed.

TEST=tried in the Chrome OS repo, observed that patches uploaded with
     --ne or --no-emails indeed do not trigger any emails, while
     patches uploaded without these command line options still trigger
     email notifications.

Change-Id: I0301edec984907aedac277d883bd0e6d3099aedc
2018-11-05 14:01:05 -08:00
David Pursehouse
36391bf5ca Merge "init: --dissociate option to copy objects borrowed with --reference" 2018-10-28 23:30:50 +00:00
Renaud Paquay
bed8b62345 Add support for long paths
* Add more file i/o wrappers in platform_utils to allow using
  long paths (length > MAX_PATH) on Windows.

* Paths using the long path syntax ("\\?\" prefix) should never
  escape the platform_utils API surface area, so that this
  specific syntax is not visible to the rest of the repo code base.

* Forward many calls from os.xxx to platform_utils.xxx in various place
  to ensure long paths support, specifically when repo decides to delete
  obsolete directories.

* There are more places that need to be converted to support long paths,
  this commit is an initial effort to unblock a few common use cases.

* Also, fix remove function to handle directory symlinks

Change-Id: If82ccc408e516e96ff7260be25f8fd2fe3f9571a
2018-10-22 08:16:35 -07:00
Nikolai Merinov
09f0abb0ef init: --dissociate option to copy objects borrowed with --reference
"repo init --reference" has two purposes: to decrease bandwidth used
at clone time, and to decrease disk usage afterward, by using the
reference repositories as an alternate store of objects even after
the clone. The downside is that it makes the borrowing repositories
dependent on the reference repositories, so it is easy to end up
with missing objects by mistake after a cleanup operation like "git
gc".

To prevent that, v2.3.0-rc0~30^2 (clone: --dissociate option to mark
that reference is only temporary, 2014-10-14), "git clone" gained a
--dissociate option that makes --reference reuse objects from the
reference repository at clone time but copy them over instead of
using the reference as an alternate. This is more straightforward to
use than plain --reference, at the cost of higher disk usage.

Introduce a --dissociate to "repo init" that brings the same benefits
to repo. The option is simply passed on to "git clone".

Change-Id: Ib50a549eb71e0a2b3e234aea57537923962a80d4
2018-10-19 23:51:23 +05:00
Jonathan Nieder
b3133a3164 Merge "update markdown/help header format" 2018-10-10 05:54:59 +00:00
Mike Frysinger
3b24e7b557 update homepage URIs
Change-Id: I482a72bf296978625b1e82ef580b0e0d4d57ff25
2018-10-10 01:35:58 -04:00
Mike Frysinger
b8f7bb04d0 update markdown/help header format
Since gitiles recommends using # headers over ---/=== underlines,
change the manifest-format.md over and all our help texts.

Change-Id: I96391d41fba769e9f26870d497cf7cf01c8d8ab3
2018-10-10 01:28:43 -04:00
Mike Frysinger
3891b7519d manifest-format: convert to markdown
The gitiles system doesn't render .txt files, so convert this to .md
for better display online.

Change-Id: Ie12e46daf008dd8c97ae2ffd21fb68bd948fe625
2018-10-05 19:32:51 -04:00
David Pursehouse
65b0ba5aa0 Remove unused pylint suppressions
pylint is not used since bb5b1a0. The pyflakes cleanup mentioned in that
commit has not been done, but given that this project is no longer being
actively developed I don't think it's worth spending time doing it.

Leaving the pylint suppressions causes confusion because it leads people
to think that we are still using pylint.

Change-Id: If7d9f280a0f408c780f15915ffdb80579ae21f69
2018-07-24 22:20:08 +08:00
Akshay Verma
0f2e45a3a6 Pass refs to ls-remote
This will fix the issue of parsing large output locally

Change-Id: I9a5cf1238147a02c92a3fca53eab9bd57f9d16b4
2018-03-24 13:00:08 +05:30
Akshay Verma
cf7c0834cf Download latest patch when no patch is specified
When someone does "repo download -c <project> <change>"
without specifying a patch number, by default patch 1 is
downloaded. An alternative is to look for the latest patch
and download the same when no explicit patch is given.
This commit does the same by identifying the latest patch
using "git ls-remote".

Change-Id: Ia5fa7364415f53a3d9436df4643e38f3c90ded58
2018-03-17 16:29:23 +05:30
Samuel Holland
5f0e57d2ca init: Remove string concat in no-op os.path.join
This also fixes a line length warning.

Change-Id: I9c1ab65f83a35581dd657a707c7bc3c69db2b1dc
2018-01-22 11:00:24 -06:00
Samuel Holland
baa0009355 Support relative paths in --reference
Put the correctly-expanded relative paths in objects/info/alternates.
From gitrepository-layout(5), this path should be "relative to the
object database, not to the repository".

Change-Id: I7b2027ae23cf7d367b80f5a187603c4cbacdb2de
2018-01-22 10:57:29 -06:00
Masaya Suzuki
305a2d029f Support --push-option in upload subcommand
Change-Id: I44836f8c66ded5a96cbf5431912e027e681f6529
2017-11-13 15:48:49 -08:00
David Pursehouse
f46902a800 forall: Clarify expansion of REPO_ environment values with -c
If a user executes:

  repo forall -c echo $REPO_PROJECT

then $REPO_NAME is expanded by the user's shell first, and passed
as $1 to the shell that executes echo. This will either result in
no output, or output of whatever REPO_NAME is set to in the user's
shell. Either way, this is an unexpected result.

The correct way to do it is:

  repo forall -c 'echo $REPO_PROJECT'

such that $REPO_NAME is passed in to the shell literally, and then
expanded to the value set in the environment that was passed to
the shell.

Update the documentation to make this clearer.

Change-Id: I713caee914172ad8d8f0fafacd27026502436f0d
2017-10-31 13:07:55 +09:00
Renaud Paquay
010fed7711 Replace all os.remove calls
os.remove raises an exception when deleting read-only files on
Windows. Replace all calls with calls to platform_utils.remove,
which deals with deals with that issue.

Change-Id: I4dc9e0c9a36b4238880520c69f5075eca40f3e66
2017-08-31 13:49:36 -07:00
Renaud Paquay
227ad2ef42 Implement islink, readlink and realpath using Win32 api
Change-Id: I18452cbb32d24db73601ad10485dbe6bb278731c
2017-08-31 13:49:01 -07:00
David Pursehouse
2c57d619bc Merge "Add option '--no-cert-checks' for 'upload' sub command." 2017-08-30 15:11:10 +00:00
David Pursehouse
d1ebc89a08 Merge changes from topic "windows-support"
* changes:
  Port os.rename calls to work on Windows
  Workaround shutil.rmtree limitation on Windows
  Add support for creating symbolic links on Windows
  Make "git command" and "forall" work on Windows
2017-08-30 10:24:03 +00:00
Zac Livingston
9ead97bb51 When starting a branch, do not use a tag or change value for branch.merge
When starting a branch, branch.merge is set to project revision unless
the revision is a SHA1. In that case, branch.merge is set to dest_branch
if defined or manifest default revision otherwise. This special handling
allows repo upload to work when the project revision is a SHA1.

Extend the special handling to also happen when the project revision
is a tag value or a change value so that repo upload will work in those
case as well.

Change-Id: Iff81ece40e770cd02535e80dcb023564d42dcf47
2017-08-25 09:10:29 +09:00
Łukasz Gardoń
bed59cec5e Add option '--no-cert-checks' for 'upload' sub command.
This option allow to bypass verification ssl certification while
establishing connection with Gerrit to upload review.

Change-Id: If2e15f5a273c18a700eb5093ca8a4d5a4cbf80cd
2017-08-23 14:06:14 +02:00
Jonathan Nieder
c94d6eb902 Revert "Migrate git-repo to create private changes rather than drafts"
This reverts commit d88f53e2b9. I merged
it too hastily without paying enough attention to compatibility with
released Gerrit versions.

Change-Id: I4028d4737df1255f11e217da183a19a010597d5b
2017-08-08 18:34:53 +00:00
Changcheng Xiao
d88f53e2b9 Migrate git-repo to create private changes rather than drafts
Considering that some users might expect changes created with
'-d' option are not public. Private changes may be a better
choice here than work-in-progress changes.

Change-Id: I46a8fb9ae38beb41cf96d6abe82bea6db2439669
2017-08-07 15:08:18 +02:00
Changcheng Xiao
87984c6db4 Add options for git-repo to support private and wip changes
This change adds options for git-repo tool to support private
changes and work-in-progress changes.

Change-Id: I343491f5949f06f1580d53f9cc0dee2dca09130f
2017-08-07 15:02:39 +02:00
David Pursehouse
ffc1401327 Merge "download: try to choose . as default project if none" 2017-08-02 07:19:45 +00:00
David Pursehouse
8a6eeed7f5 Merge "Always print percentage when syncing quietly" 2017-08-02 07:01:08 +00:00
Tim Schumacher
7be072efa6 Always print percentage when syncing quietly
Change-Id: I574396e63520781067ed1e991c41caf7640e5731
2017-07-15 16:44:55 +00:00
Nicolas Cornu
7482a96443 download: try to choose . as default project if none
Change-Id: I28b5e3be5f3c9a4c077af87d6a3e0cc3b96a1b9d
2017-07-12 10:15:06 +02:00
David Pursehouse
3bcd30545e Fix "list comprehension redefines 'x'" warnings from pyflakes
$ git ls-files | grep py$ | xargs pyflakes
  subcmds/stage.py:101: list comprehension redefines 'p' from line 63
  subcmds/sync.py:784: list comprehension redefines 'p' from line 664
  subcmds/upload.py:467: list comprehension redefines 'avail' from line 454

Change-Id: Ia65d1a72ed185ab3357e1a91ed4450c719e75a7c
2017-07-10 23:26:04 +00:00
Martin Kelly
224a31a765 init: add missing submodule arg
The submodule argument to Sync_LocalHalf was missing in
MetaBranchSwitch, causing submodules not to get synced when the
-b/--manifest-branch argument to init is used.

Change-Id: Ie86d271abac2020725770be36ead83be3326e64b
Signed-off-by: Martin Kelly <mkelly@xevo.com>
2017-07-10 14:50:52 -07:00
Nicolas Cornu
b54343d9fd Tell the user if it will upload a draft
Change-Id: Ie004ec9d61603f3f618c47597947b82c59f2839c
2017-07-10 13:20:01 +00:00
David Pursehouse
259f16520a Merge "Add a newline after "Fetching projects" progress output" 2017-06-28 04:24:03 +00:00
Nicolas Cornu
8419ab22d6 sync: Continue job if some fetchs failed but force-broken is set
With --force-broken it continue to fetch other projects but nothing
is added in directory because it abort some lines later.

Change-Id: I32c4a4619b3028893dc4f98e8d4e5bc5c09adb27
2017-06-16 12:23:26 +02:00
Tim Schumacher
913327f10c
Add a newline after "Fetching projects" progress output
Output before change:

    Fetching project platform/packages/providers/UserDictionaryProvider
    Fetching projects:  66% (773/1171)  Fetching project platform/external/regex-re2
    Fetching project device/generic/mini-emulator-x86_64

Output after change:

    Fetching project platform/packages/providers/UserDictionaryProvider
    Fetching projects:  66% (773/1171)
    Fetching project platform/external/regex-re2
    Fetching project device/generic/mini-emulator-x86_64

Change-Id: I4da84da58316c69294e4da2792f83885dc942701
2017-06-13 13:03:39 +02:00
Renaud Paquay
a65adf74f9 Workaround shutil.rmtree limitation on Windows
By default, shutil.rmtree raises an exception when deleting readonly
files on Windows.

Replace all shutil.rmtree with platform_utils.rmtree, which adds an
error handler to make files read-write when they can't be deleted.

Change-Id: I9cfea9a7b3703fb16a82cf69331540c2c179ed53
2017-05-29 19:32:31 +09:00
Renaud Paquay
2e70291162 Make "git command" and "forall" work on Windows
Python on Windows does not support non blocking file operations.
To workaround this issue, we instead use Threads and a Queue to
simulate non-blocking calls. This is happens only when running
with the native Windows version of Python, meaning Linux and Cygwin
are not affected by this change.

Change-Id: I4ce23827b096c5138f67a85c721f58a12279bb6f
2017-05-29 19:29:30 +09:00
David Pursehouse
a24671f661 Merge "init: allow relative path on --reference argument" 2017-05-29 04:54:12 +00:00
David Riley
e0684addee sync: Add support to dump a JSON event log of all sync events.
Change-Id: Id4852968ac1b2bf0093007cf2e5ca951ddab8b3b
2017-05-29 13:39:54 +09:00
YOUNG HO CHA
c79d3b8fd1 init: allow relative path on --reference argument
Change-Id: I41d6be6bc035fdddb5a27c072994439986d58d58
Signed-off-by: YOUNG HO CHA <ganadist@gmail.com>
2017-05-28 00:33:25 +00:00
David Pursehouse
fbcbcabe98 Merge "init: add --submodules to sync manifest submodules" 2017-05-27 04:24:58 +00:00
David Pursehouse
666debc518 gitc_delete: Remove unused imports
Change-Id: I672189ba99e18dca3956e2396c921d1ef0ca2ddd
2017-05-26 21:53:34 +09:00
David Pursehouse
c354a9b922 abandon: fix usage of undefined variable
As reported by pyflakes:

  subcmds/abandon.py:84: undefined name 'p'

The name of the variable should be 'proj'.

Change-Id: Ic09eb92e8db6b510e99efce010bd0bb094d7cbfe
2017-05-26 21:52:23 +09:00
Martin Kelly
e4e94d26ae init: add --submodules to sync manifest submodules
repo sync can sync submodules via the --fetch-submodules option.
However, if the manifest repo has submodules, those will not be synced.
Having submodules in the manifest repo -- while not commonly done -- can
be useful for inheriting a manifest from another project using <include>
and layering changes on top of it.  In this way, you can avoid having to
deal with merge conflicts between your own manifests and the other
project's manifests (for example, if you're managing an Android fork).

Add a --submodule option to init that automatically syncs the submodules
in the manifest repo whenever the manifest repo changes.

Change-Id: I45d34f04517774c1462d7f233f482d1d81a332a8
Signed-off-by: Martin Kelly <mkelly@xevo.com>
2017-05-23 16:51:31 -07:00
Marc Herbert
ffb4b89099 sync.py: report the remote URL on fatal git remote errors
repo can be configured to download from any number of remote git repos.
However when one fails repo doesn't report which one. Example:
Fatal: remote error: Daily ls-remote rate limit exceeded for IP xx.xx.xx.xx

TEST=repo init -q -u https://chromium.googlesource.com/chromiumos/manifest.git
  # Apply patch in ./.repo/repo/
  # Simulate a git remote error:
  sed -i -e 's#chromiumos/docs#chromiumos/XXdocs#' .repo/manifests/full.xml
  repo sync --quiet --force-sync docs
  # error message now shows the remote URL

Optional test tip: reduce the time.sleep(random(...)) in ./.repo/repo/project.py

Change-Id: I4509383b6a43a8e66064778e8ed612d8a735c8b6
2017-04-04 22:10:34 -07:00
Naseer Ahmed
f4dda9a1be init: Add no-tags and current branch options
This avoids fetching tags and branches for huge manifests

Change-Id: I19c9724d75364440b881b297d42b906f541f73ff
2016-12-01 19:03:41 -05:00
David Pursehouse
ae81c964b6 Merge "implement optional '--all' in the abandon command" 2016-10-28 07:52:31 +00:00
Andrew Wheeler
4d5bb68d58 status: add -q/--quiet option
The --quiet option reduces the output to just
a list of projects with modified workspaces (and
orphans if -o is specified)

A common use case is when performing a full-workspace
merge.  The integrator will kick-off a merge via:

    repo forall -c git merge <some tag>

And then produce a short list of conflicted projects via:

    repo status -q

The integrator can then iteratively fix and clean up all conficted
components.  The merge is complete when:

    repo status -q

    returns no output.

Change-Id: Ibbba8713eac35befd8287c95948874e23fd5c7e2
2016-10-17 15:24:09 -05:00
Kyunam.jo
2e14792a94 implement optional '--all' in the abandon command
when you want to delete all local branches, you should be find
all branches' name, and type them behind 'repo abandon' command.

Usage:
    repo abandon --all [<project>...]

Change-Id: I4d391f37fb9d89b8095488c585468eafc1a35f31
2016-10-17 02:29:42 +00:00
David Pursehouse
82f67987a3 Merge "sync: Fix semaphore release bug that causes thread 'leaks'" 2016-10-17 01:10:16 +00:00
Kyunam.jo
699bcd40be Removed duplication code in abandon.py
code about getting argument is duplicated.
so this line is removed

Change-Id: Id321b999c7dacdb403cd986cbf35f8db62efc157
2016-10-12 10:02:30 +09:00
Andrew Wheeler
7f1ccfbb7b sync: Fix semaphore release bug that causes thread 'leaks'
When repo syncs a manifest that utilizes multiple branches
in the same project, then the sync will use an extra
thread for each "duplicate".  For example, if
the manifest includes the project "foo" and "bar"
twice, then "repo sync -jN" will fetch with N+2 threads.

This is caused by _FetchHelper() releasing the thread semaphore
object each time it's called, even though _FetchProjectList()
may call this function multiple times within the scope of a
single thread.

Fix by moving the thread semaphore release to
_FetchProjectList(), which is only called once per thread
instance.

Change-Id: I1da78b145e09524d40457db5ca5c37d315432bd8
2016-10-11 14:10:34 -05:00
Dan Willemsen
eceeb1b1f5 Support broken symlinks when cleaning obsolete paths
When there's a symlink to a directory, os.walk still lists the symlink
in dirs, even if it isn't configured to follow symlinks. This will fail
the listdirs check if the symlink is broken (either before or during the
cleanup). So instead, check for directory symlinks and remove them using
os.remove.

Bug: Issue 231
Change-Id: I0ec45a26be566613a4a39bf694a3d9c6328481c2
2016-09-27 03:05:11 +00:00
Dan Willemsen
4350791e0d On project cleanup, don't remove nested projects
When there are nested projects in a manifest, like on AOSP right now:

<project path="build" name="platform/build" />
<project path="build/blueprint" name="platform/build/blueprint" />
<project path="build/kati" name="platform/build/kati" />
<project path="build/soong" name="platform/build/soong" />

And the top "build" project is removed (or renamed to remove the
nesting), repo just wipes away everything under build/ and re-creates
the projects that are still there. But it only checks to see if the
build/ project is dirty, so if there are dirty files in a nested
project, they'll just be blown away, and a fresh worktree checked out.

Instead, behave similarly to how `git clean -dxf` behaves and preserve
any subdirectories that have git repositories in them. This isn't as
strict as git -- it does not check to see if the '.git' entry is a
readable gitdir, just whether an entry named '.git' exists.

If it encounters any errors removing files, we'll print them all out to
stderr and tell the user that we were unable to clean up the obsolete
project, that they should clean it up manually, then sync again.

Change-Id: I2f6a7dd205a8e0b7590ca5369e9b0ba21d5a6f77
2016-09-20 17:16:12 -07:00
David Pursehouse
01b7d758d5 Merge "When syncing a project with a shared object store, disable automatic pruning." 2016-09-14 20:48:24 +00:00
David Pursehouse
d9de945d8a Merge "upload: short circuit when nothing is pending" 2016-09-14 05:20:36 +00:00
Gabe Black
2ff302929c When syncing a project with a shared object store, disable automatic pruning.
The shared object stores confuse git and make it throw away objects which are
still in use. We'll avoid that problem by disabling automatic pruning on those
projects, but there's nothing preventing a user from changing the config back
or pruning a repository manually.

BUG=chromium:375945
TEST=Ran repo sync on fresh ChromeOS checkout, starting with a branch of repo
with this change. Verified that the kernel projects and no others were
identified as having shared object stores, and that repo successfully disabled
automatic pruning in their configs. Re-enabled pruning and ran repo sync just
on one of the kernel directories. Verified that pruning was re-disabled as a
result.

Change-Id: I728ed5b06f0087aeb5a23ba8f5410a7cd10af5b0
2016-09-14 00:19:44 -04:00
Vadim Bendebury
e5c0ea0a95 Consider local project to be default for 'repo start'
The requirement to explicitly specify the local project when starting
a new repo branch is somewhat counter intuitive.

This patch uses the current directory's git tree as the default
project.

Tested by running

  'repo start <name>'

observed that the result is the same as if running

  'repo start <name> .'

Change-Id: If106caa801b4cd5ba70dbe8354a227d59f100aa3
2016-09-14 00:17:45 -04:00
Mike Frysinger
163a3be18b upload: short circuit when nothing is pending
When nothing is pending, most of this code is already short-circuited.
Hoist the single check up to make this more obvious/slightly faster.

Change-Id: Iec3a7e08eacd23a7c5f964900d5776bf5252c804
2016-09-14 00:16:37 -04:00
David Pursehouse
6a0a3648f1 Merge "init: Add --no-clone-bundle option" 2016-08-17 04:57:28 +00:00
David Pursehouse
6118faa118 Merge "init: Respect --quiet option when synching manifest repository" 2016-08-17 04:04:14 +00:00