Commit Graph

1497 Commits

Author SHA1 Message Date
Mike Frysinger
fb21d6ab64 sync: use subprocess.run to verify tags
The code is a bit simpler & easier to reason about.

Change-Id: I149729c7d01434b08b58cc9715dcf0f0d11201c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297022
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-16 16:26:41 +00:00
Raman Tenneti
21dce3d8b3 init: added --use-superproject option to clone superproject.
Added --no-use-superproject to repo and init.py to disable use of
manifest superprojects.

Replaced the term "sha" with "commit id".

Added _GetBranch method to Superproject object.

Moved shared code between init and sync into SyncSuperproject function.
This function either does git clone or git fetch. If git fetch fails
it does git clone.

Changed Superproject constructor to accept manifest, repodir and branch
to avoid passing them to multiple functions as argument.

Changed functions that were raising exceptions to return either True
or False.

Saved the --use-superproject option in config as repo.superproject.
Updated internal-fs-layout.md document.

Updated the tests to work with the new API changes in Superproject.

Performance for the first time sync has improved from 20 minutes to
around 15 minutes.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL.

$ repo init took around 20 seconds longer because of cloning of superproject.

$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real	0m35.919s
user	0m21.947s
sys	0m8.977s

First run
$ time repo sync --use-superproject
...
real	16m41.982s
user	100m6.916s
sys	19m18.753s

No difference in repo sync time after the first run.

Bug: [google internal] b/179090734
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707

Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-11 18:59:29 +00:00
Mike Frysinger
e3315bb49a diffmanifests/sync: simplify repodir lookup
We have access to repodir on the command object itself, so we don't
need to pull it indirectly out of the manifest object.

Change-Id: I8688fb1c84979825efa966dc787e78c6f7ba3823
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296542
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-11 02:38:53 +00:00
Mike Frysinger
38867fb6d3 git_config: add SetBoolean helper
A little sugar simplifies the code a bit.

Change-Id: Ie2b8a965faa9f9ca05c7be479d03e8e073cd816d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296522
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-11 01:48:12 +00:00
Raman Tenneti
ce64e3d47b superproject: Pass branch to git ls-tree.
Tested the code with the following commands.

$ ./run_tests -v

Bug: [google internal] b/179702819
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I7d2b609ac2f927c94701757aa1502ba236afe7c0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296342
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-02-08 23:43:34 +00:00
Raman Tenneti
8d43dea6ea sync: pass --bare option when doing git clone of superproject.
Changed "git pull" to "git fetch" as we are using --bare option. Used the
following command to fetch:
  git fetch origin +refs/heads/*:refs/heads/* --prune

Pass --branch argument to Superproject's UpdateProjectsRevisionId function.

Returned False/None when directories don't exist instead of raise
GitError exception from _Fetch and _LsTree functions. The caller of Fetch
does Clone if Fetch fails.

Tested the code with the following commands.

$ ./run_tests -v

Tested the init and sync code by copying all the repo changes into my Android
AOSP checkout and running repo sync with --use-superproject option.

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-02-08 17:34:55 +00:00
Raman Tenneti
1fd7bc2438 sync: superproject performance changes.
After updating all project’s revsionIds with the SHAs from superproject,
write the updated manifest into superproject_override.xml file. Reload
that file for future Reloads. This file is created in exp-superproject
directory.

Moved most of the code that is superproject specific into
git_superproject.py and wrote test code.

If git pull fails, did a git clone of the superproject.

We saw performance gains for consecutive repo sync's. The time to sync
went down from around 120 secs to 40 secs when repo sync is executed
consecutively.

Tested the code with the following commands.

$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.

First run
$ time repo sync --use-superproject
...
real	21m3.745s
user	97m59.380s
sys	19m11.286s

After two consecutive sync runs
$ time repo sync -c -j8 --use-superproject
real	0m39.626s
user	0m29.937s
sys	0m38.155s

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>

Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-07 22:25:38 +00:00
Raman Tenneti
b5c5a5e068 manifest: set revisionId as revision attribute it it is not being set in ToXml.
As we were testing superproject setting revisionId attribute to SHA and
reloading the manifest, we found out revisionId attribute is not being
saved. Made the change to save the revisionId if it is not being saved.

Tested the code with the following commands.

$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I95fdf655b19648ad3e9aba10b9bed8bb9439deb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296182
Reviewed-by: Ian Kasprzak <iankaz@google.com>
2021-02-07 17:13:35 +00:00
Ian Kasprzak
0286e31ec7 Update _CheckForImmutableRevision to use git rev-list
_CheckForImmutableRevision is used to see if repo can
skip fetching a project, but 'git rev-parse' with partial
clone does a data fetch to accomplish this.

Changed to use: 'git rev-list -1 --missing=allow-any <SHA>^0' which
checks the local ref without fetching from the server first.

Bug: [google internal] b/179477822

Testing:
- Unit tests
- Verified init/sync working on aosp-master
- Verified wwith a pinned manifest that local ref check works (no fetch)

Change-Id: If327b893c6658421f41df1f58c337f53b4c60ce6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296142
Reviewed-by: Dan Willemsen <dwillemsen@google.com>
Tested-by: Ian Kasprzak <iankaz@google.com>
2021-02-05 22:00:31 +00:00
Raman Tenneti
ef267722f8 sync: Added --filter=blob:none for git clone of superproject.
+ This is without --depth option. This is done for reachability.
  Server doesn't know what you know about in the history so it always
  sends you the whole thing Which is very slow.

  If we have the full history it can send you incremental update history
  which is very small and fast.

Tested the code with the following commands.

$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.

.../WORKING_DIRECTORY$ repo sync --use-superproject

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I239de6d8f1c2ed6b4c69e7a78b8aa95338fa838c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/295362
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-02-02 16:18:06 +00:00
Raman Tenneti
7caa3658b2 sync: Do a git pull with --use-superproject if superproject tree already exists.
Tested the code with the following commands.

$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.

.../WORKING_DIRECTORY$ repo sync --use-superproject

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I7e4b1e51ca1d18b836a5fa8d139a0765262ba500
2021-02-01 12:24:51 -08:00
Raman Tenneti
9e7875315f sync: Added --filter=blob:none (and no-depth) wduring git clone of superproject.
Tested the code with the following commands.

$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.

.../WORKING_DIRECTORY$ repo sync --use-superproject

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Ieea31445ca89ba1d217e779ec7a7a2ebe81ac518
2021-02-01 20:08:00 +00:00
Gaurav Pathak
db3128f2ec git_command.py: Handle unicode decode error
repo diffmanifests saves git commit messages in buf and uses default
utf-8 decoding, in some scenarios git commit message can itself contain
a non UTF-8 character due to a typo or incorrect i18n.commitEncoding.

e.g.
d354d9afe923 [PATCH] fbcon: don\xb4t call set_par() in fbcon_init() if vc_mode == KD_GRAPHICS

Convert the buf containing git commits to string if decoding to utf-8
encounters an error.

Signed-off-by: Gaurav Pathak <gaurav.pathak@pantacor.com>
Change-Id: If818562f0faaa5062c765fbea11dc0e1c86a24d7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294742
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-28 17:38:24 +00:00
Anders Björklund
2a2da80ba6 sync: Disable info about disabling pruning when quiet
If you have a lot of shared projects, it spams.

Bug: https://crbug.com/gerrit/13961
Change-Id: If3f5baef65930830af9a2cd01a1b593dd518ab09
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294049
Tested-by: Anders Björklund <anders.bjorklund.2@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-22 11:43:13 +00:00
Raman Tenneti
6a872c9dae sync: Added --use-superproject option and support for superproject.
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,

.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-21 19:41:52 +00:00
Mike Frysinger
df6c506a8a launcher: bump version for new release
Change-Id: I8a39630d482fc389cf497399102f795d7e576ff9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294122
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-21 16:17:19 +00:00
Ian Kasprzak
febe73ff16 Update "evt" field to be logged as a string type.
Testing:
- Unit tests
- Verified git trace log has "evt": "2" (vs "evt": 2 previously)

Bug: https://crbug.com/gerrit/13966
Change-Id: I2e0c98dda0cccdd5cb6328105c11b93cd42676eb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294123
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Ian Kasprzak <iankaz@google.com>
2021-01-19 20:35:56 +00:00
Mike Frysinger
e5670c8812 launcher: add a requirements framework to declare version dependencies
Currently we don't have a way for the checked out repo version to
declare the version of tools it needs before we start running it.
For somethings, like git, it's not a big deal as it can handle all
the asserts itself.  But for things like Python, it's impossible
to reliably check before executing.

We're in this state now:
- we've been allowing Python 3.4, so the launcher accepts it
- the repo codebase starts using Python 3.6 features
- launcher tries to import us but hits syntax errors
- user is left confused and assuming new repo is broken because
  they're seeing syntax errors

This scenario is playing out with old launchers that still accept
Python 2, and will continue to play out as time goes on and we want
to require newer versions of Python 3.

Lets create a JSON file to declare all these system requirements.
That file format is extremely stable, so loading & parsing from
even ancient versions of Python shouldn't be a problem.  Then the
launcher can read these settings and check the system state before
attempting to execute any code.  If the tools are too old, it can
clearly diagnose & display information to the user as to the real
problem (and not emit tracebacks or syntax errors).

We have a couple of different tool version checks already (git,
python, ssh) and can harmonize them in a single place.

This also allows us to assert a reverse dependency if the need
ever comes up: force the user to upgrade their `repo` launcher
before we'll let them run us.  Even though the launcher warns
whenever a newer release is available, some users seem to ignore
that, or they don't use repo that often (on the scale of years),
and their upgrade jump is so dramatic that they fall back into
the syntax error pit.

Hopefully by the end of the year we can assume enough people
have upgraded their launcher such that we can delete all of the
duplicate version checks in the codebase.  But until then, we'll
keep them to maintain coverage.

Change-Id: I5c12bbffdfd0a8ce978f39aa7f4674026fe9f4f8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293003
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-19 16:48:21 +00:00
Raman Tenneti
48b2d10d8f manifest_xml: - Added doc and testing of unknown tags/elements.
Added this test to verify that older versions of repo can handle
"<superproject" element. Tested by adding "<iankaz" unknown element.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I858d56f38cefcfcd14474efdd631a5a940c3ce47
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293482
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-12 21:49:13 +00:00
Mike Frysinger
0588f3dc52 version: add remote tracking information
This tells us what --repo-rev the user is using.

Bug: https://crbug.com/1164415
Change-Id: Idb6c48e6ca5a4783c529717e6be38266bf7038b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293143
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-08 20:41:27 +00:00
Raman Tenneti
1bb4fb222d manifest_xml: initial support for <superproject>
At most one superproject may be specified. It will be used
to specify the URL of superproject.

It would have 3 attributes: remote, name, and default.
Only "name" is required while the others have reasonable defaults.

<remote name="superproject-url" review="<url>" />
<superproject remote="superproject-url" name="platform/superproject"/>

TODO: This CL only implements the parsing logic and further work
will be in followup CLs.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I5b4bba02c8b59601c754cf6b5e4d07a1e16ce167
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292982
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-08 19:49:52 +00:00
Mike Frysinger
b64bec6acc launcher: bump version for new release
Change-Id: Ie0abee81e86046f412b42f08100041cfd3689c4a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-07 22:44:53 +00:00
Ian Kasprzak
343d585ff9 Fix bug in git trace2 event Write() function when no config present.
See https://bugs.chromium.org/p/gerrit/issues/detail?id=13706#c9

Added additional unit tests for Write() for additional test coverage.

Testing:
- Unit tests
- Verified repo works with:
  - Valid trace2.eventtarget
  - Invalid trace2.eventtarget

Bug: https://crbug.com/gerrit/13706
Tested-by: Ian Kasprzak <iankaz@google.com>
Change-Id: I6b027cb2399bd03e453a132ad82e022a1f48476e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292762
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-07 14:31:51 +00:00
Mike Frysinger
acf63b2892 drop pyversion & is_python3 checking
We're committed to Python 3 at this point, so purge all the
is_python3 related dynamic checks.

Bug: https://crbug.com/gerrit/10418
Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:58 +00:00
Mike Frysinger
784ccfc040 strip python2-only coding:utf-8 & print_function settings
We're committed to Python 3 at this point, so clean up boilerplate.

Bug: https://crbug.com/gerrit/10418
Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:05 +00:00
Mike Frysinger
1379a9b185 launcher: add test for version requirements
Make sure the modules stay in sync in case one is updated but we
forgot to update the other.

Bug: https://crbug.com/gerrit/13795
Change-Id: I6de9533d45c083e5f7ad792ee6d541e23647de3f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292444
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 17:56:51 +00:00
Mike Frysinger
128f34e874 main: require Python 3.5 now
We've been warning about Python 3.4 for almost a year.  This drops
support for these systems:
* Ubuntu Trusty: released Apr 2014, EOL Apr 2022
* Debian Jessie: released Apr 2015, EOL Jun 2020

So the min required distros would now be:
* Ubuntu Xenial: released Sep 2015 w/Python 3.5
* Debian Stretch: released Jun 2017 w/Python 3.6

I don't think we're quite ready to drop Python 3.5 which would affect
Ubuntu Xenial -- we'd have to update to Ubuntu Bionic from Apr 2018.
Let's see how much the community reacts to loss of Python 3.4 first.

Bug: https://crbug.com/gerrit/10418
Change-Id: Ib24a57818fdca49e23db53e1bdd1f4c76b4963f7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291502
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 17:55:15 +00:00
Ian Kasprzak
30bc354e25 Enable git trace2 event format logging.
Ways to enable logging:
1) Set git's trace2.eventtarget config variable to desired logging path
2) Specify path via --git_trace2_event_log option

A unique logfile name is generated per repo execution (based on the repo session-id).

Testing:
1) Verified git config 'trace2.eventtarget' and flag enable logging.
2) Verified version/start/end events are expected format:
  https://git-scm.com/docs/api-trace2#_event_format
3) Unit tests

Bug: https://crbug.com/gerrit/13706
Change-Id: I335eba68124055321c4149979bec36ac16ef81eb
Tested-by: Ian Kasprzak <iankaz@google.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292262
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2021-01-06 17:49:41 +00:00
Mike Frysinger
ce9b6c43b2 launcher: abort if python3 reexec failed
We don't support Python 2 anymore, so stop allowing it to fallback.
If we try to run the latest version with Python 2, it just hits
syntax errors which confuses people.  Dump a clear error message
that their system is too old and give up.

Bug: https://crbug.com/gerrit/13795
Change-Id: I38c243cf09502f670cddad72c2d0148f736515e0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292443
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-05 22:42:13 +00:00
Mike Frysinger
47692019b3 launcher: support Python 3.5 for now
The codebase still supports Python 3.5, so allow use of that instead
of requiring Python 3.6+.  Supporting this mode well is a bit tricky
as we want to first scan for newer versions before falling back to
older ones.  And we have to avoid infinite loops in the process.

Bug: https://crbug.com/gerrit/13795
Change-Id: I47949a173899bfa9ab20d3fefa1a97bf002659f6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-05 22:18:24 +00:00
Remy Böhmer
1469c28ec3 project: detach HEAD in internal worktree checkout.
When checkout is done with Git worktrees then the HEAD in the
bare-git repositories point to the initialized default (e.g.
'refs/heads/master'). This default branch does not exist
locally and is not automatically created.
When a user now creates a branch in any git repository named
'master' then it is no longer possible to get rid of this branch,
neither is it possible to switch to another branch and switch
back to this master branch. Git concludes the 'master' branch is
already checked out (in the bare Git) and that results in a
lockdown of this master branch.

To repoduce this issue, run these commands in a repo tree
checked out with --worktree:
- git checkout master # assuming the remote repo has a master branch,
                      # a local tracking branch master is created here
- git checkout -b temp
- git checkout master # This one now fails
- git branch -d master # fails too
The failure is caused by Git assuming the master branch is checked out
by the bare git repository since HEAD is pointing towards it.

To workaround this, we always detach HEAD in the bare-git when
syncing.  We don't need it to point to a ref in general, but we
would like it to be valid so git tools "just work" if they're run
in here.

Signed-off-by: Remy Bohmer <oss@bohmer.net>
Change-Id: I15c96604363c41f0d01c42f533174393097daeb5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290985
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-12-26 07:30:40 +00:00
Chris McDonald
8add62325d Add parallelism to 'branches' command
Spread the operation of querying which local branches exist across a
pool of processes and build the name map of projects -> branches as
these tasks finish rather than blocking on the entire query. The search
operations are submitted in batches to reduce the overhead of interprocess
communication. The `chunksize` argument used to control this batch size
was selected by incrementing through powers of two until it stopped being
faster.

Change-Id: Ie3d7f799ee8e83e5058536caf53e2979175408b7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291342
Tested-by: Chris Mcdonald <cjmcdonald@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-12-14 23:35:12 +00:00
Jashank Jeremy
974774761c docs: Add Markdown inline code marker around inline XML example.
Presently, this tag is not rendered --- by Gitiles, at least --- which
makes the example very confusing indeed.

Signed-off-by: Jashank Jeremy <jashank@rulingia.com.au>
Change-Id: Ia76a60d8ee0ecce8ceb32661afbd48f3b2d80fbf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291362
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jashank Jeremy <jashank.jeremy@gmail.com>
2020-12-13 03:25:36 +00:00
Mike Frysinger
dc60e54d36 gitc: write the manifest directly
Rather than pull the client dir out to construct the manifest
filename which the manifest itself already has, pull the filename
out and use that.

Change-Id: I33991084dcb3205f819bb841084e3c48d6ccb284
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291264
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-12-11 18:07:15 +00:00
Mike Frysinger
0a849b660f replace javadoc docs with standard python style
We don't use javadoc in this project, so clean up the few places
that slipped in with the gitc code.

Change-Id: Ia365fb2d1e3188ad16b2f65b1a3b7e8466001946
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291262
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-12-11 17:55:38 +00:00
Mike Frysinger
5e2f32fe13 init: reset hard to --repo-rev
When updating the tracking ref to whatever the user requested,
make sure we reset state completely rather than trying to update
the ref to it.  This avoids confusing git as to the current state
of the tree, and is more inline with user intentions: if they made
a local change to the checkout, but ran repo init with a specific
rev, we shouldn't stay wedged forever until they manually clean it
all up.

Bug: https://crbug.com/gerrit/12801
Change-Id: Ieba8d9c15781b4d0649bf01c7460694da63387b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290923
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-12-06 21:46:30 +00:00
Mike Frysinger
51e39d536d manifest_xml: harmonize list fields
We allow project.groups to be whitespace or comma delimited, but
repo-hooks.enabled-list is only whitespace delimited.  This hasn't
been a big deal as it's only ever had one valid value, but if we
want to add more, we should harmonize these a bit.

Refactor the groups method to be more generic, and run the enabled-
list attribute through it.  Then add missing docs for it.

Change-Id: Iaa96a0faa9c4a68b313b49336751831b73bf855d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290743
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-12-04 17:27:11 +00:00
Fredrik de Groot
6342d56914 Fix tests after "use new main branch"
Tests worked fine if init.defaultBranch main was used,
but failed due to git branch reasons if master was still used.

Since we can only use init.defaultBranch if git version >= 2.28,
I also went with a template dir HEAD main tweak if lower so tests
now pass regardless of client git default branch and version.

Test: Ran tests with ~/.gitconfig:init.defaultBranch=master
Test: Ran tests with ~/.gitconfig:init.defaultBranch=main
Test: Ran tests for both code branches of git require

Change-Id: I49fa1e4ae45b8aec16a093132ee9fa466cbc11ec
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290404
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-12-03 07:29:59 +00:00
Mike Frysinger
9dfd69f773 run_tests: rewrite to use Python 3
Some distros still have `pytest` as Python 2 and sep `pytest-3`.
Rewrite this script to use `pytest-3` if available.

Change-Id: I430ed8792e7b0da9b217f948f2e983aa62bf1299
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290503
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-12-01 19:29:47 +00:00
Mike Frysinger
08eb63cea4 setup: update Python version info
Change-Id: I91056260d00215cfe9047d17664e3c3158c7bbcc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290502
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-12-01 19:29:43 +00:00
Fredrik de Groot
352c93b680 manifest: add support for groups in include
Attrib groups can now be added to manifest include, thus
all projects in an included manifest file can easily be tagged
with a group without modifying all projects in that manifest file.

Include groups will add and recurse, meaning included manifest
projects will carry all parent includes. Intentionally, no support
added for group remove, to keep complexity down.

Group handling for projects is untouched, meaning a group set on
a project will still append to whatever was or was not inherited
in parent manifest includes, resulting in union of groups inherited
and set for the project itself.

Test: manual multi-level manifest include structure, in serial and parallel,
      with different groups set on init
Test: added unit tests to cover the inheritance

Change-Id: Id2229aa6fd78d355ba598cc15c701b2ee71e5c6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/283587
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-11-26 09:13:14 +00:00
Remy Bohmer
7f7acfe9fd Concentrate the RepoHook knowledge in the RepoHook class
The knowledge about running hooks and all its exception handling
is scattered over multiple files. This makes the code harder
to read, but also it requires duplication of logic in case
other RepoHooks are added to different commands.
This refactoring also creates uniform behavior of the hooks
across multiple commands and it guarantees the re-use of the same
arguments on all of them.

Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: Ia4d90eab429e4af00943306e89faec8db35ba29d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/277562
Tested-by: Remy Bohmer <oss@bohmer.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-11-23 09:59:16 +00:00
Remy Bohmer
169b0218b3 Fix --reference option under Windows
When intializing a new repo with the --reference option on Windows 10
the objects/info/alternates in each git repository is created with
Windows line endings (\r\n), leading to the following error:

error: object directory C:/<PATH_TO_MIRROR>/<REPO_NAME>.git/objects?
does not exist; check .git/objects/info/alternates

This can be fixed by simply using unix line endings on both
Windows and unix platforms.

Reported-by: Francisco Javier Alvarez Garcia <javier.alvarez.garcia.17@gmail.com>
Follow-up-from: I268fe029ede68802c21037b0f2ae8a95afb85e48
Bug: https://crbug.com/gerrit/13208
Change-Id: I6da60c4ca957778b3c42ab6b9ad85c40483f0042
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289431
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <oss@bohmer.net>
2020-11-23 09:17:32 +00:00
Remy Bohmer
44bc9643ed Always use Unix EOL for worktree .git and gitdir files
Worktree .git and gitdir reference files are written by Git with
Unix line ending, even on Windows & macOS. The conversion to
relative paths makes these files end with DOS line endings in
Windows.  The Git integration in Visual Studio 2019 cannot deal
with these DOS line endings and considers these worktrees invalid.

Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: I088cfd994f3cc31db4e0ca7791fa0a4ee3ac222f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289310
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <linux@bohmer.net>
2020-11-20 20:53:43 +00:00
Mike Frysinger
d7f8683daf project: do not update local published/ refs in dryrun mode
Bug: https://crbug.com/gerrit/13087
Change-Id: I197e6d6d07c7d325ac294b597d42e895f77c737f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/289182
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-20 04:08:19 +00:00
Mike Frysinger
8c1e9cbef1 manifest_xml: refactor manifest parsing from client management
We conflate the manifest & parsing logic with the management of the
repo client checkout in a single class.  This makes testing just one
part (the manifest parsing) hard as it requires a full checkout too.

Start splitting the two apart into separate classes to make it easy
to reason about & test.

Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-18 19:10:57 +00:00
Mike Frysinger
a488af5ea5 main: require Python 3 now
We've been warning about this for more than 6 months (with public
announcements even older).  Lets make it a failure now to see who
hasn't upgraded yet.

Bug: https://crbug.com/gerrit/10418
Change-Id: Iec3e2cbf87de434021921616683d360bc4fef77a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280796
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-17 15:04:20 +00:00
Mike Frysinger
e283b95cf2 tests: use new main branch
Now that we clone "main" by default, use that for our local test.

Change-Id: Ib8420074bdfabfcb9d5252a3a0ecd3d852ca36e8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-17 04:29:09 +00:00
Erwan Yvin
dc5c4d1d11 sync: respect --force-sync when fetching manifest project updates
The --force-sync option was being passed down for all updates except
for the manifest project, so add that there too.

Bug: https://crbug.com/gerrit/11034
Change-Id: I33818b652f828c6b847dbc70f1fedfac5ac17bbe
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228146
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-11-17 03:06:06 +00:00
Mike Frysinger
23411d3f9c manifest: add a --json output option
Sometimes parsing JSON is easier than parsing XML, especially when
the XML format is limited (which ours is).  Add a --json option to
the manifest command to quickly emit that form.

Bug: https://crbug.com/gerrit/11743
Change-Id: Ia2bb254a78ae2b70a851638b4545fcafe8c1a76b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280436
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-17 01:38:00 +00:00