Commit Graph

1900 Commits

Author SHA1 Message Date
Mike Frysinger
5892973212 sync: dump git output on errors
Bug: https://crbug.com/gerrit/14700
Change-Id: I1ae53dc7f3792b7e8f11d73f706864fb6591eee8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-07-04 02:28:00 +00:00
Xin Li
0cb6e92ac5 Add the ability to administratively enroll repo into using superproject.
Repo will remember a choice and an expiration time of the choice, per
user, about whether to use superproject by default. When not specified
from command line and the choice is not expired, repo would use the
user default value.

When a user default value is not present and when the system wide
enable default is provided in git's system configuration, repo would
ask the user for a confirmation which will be valid for two weeks.

git_config.py: Add support for system config. When reading system
	config, we would use --system to avoid hardcoding a path as the
	value may be different on some other distributions.

git_superproject.py: Add a new subroutine, _UseSuperproject(), which
	returns whether superproject should be used and whether it
	is from a user configuration.

	The value is determined in the following order:

	1. If the user specifies either --use-superproject or
	--no-use-superproject, then that choice is being used.

	2. If neither is specified, we would then check the saved value
	(upon repo init) and use that choice when there was a choice.

	3. We then check if there is a saved and unexpired value for
	user's choice in their ~/.gitconfig, and use the unexpired
	choice, if available.

	4. Finally, if all the above didn't give us a decision, and if
	the git system configuration is providing a rollout hint, present
	a prompt to user for their decision and save it in ~/.gitconfig.

subcmds/sync.py: Make use of the new UseSuperproject() provided by
git_superproject.py.

While there also silent stderr from git describe when determining the
version of repo.

Bug: [google internal] b/190688390
Change-Id: Iad3ee03026342ee500e5d65e2f0fa600d7637613
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309762
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
2021-07-01 16:27:43 +00:00
Xin Li
0e776a5837 Fix an issue when syncing with --use-superproject and clone bundles.
It is possible that a clone bundle contained the object referenced by
the branch in the manifest and in the superproject, but not the branch
itself (for example, the branch may be newly created from an existing
branch, or is not vislble to the user downloading the clone bundle).

When --use-superproject is enabled, because we are overriding
revisionExpr with the SHA1 revision provided by the superproject, the
verification step would succeed, but because the expected branch do not
exist, it would confuse git-repo at a later time, as it is expecting the
remote branch to exist in the local clone.

In project.py, fix this by making SetRevisionId() to always remember
the actual branch name and verify it in _CheckForImmutableRevision()
so that we only skip the fetch step when both objects exists locally.

Bug: [google internal] b/191974277
Change-Id: I49d3ca0667f524c8c45f416492faf95b1dd822fb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
2021-06-30 15:31:15 +00:00
Raman Tenneti
1da6f30579 superproject - don't update the commit ids of projects that have revisionId.
Pinned manifests and release manifest have revisionId set for all
projects. For such projects don't update the commit ids.

Tested the code with the following commands.

$ ./run_tests -v

$ repo_dev sync --use-superproject -j8

$ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml

Bug: [google internal] b/191995372
Change-Id: I4681135b1d15f4a63527b6f0356d76ec842485d6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310582
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-06-29 16:46:07 +00:00
Raman Tenneti
784e16f3aa superproject: Don't exit if superproject tag doesn't exist in manifest.
Don't exit if there are missing commit ids in superproject.

This change implements the following suggestion from delphij@:

"we should note the event (so we know that --use-superproject but there
 were some errors, e.g. manifest didn't specify commit id for some
 reason, or if there is no superproject but --use-superproject is
 used), print out a message telling the use that this is not support,
 but continue as if --no-use-superproject was specified?"

Changes:

superproject:
+ Added git_trace2_event_log as an argument to the constructor.
+ Sync method returns SyncResult a NamedTuple of
  ++ success - True if sync of superproject is successful, or False.
  ++ fatal - True if caller should exit, Or False.
+ UpdateProjectsRevisionId returns UpdateProjectsResult a NamedTuple of
   ++ manifest_path - path name of the overriding manifest file instead
      of None
   ++ fatal - True if caller should exit, Or False
+ _GetAllProjectsCommitIds returns CommitIdsResult a NamedTuple of
  ++ commit_ids - a dictionary with the projects/commit ids on success,
     otherwise None
  ++ fatal - True if caller should exit, Or False
+ Added  _SkipUpdatingProjectRevisionId a helper function to see if a
  project's revision id needs to be updated or not. This function is
  used to exclude projects from local manifest file.
+ Added the following error events into git_trace2_event_log
  ++ If superproject is missing in a manifest
  ++ If there are missing commit ids for projects.

command.py:
+ Deleted unused import - platform
+ Added git_trace2_event_log as a member so all subcmds can log error
  events.

main.py:
+ Initialized git_trace2_event_log as a member of command object.

init.py:
+ Deleted unused import - optparse

init.py:
+ Called sys.exit only if Sync returns exit=True

sync.py:
+ Called sys.exit only if Superproject's UpdateProjectsRevisionId returns
  exit=True
+ Reloaded the manifest only if manifest path is returned by
  UpdateProjectsRevisionId. If not, fall back to the old way of doing
  repo sync.

test_git_superproject:
+ Added code to verify error events are being logged.
+ Added a test for no superproject tag
+ Added test for UpdateProjectsRevisionId not updating the revision id
  with the commit ids.

Tested the code with the following commands.

+ Positive test case with aosp-master.
  $ repo_dev init -u persistent-https://android.git.corp.google.com/platform/manifest -b master --use-superproject
  NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
  .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.

  Your identity is: Raman Tenneti <rtenneti@google.com>
  If you want to change this, please re-run 'repo init' with --config-name

  repo has been initialized in .../android/aosp

  $ repo_dev sync -j40 --use-superproject
  remote: Total 12 (delta 4), reused 12 (delta 4)
  NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
  .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
  ...
  repo sync has finished successfully.

+ Negative test case without superproject tag.
  $ repo_dev sync -j40 --use-superproject
  NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
  repo error: superproject tag is not defined in manifest: .../android/aosp/.repo/manifest.xml
  error: Cannot get project commit ids from manifest
  error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
  ...
  Checking out: 100% (1022/1022), done in 3.589s
  repo sync has finished successfully.

+ Test for missing commit_id for a project.
  $ repo_dev sync -j40 --use-superproject
  NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
  .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
  error: please file a bug using go/repo-bug to report missing commit_ids for: ['build/blueprint']
  error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option
  ...
  Checking out: 100% (1022/1022), done in 3.364s
  repo sync has finished successfully.

$ ./run_tests -v
  ...
  ...== 164 passed in 2.87s ==...

Bug: [google internal] b/189371541
Change-Id: I5ea49f87e8fa41be590fc0c914573e16c8cdfcfa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309162
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-06-16 04:48:35 +00:00
Jack Neus
b8c84483a5 repo: improve duplicate default check
If one default is totally empty, we don't need to fail.

BUG=b:187795796
TEST=unit tests

Change-Id: Id226a7a7cd183dbdee58f4681b84885cc9211375
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309102
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
2021-06-15 18:06:13 +00:00
Mike Frysinger
d58d0dd3bf commands: pass settings via __init__
Instead of setting properties on the instantiated command, pass them
via the constructor like normal objects.

Change-Id: I8787499bd2be68565875ffe243c3cf2024b36ae7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309324
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-06-15 06:08:13 +00:00
Mike Frysinger
d88b369a42 commands: document the "event_log" class attribute
Add some notes explaining why it's instantiated at the Command class
level and not individual objects.

Change-Id: Ib8081bb8480e85f6d3dfc23953c6bbc6ecc64934
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309323
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-06-15 06:08:00 +00:00
Mike Frysinger
4f21054c28 commands: document the "common" class attribute
Switch it to uppercase to make it clear it's a constant, and add
documentation so its usage is clear.

Change-Id: I6d281a66a90b5908b3131585c9945e88cfe815ea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309322
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-06-15 06:07:37 +00:00
Jack Neus
5ba2120362 repo: properly handle NoneType in Default/Remote equality checks
BUG=none
TEST=none

Change-Id: I4ccdbbc7ba4b6f6e20c6959db1b46fdb44ea2819
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308982
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
2021-06-11 13:54:32 +00:00
Raman Tenneti
78f4dd3138 superproject: add projects from local manifest to local::<filename> group.
With repo sync --use-superproject, don't update the commit ids of every project
that comes from local manifest.

Tested the code with the following commands.

$ ./run_tests -v

+ Test with local.xml

1. repo init --use-superproject -u persistent-https://googleplex-android.git.corp.google.com/a/platform/manifest

2. cd .repo
cp -r /google/src/head/depot/google3/wireless/android/build_tools/translations/pipeline/local_manifests local_manifests
cd ..

local$ time repo_dev sync --use-superproject
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
.../local/.repo/exp-superproject/feb2c2847da5e274f3d530d5ab438af8-superproject.git: Initial setup for superproject completed.
...

Bug: [google internal] b/189360443
Bug: [google internal] b/189139268
Bug: https://crbug.com/gerrit/14499
Change-Id: Ideaf268c294e9b500b2b9726ffbd733dd8d63004
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308822
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2021-06-10 00:16:36 +00:00
Raman Tenneti
fc7aa90623 trace2_event_log: Added logging of error events.
Added error event in preperation for superproject to log errors.

Testing:
+ Unit tests
   ./run_tests -v

Bug: [google internal] b/189371541
Change-Id: Ife1dd28d52d9e9925b7b34ae913f8eb5fa19037c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308863
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-06-09 14:24:20 +00:00
Raman Tenneti
50c91ecf4f superproject: revert not updating commit ids if remote is different.
superproject supports multiple remotes. Get all commit ids
from superproject for all projects that are in the manifest.

$ ./run_tests -v

Bug: [google internal] b/186395810
Change-Id: I6edce3918853a7a3a65aec5528e6a43a544eff53
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308862
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-06-08 22:43:32 +00:00
Mike Frysinger
816d82c010 run_tests: fix pytest selection inside tox venv
Finding the "right" pytest is challenging.  In Debian, `pytest` is
Python 2 while `pytest-3` is the Python 3 version ... but only when
outside of a virtualenv.  Inside of a virtualenv (e.g. the ones that
tox creates), we always want `pytest`.

Change-Id: Ic1fe84c10f06227bceeb9baad6a3c4598bbe9860
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303802
Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-06-03 15:27:51 +00:00
Raman Tenneti
2b37fa3f26 superproject: change the warning message to say it is beta.
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
remote: Total 3 (delta 0), reused 3 (delta 0)
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
...

$ ./run_tests -v

Bug: [google internal] b/189946009
Change-Id: Ifb3ef266a72b67f3c4a2a3ac2033b10e03b789d4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308522
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-06-03 14:38:09 +00:00
Peter Kjellerstedt
a3b2edf1af Drop support for Python 3.5
Running repo with Python 3.5 fails due to the use of the encoding
parameter to subprocess.run(). There are also f-strings being used in
some of the tests.

This drops support for these systems:
* Ubuntu Xenial: released Apr 2016, EOS Apr 2021, EOL Apr 2024
* Debian Stretch: released Jun 2017, EOL Jun 2022

So the minimum required distros now are:
* Ubuntu Bionic: released Apr 2018 w/Python 3.6
* Debian Buster: released Jul 2019 w/Python 3.7

Change-Id: I1144f7ab6f882b10cac0131982df081fe4ac44f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303363
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
2021-06-03 11:34:17 +00:00
Raman Tenneti
e253b43e17 superproject: require git version 2.28.0 or higher.
Tested the code with the following commands.

$ repo init --use-superproject -u https://android.googlesource.com/platform/manifest

+ Tested for the wrong git version.
  $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
  WARNING: --use-superproject is experimental and not for general use
  superproject requires a git version 2.28 or later
  error: git update of superproject failed

$ ./run_tests -v

Bug: https://crbug.com/gerrit/14617
Bug: [google internal] b/189846687
Change-Id: I5cd4158ea29b3b3c8c81234f4e818165de346e63
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308442
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-06-03 00:30:10 +00:00
Peter Kjellerstedt
5d58c18146 tests: Make the tests pass for Python < 3.8
Before Python 3.8, xml.dom.minidom sorted the attributes of an element
when writing it to a file, while later versions output the attributes
in the order they were created. Avoid these differences by sorting the
attributes for each element before comparing the generated manifests
with the expected ones.

Bug: https://crbug.com/gerrit/14382
Change-Id: Ie2597727afcc48f9063a7261ad970e8a549f0587
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303326
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-26 13:36:20 +00:00
Peter Kjellerstedt
d177609cb0 sync: Sync submodules correctly again
Commit b4429439 (sync: refactor main fetch loop) broke syncing of
submodules with sync-s="true". The first `repo sync` would just fetch
the superrepo, while the second `repo sync` would fetch the submodules.
This was due to the new _FetchMain() function not passing back the
all_projects variable that it had modified.

Change-Id: Ie8187cde7bb894e4e9a6b76c2aed83873d9f69a4
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/307065
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-20 16:23:10 +00:00
Mike Frysinger
b16b9d26bd project: fix error display when output_redir is disabled
We always pass in output_redir when syncing, but that's the common case:
there are a few situations (like `repo init`) where we don't pass in a
buffer, and if any errors show up in that case, we'd crash.  Rely on the
print function to handle this logic for us.

Bug: https://crbug.com/gerrit/14568
Change-Id: I8cd47e82329797ffc42534418a3dfbd8429205be
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/307222
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-20 06:15:38 +00:00
Raman Tenneti
993af5e136 superproject: Use bugurl from contactinfo in the missing commits error message.
+ In XmlManifest._Unload set 'bugurl' to Wrapper().BUG_URL.
+ contactinfo returns a namedtuple.
+ bug_url can be accessed as self._manifest.contactinfo.bugurl.

Tested the code with the following commands.

$ ./run_tests -v

Added contactinfo tag to default.xml and verified that bugurl is used.

Bug: [google internal] b/186220520.
Change-Id: Iaafd6465e072b2e47a0a0b548bf6cb608a0b0a04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/306342
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-18 15:35:54 +00:00
Mike Frysinger
339f2df1dd ssh: rewrite proxy management for multiprocessing usage
We changed sync to use multiprocessing for parallel work.  This broke
the ssh proxy code as it's all based on threads.  Rewrite the logic to
be multiprocessing safe.

Now instead of the module acting as a stateful object, callers have to
instantiate a new ProxyManager class that holds all the state, an pass
that down to any users.

Bug: https://crbug.com/gerrit/12389
Change-Id: I4b1af116f7306b91e825d3c56fb4274c9b033562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305486
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-05-10 21:16:06 +00:00
Mike Frysinger
19e409c818 ssh: move proxy usage to the sync subcommand
The only time we really need ssh proxies is when we want to run many
connections and reuse them.  That only happens when running sync.
Every other command makes at most two connections, and even then it's
only one or none.  So the effort of setting up & tearing down ssh
proxies isn't worth it most of the time.

The big reason we want to move this logic to sync is that it's now
using multiprocessing for parallel work.  The current ssh proxy code
is all based on threads, which means none of the logic is working
correctly.  The current ssh design makes it hard to fix when all of
the state lives in the global/module scope.

So the first step to fixing this is top move the setup & teardown to
the one place that really needs it: sync.  No other commands will use
proxies anymore, just direct connections.

Bug: https://crbug.com/gerrit/12389
Change-Id: Ibd351acdec39a87562b3013637c5df4ea34e03c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305485
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-10 21:10:29 +00:00
Mike Frysinger
4a58100251 launcher: bump version for new release
Change-Id: I1f204bb1e5ce6b13c623215236deef01efbc0f6c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305822
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-10 17:10:04 +00:00
Craig Northway
0e8828c47b Handle 400 error code when attempting to fetch clone bundle.
Gitlab returns a 400 error when trying to fetch clone.bundle
from a repository containing the git-repo tool. The repo
launcher doesn't then fall back to not using a clone.bundle
file and the repo init fails.

Change-Id: Ia3390d0638ef9a39fb2fab84625b269d28caf1cc
Signed-off-by: Craig Northway <cnorthway@codeaurora.org>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305382
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-10 16:34:34 +00:00
Raman Tenneti
23ea754524 sync: added --no-use-superproject to disable superproject.
Tested the code with the following commands.

$ ./run_tests -v

$ repo_dev sync -c -j8 --no-use-superproject
Fetching: 100% (1041/1041), done in 1m22.743s

$ repo_dev sync -c -j8 --use-superproject
WARNING: --use-superproject is experimental and not for general use
..

Bug: [google internal] b/187459275
Change-Id: I3f4269df38cd24a21723e8b2be5a1f013e7b5a91
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305682
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-08 00:04:00 +00:00
Raman Tenneti
f907ced0fe sync: Recommend using --no-use-superproject if sync fails.
If superproject was not available for a branch, then the next
repo sync would also fail because --use-superproject is
remembered across repo init. In such cases, hoping the hint to
to use --no-use-superproject will help.

Tested the code with the following commands and by forcing
a failure.

$ ./run_tests -v

Bug: [google internal] b/187459275
Change-Id: Ie250812b7ba83afc230b5b1d154ba11f245f8b8a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305622
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-05-07 20:15:50 +00:00
Mike Frysinger
b44294395f sync: refactor main fetch loop
This is a large chunk of code that is largely isolated.  Move it into
a class method to make it easier to manage & reason about, and in a
follow up CL, easier to scope.

Bug: https://crbug.com/gerrit/12389
Change-Id: I0c69d95a9e03478d347b761580b2343bffa012d5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305484
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-05-06 19:46:09 +00:00
Mike Frysinger
5291eafa41 ssh: move all ssh logic to a common place
We had ssh logic sprinkled between two git modules, and neither was
quite the right home for it.  This largely moves the logic as-is to
its new home.  We'll leave major refactoring to followup commits.

Bug: https://crbug.com/gerrit/12389
Change-Id: I300a8f7dba74f2bd132232a5eb1e856a8490e0e9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305483
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-06 19:09:16 +00:00
Mike Frysinger
8e768eaaa7 git_command: switch version caches to functools
Simplifies the code a bit to use the stdlib cache helper.

Change-Id: I778e90100ce748a71cc3a5a5d67dda403334315e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305482
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-06 18:36:25 +00:00
Raman Tenneti
2f8fdbecde manifest_xml: cleanup of contactinfo test for readability with f-strings.
Tested the code with the following commands.

$ ./run_tests -v

Bug: [google internal] b/186220520.
Change-Id: I1c0f8958ff4c615707eec218250e8de753ec6562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305282
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-05-05 15:59:58 +00:00
Mike Frysinger
148e1ce81a sync: fix recursive fetching
Commit b2fa30a2b8 ("sync: switch network
fetch to multiprocessing") accidentally changed the variable passed to
the 2nd fetch call from |missing| to |to_fetch| due to a copy & paste
of the earlier changed logic.  Undo that to fix git submodule fetching.

Bug: https://crbug.com/gerrit/14489
Change-Id: I627954f80fd2e80d9d5809b530aa6b0ef9260abb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305262
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 22:43:09 -04:00
Mike Frysinger
219431e1c9 sync: fix recursive fetching
Commit b2fa30a2b8 ("sync: switch network
fetch to multiprocessing") accidentally changed the variable passed to
the 2nd fetch call from |missing| to |to_fetch| due to a copy & paste
of the earlier changed logic.  Undo that to fix git submodule fetching.

Bug: https://crbug.com/gerrit/14489
Change-Id: I627954f80fd2e80d9d5809b530aa6b0ef9260abb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305262
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-05 02:42:23 +00:00
Mike Frysinger
32ca6687ae git_config: hoist Windows ssh check earlier
The ssh master logic has never worked under Windows which is why this
code always returned False when running there (including cygwin).  But
the OS check was still done while holding the threading lock.  While
it might be a little slower than necessary, it still worked.

The switch from the threading module to the multiprocessing module
changed global behavior subtly under Windows and broke things: the
globals previously would stay valid, but now they get cleared.  So
the lock is reset to None in children workers.

We could tweak the logic to pass the lock through, but there isn't
much point when the rest of the code is still disabled in Windows.
So perform the platform check before we grab the lock.  This fixes
the crash, and probably speeds things up a few nanoseconds.

This shouldn't be a problem on Linux systems as the platform fork
will duplicate the existing process memory (including globals).

Bug: https://crbug.com/gerrit/14480
Change-Id: I1d1da82c6d7bd6b8cdc1f03f640a520ecd047063
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305149
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 19:49:58 -04:00
Mike Frysinger
5ba80d404c git_config: hoist Windows ssh check earlier
The ssh master logic has never worked under Windows which is why this
code always returned False when running there (including cygwin).  But
the OS check was still done while holding the threading lock.  While
it might be a little slower than necessary, it still worked.

The switch from the threading module to the multiprocessing module
changed global behavior subtly under Windows and broke things: the
globals previously would stay valid, but now they get cleared.  So
the lock is reset to None in children workers.

We could tweak the logic to pass the lock through, but there isn't
much point when the rest of the code is still disabled in Windows.
So perform the platform check before we grab the lock.  This fixes
the crash, and probably speeds things up a few nanoseconds.

This shouldn't be a problem on Linux systems as the platform fork
will duplicate the existing process memory (including globals).

Bug: https://crbug.com/gerrit/14480
Change-Id: I1d1da82c6d7bd6b8cdc1f03f640a520ecd047063
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305149
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 23:49:29 +00:00
Raman Tenneti
1c3f57e8f1 manifest_xml: initial support for <contactinfo>
It will be used to let manifest authors self-register contact info.
This element can be repeated, and any later entries will clobber
earlier ones. This would allow manifest authors who extend
manifests to specify their own contact info.

It would have 1 required attribute: bugurl.
"bugurl" specifies the URL to file a bug against the manifest owner.

<contactinfo bugurl="bug-url"/>

TODO: This CL only implements the parsing logic and further work
will be in followup CLs.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: [google internal] b/186220520.
Change-Id: I47e765ba2dab5cdf850191129f4d4cd6b803f451
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305203
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-04 22:36:01 +00:00
Mike Frysinger
05638bf771 sync: use manifest_name passed in
Commit fb527e3f52 ("sync: create dedicated
manifest project update func") refactored code from the main body into a
dedicated method.  The manifest_name was passed as an argument, but never
used it, and instead reaches back out to the command line options.  This
ignores the logic in the main loop where manifest_name might have changed
(like when using smart sync).

Change-Id: I4b84638fbb10c2b6f8f4b555e1475b0669c2daf4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305148
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 20:00:50 +00:00
Mike Frysinger
c99322a6a9 sync: switch to multiprocessing.Event
We've switched most of this command over to multiprocessing and off
of _threading, so do the Event object too.  The APIs are the same
between the modules, so we shouldn't need to update anything else.

Change-Id: I52d31f1c6ef2bcbe7bbc1dd1add79a8d5d08784a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305147
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 20:00:41 +00:00
Mike Frysinger
14208f4c93 sync: fix logic error with linkfile errors
Make sure err_update_linkfiles is always initalized.

Bug: https://crbug.com/gerrit/11008
Change-Id: I7bdd91f82507608ef967daf0fa0f9c859454e19f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305146
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 20:00:20 +00:00
Mike Frysinger
2ee0a62db0 release-process: document the rate limiting in automatic updates
We check for updates only once per day, so clarify the docs.

Change-Id: Ib669ca6ebc67bc13204996fa40e1a3a82012295e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305145
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 19:37:48 +00:00
Mike Frysinger
0ae9503a86 sync: fix print error when handling server error
When converting this logic from print() to the output buffer, this
error codepath should have dropped the use of the file= redirect.

Bug: https://crbug.com/gerrit/14482
Change-Id: Ib484924a2031ba3295c1c1a5b9a2d816b9912279
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 12:48:42 -04:00
Mike Frysinger
c177f944d9 subcmds: force consistent help text format
We're inconsistent with help text as to whether it uses title case and
whether it ends in a period.  Add a test to enforce a standard, and use
the style that Python optparse & argparse use themselves (e.g. with the
--help option): always lowercase, and never trailing period.

Change-Id: Ic1defae23daeac0ac9116aaf487427f50b34050d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305144
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 16:40:53 +00:00
Mike Frysinger
aedd1e5ef0 sync: fix print error when handling server error
When converting this logic from print() to the output buffer, this
error codepath should have dropped the use of the file= redirect.

Bug: https://crbug.com/gerrit/14482
Change-Id: Ib484924a2031ba3295c1c1a5b9a2d816b9912279
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 16:40:28 +00:00
Raman Tenneti
5a41b0be01 superproject: skip updating commit ids if remote's fetchUrl don't match.
Tested the code with the following commands.

$ ./run_tests -v

+ Test with local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml  .repo/local_manifests/local.xml

  $ repo_dev sync -c -j8

+ Test without local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
  $ repo_dev sync -c -j8

Bug: [google internal] b/186395810
Change-Id: Id618113a91c12bcb90a30a3c23d3d6842bcb49e1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304942
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-05-04 15:32:23 +00:00
Mike Frysinger
d68ed63328 init/sync: add --no-tags to match --tags
While this provides a way to undo earlier command line options (e.g.
`repo sync --tags --no-tags`) which can be helpful for scripting &
automation, this more importantly allows the user to override the
manifest settings for syncing tags from a project.

Bug: https://crbug.com/gerrit/12401
Change-Id: Id4c36cd82e6ca7cb073b5d63a09f6c7ccdebba83
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304904
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 11:32:17 +00:00
Mike Frysinger
7356114d90 add --no-current-branch option to invert --current-branch
For most commands, this is more about providing a way to undo earlier
command line options (e.g. `repo info -c --no-current-branch`) which
can be helpful for scripting & automation.  But for the sync command,
this is helpful to undo the setting that exists in the manifest itself.

With this in place, tweak the sync current_branch_only logic to only
apply the manifest settings when the user hasn't specified a command
line option.

Bug: https://crbug.com/gerrit/12401
Change-Id: I21e2384624680cc740d1b5d1e49c50589d2fe6a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304903
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 11:31:48 +00:00
Mike Frysinger
b8e09ea1d6 harmonize --current-branch short option across subcommands
We're inconsistent with the short option for this flag:
* gitc-init: <none as -c is already used>
* info:      -b
* init:      -c
* overview:  -b
* sync:      -c
* upload:   --cbr

Since info & overview are not as heavily used as the others, switch
them from -b to -c.  We leave -b in as a hidden alias for now.

Similarly, switch upload from --cbr to just -c.  A lot of people
use --cbr, so we leave this as a hidden alias for now too.

Ideally gitc-init wouldn't use -c, but that ship has sailed, and
we're more likely to deprecate gitc entirely at this point.

This provides a consistent set of options across subcommands.

Bug: https://crbug.com/gerrit/12401
Change-Id: Iec249729223866fe1ea0ebabed12ca851cc38b35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304902
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-03 16:57:23 +00:00
Raman Tenneti
feb28914bd superproject: Don't update the commit ids of projects if remote is different.
1) Skip setting the revision id (commit id) for the projects whose
   remote doesn't match superproject's remote.
2) exp-superproject/superproject_override.xml includes local_manfiest's
   projects. When we load this XML, don't reload projects from local.xml
   (otherwise we will get duplicate projects errors).

Tested the code with the following commands.

$ ./run_tests -v

+ Test with local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml  .repo/local_manifests/local.xml

  $ repo_dev sync -c -j8

+ Test without local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
  $ repo_dev sync -c -j8

Bug: [google internal] b/186395810
Change-Id: I4e9d4ac2d94a9fc0cef0ccd787b6310758009e86
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304882
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-03 05:13:23 +00:00
Mike Frysinger
d1f3e149df upload: search local projects in parallel
Search for project branches to upload in parallel.  This can cut the
lookup time in half for large projects.  We still run the actual hooks
in serial once we have the list of projects to process, but we would
need to rethink things quite a bit before we could handle running them
in parallel too.

Change-Id: I8da0cbc5010566aa860e1a158f3dc07f0709dcff
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304842
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-02 00:06:32 +00:00
Mike Frysinger
29626b4f46 project: fix m/ generation when switching manifest branches
We were updating the per-checkout m/ pseudo ref when syncing, but we
only created the common m/ redirect when initializing a project for
the first time.  This is fine unless the user switches the manifest
branch in an existing project, then we never create that redirect.

Bug: https://crbug.com/gerrit/14468
Change-Id: I5325e7e602dcb4ce150bef258901ba5e9fdea461
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304822
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-02 00:05:55 +00:00