Commit Graph

82 Commits

Author SHA1 Message Date
Peter Kjellerstedt
5d58c18146 tests: Make the tests pass for Python < 3.8
Before Python 3.8, xml.dom.minidom sorted the attributes of an element
when writing it to a file, while later versions output the attributes
in the order they were created. Avoid these differences by sorting the
attributes for each element before comparing the generated manifests
with the expected ones.

Bug: https://crbug.com/gerrit/14382
Change-Id: Ie2597727afcc48f9063a7261ad970e8a549f0587
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303326
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-26 13:36:20 +00:00
Raman Tenneti
993af5e136 superproject: Use bugurl from contactinfo in the missing commits error message.
+ In XmlManifest._Unload set 'bugurl' to Wrapper().BUG_URL.
+ contactinfo returns a namedtuple.
+ bug_url can be accessed as self._manifest.contactinfo.bugurl.

Tested the code with the following commands.

$ ./run_tests -v

Added contactinfo tag to default.xml and verified that bugurl is used.

Bug: [google internal] b/186220520.
Change-Id: Iaafd6465e072b2e47a0a0b548bf6cb608a0b0a04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/306342
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-18 15:35:54 +00:00
Mike Frysinger
339f2df1dd ssh: rewrite proxy management for multiprocessing usage
We changed sync to use multiprocessing for parallel work.  This broke
the ssh proxy code as it's all based on threads.  Rewrite the logic to
be multiprocessing safe.

Now instead of the module acting as a stateful object, callers have to
instantiate a new ProxyManager class that holds all the state, an pass
that down to any users.

Bug: https://crbug.com/gerrit/12389
Change-Id: I4b1af116f7306b91e825d3c56fb4274c9b033562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305486
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-05-10 21:16:06 +00:00
Mike Frysinger
5291eafa41 ssh: move all ssh logic to a common place
We had ssh logic sprinkled between two git modules, and neither was
quite the right home for it.  This largely moves the logic as-is to
its new home.  We'll leave major refactoring to followup commits.

Bug: https://crbug.com/gerrit/12389
Change-Id: I300a8f7dba74f2bd132232a5eb1e856a8490e0e9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305483
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-06 19:09:16 +00:00
Mike Frysinger
8e768eaaa7 git_command: switch version caches to functools
Simplifies the code a bit to use the stdlib cache helper.

Change-Id: I778e90100ce748a71cc3a5a5d67dda403334315e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305482
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-06 18:36:25 +00:00
Raman Tenneti
2f8fdbecde manifest_xml: cleanup of contactinfo test for readability with f-strings.
Tested the code with the following commands.

$ ./run_tests -v

Bug: [google internal] b/186220520.
Change-Id: I1c0f8958ff4c615707eec218250e8de753ec6562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305282
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-05-05 15:59:58 +00:00
Raman Tenneti
1c3f57e8f1 manifest_xml: initial support for <contactinfo>
It will be used to let manifest authors self-register contact info.
This element can be repeated, and any later entries will clobber
earlier ones. This would allow manifest authors who extend
manifests to specify their own contact info.

It would have 1 required attribute: bugurl.
"bugurl" specifies the URL to file a bug against the manifest owner.

<contactinfo bugurl="bug-url"/>

TODO: This CL only implements the parsing logic and further work
will be in followup CLs.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: [google internal] b/186220520.
Change-Id: I47e765ba2dab5cdf850191129f4d4cd6b803f451
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305203
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-04 22:36:01 +00:00
Mike Frysinger
c177f944d9 subcmds: force consistent help text format
We're inconsistent with help text as to whether it uses title case and
whether it ends in a period.  Add a test to enforce a standard, and use
the style that Python optparse & argparse use themselves (e.g. with the
--help option): always lowercase, and never trailing period.

Change-Id: Ic1defae23daeac0ac9116aaf487427f50b34050d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305144
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 16:40:53 +00:00
Raman Tenneti
feb28914bd superproject: Don't update the commit ids of projects if remote is different.
1) Skip setting the revision id (commit id) for the projects whose
   remote doesn't match superproject's remote.
2) exp-superproject/superproject_override.xml includes local_manfiest's
   projects. When we load this XML, don't reload projects from local.xml
   (otherwise we will get duplicate projects errors).

Tested the code with the following commands.

$ ./run_tests -v

+ Test with local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml  .repo/local_manifests/local.xml

  $ repo_dev sync -c -j8

+ Test without local.xml
  $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
  $ repo_dev sync -c -j8

Bug: [google internal] b/186395810
Change-Id: I4e9d4ac2d94a9fc0cef0ccd787b6310758009e86
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304882
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-05-03 05:13:23 +00:00
Mike Frysinger
f69c7ee318 manifest_xml: ban use of newlines in paths
There should be no valid use of these anywhere, so just ban them
to make things easier for people.

Bug: https://crbug.com/gerrit/14156
Bug: https://crbug.com/gerrit/14200
Change-Id: I8d2cf988c510c98194c43a329a2b9bf313a3f0a8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304662
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-04-30 05:54:11 +00:00
Peter Kjellerstedt
b8bf291ddb tests: Make ReviewableBranchTests.test_smoke work with git < 2.28.0
Bug: https://crbug.com/gerrit/14380
Change-Id: Id015bd98b008e1530ada2c7e4332c67e8e208e25
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303325
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-04-14 16:22:52 +00:00
Raman Tenneti
a5b40a2845 repo: Add a new "command" event type to git trace2 logging in repo.
Add a new "event": "command", which is emitted at when all command
arguments have been processed.

Additional fields:

"name": Name of the primary command (ex: repo, git)

"subcommands"': List of the sub-commands once command-line arguments
                are processed

Examples:

Command: repo --version

Event: {"event": "command", <common fields>,
        "name": "repo",
	"subcommands": ["version"]
	}

Bug: [google internal] b/178507266
Testing:
- Unit tests
- Verified repo git trace2 logs had expected data

Change-Id: I825bd0ecedee45135382461a4ba10f987f09aef3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300343
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-18 14:58:24 +00:00
Mike Frysinger
0458faa502 manifest: allow toplevel project checkouts
Re-allow checking out projects to the top of the repo client checkout.
We add checks to prevent checking out files under .repo/ as that path
is only managed by us, and projects cannot inject content or settings
into it.

Bug: https://crbug.com/gerrit/14156
Bug: https://crbug.com/gerrit/14200
Change-Id: Id6bf9e882f5be748442b2c35bbeaee3549410b25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299623
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-12 16:31:14 +00:00
Raman Tenneti
080877e413 superproject: pass groups to ToXml method.
Added the following methods to XmlManifest class.
+ GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower()
+ GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func.

+ Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr.
+ Used the above methods to get groups in command.py::GetProjects
  and part of init.py.

TODO: clean up these funcs to take structured group data more instead
      of passing strings around everywhere that need parsing.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL
and verified prebuilts/fullsdk-linux directory has all the folders.

Tested repo init and repo sync with --use-superproject and without
--use-superproject argument.

$ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main  --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject

$ repo_dev sync -c -j32

Bug: [google internal] b/181804931
Bug: https://crbug.com/gerrit/13707
Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-03-11 01:24:52 +00:00
Ian Kasprzak
835a34bdb9 Log repo.* config variables in git trace2 logger.
Bug: [google internal] b/181758736
Testing:
- Unit tests
- Verified repo git trace2 logs had expected data

Change-Id: I9af8a574377bd91115f085808c1271e9dee16a36
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299182
Tested-by: Ian Kasprzak <iankaz@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
2021-03-08 17:32:09 +00:00
Mike Frysinger
934cb0a849 tests: fix duplicate method from copy & paste error
Change-Id: Ib748c61b1e65aee6dff8b97a9753d14c470a827f
Reported-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299002
Reviewed-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-04 16:17:11 +00:00
Mike Frysinger
5413397204 manifest: relax include name rules for user-specified path
Allow the user to specify relative or absolute or any other funky
path that they want when using `repo init` or `repo sync`.  Our
goal is to restrict the paths in the remote manifest git repo we
cloned from the network, not protect the user from themselves.

Bug: https://crbug.com/gerrit/14156
Change-Id: I1ccfb2a6bd1dce2bd765e261bef0bbf0f8a9beb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298823
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-03-02 03:18:57 +00:00
Mike Frysinger
a29424ea6d manifest: validate project name & path and include name attributes
These attribute values are used to construct local filesystem paths,
so apply the existing filesystem checks to them.

Bug: https://crbug.com/gerrit/14156
Change-Id: Ibcceecd60fa74f0eb97cd9ed1a9792e139534ed4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298443
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-28 16:07:12 +00:00
Mike Frysinger
a00c5f40e7 manifest: refactor the filesystem checking logic for more reuse
This function is currently written with copyfile & linkfile in mind.
Generalize the logic & function arguments slightly so we can reuse
in more places that make sense.

This changes the validation logic slightly too in that we no longer
allow "." for the dest attribute with copyfile & linkfile, nor for
the src attribute with copyfile.  We already rejected those later on
when checking against the active filesystem, but now we reject them
a little sooner when parsing.

The empty path check isn't a new requirement exactly -- repo used to
crash on it, so it was effectively blocked, but now we diagnosis it.

Bug: https://crbug.com/gerrit/14156
Change-Id: I0fdb42a3da60ed149ff1997c5dd4b85da70eec3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298442
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-28 16:07:12 +00:00
Raman Tenneti
ceba2ddc13 sync: superproject - support for switching hosts and switching branches.
+ superproject will be fetched into a directory with the name
  “<remote name>-superproject.git” instead of the current
  “superproject.git” folder.

+ Deleted  _Clone method and added _Init method.

+ _Init method will do “git init --bare <remote>-superproject.git”.
  It will create the folder and set up a bare repository in
  <remote>-superproject.git folder.

+ _Fetch method, will pass <remote url>, <branch> arguments.
  Moved the --filter argument from “git clone” to “git fetch”.
  _Fetch method will execute the following command to fetch
  superproject. Added --no-tags argument.

  master:  git fetch <remote url> --force --no-tags --filter blob:none
  branch:  git fetch <remote url> --force --no-tags --filter blob:none \
           <branch>:<branch>

+ Performance improvements for aosp-master
  ++ repo init performance improved from 35 seconds to 17 seconds.
  ++ repo init --use-superproject is around 5 to 7 secsonds slower.
  ++ repo sync --use-superproject is around 3 to 4 minutes faster.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL.

$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
  real	0m20.648s
  user	0m8.046s
  sys	0m3.271s

+ Without superproject
$ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main
  real	0m13.078s
  user	0m9.783s
  sys	0m2.528s

$ time repo_dev sync -c -j32 --use-superproject
...
  real	15m7.072s
  user	110m7.216s
  sys	20m17.559s

+ Without superproject
$ time repo sync -c -j32
...
  real	19m25.644s
  user	91m56.331s
  sys	20m59.170s

Bug: [google internal] b/180492484
Bug: [google internal] b/179470886
Bug: [google internal] b/180124069
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707

Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-25 20:45:26 +00:00
Mike Frysinger
adbd01e0d3 tests: fix init subcmd after url change
My recent 401c6f0725 ("init: make
--manifest-url flag optional") commit broke the unittest.

Change-Id: I19ad0e8c8cbb84ab5474ebc370e00acfe957e136
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298223
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 17:07:11 +00:00
Mike Frysinger
37ac3d626f tests: refactor manifest tests
The XmlManifestTests class is getting to be large and we're only
adding more to it.  Factor out the core logic into a new TestCase
so we can reuse it to better group more tests.

Change-Id: I5113444a4649a70ecfa8d83d3305959a953693f7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298222
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-25 17:06:56 +00:00
Mike Frysinger
057905fa1d error: fix pickling of all exceptions
Make sure all our custom exceptions can be pickled so that if they
get thrown in a multiprocess subprocess, we don't crash & hang due
to multiprocessing being unable to pickle+unpickle the exception.

Details/examples can be seen in Python reports like:
https://bugs.python.org/issue13751

Change-Id: Iddf14d3952ad4e2867cfc71891d6b6559130df4b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297382
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-19 20:06:03 +00:00
Raman Tenneti
21dce3d8b3 init: added --use-superproject option to clone superproject.
Added --no-use-superproject to repo and init.py to disable use of
manifest superprojects.

Replaced the term "sha" with "commit id".

Added _GetBranch method to Superproject object.

Moved shared code between init and sync into SyncSuperproject function.
This function either does git clone or git fetch. If git fetch fails
it does git clone.

Changed Superproject constructor to accept manifest, repodir and branch
to avoid passing them to multiple functions as argument.

Changed functions that were raising exceptions to return either True
or False.

Saved the --use-superproject option in config as repo.superproject.
Updated internal-fs-layout.md document.

Updated the tests to work with the new API changes in Superproject.

Performance for the first time sync has improved from 20 minutes to
around 15 minutes.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL.

$ repo init took around 20 seconds longer because of cloning of superproject.

$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real	0m35.919s
user	0m21.947s
sys	0m8.977s

First run
$ time repo sync --use-superproject
...
real	16m41.982s
user	100m6.916s
sys	19m18.753s

No difference in repo sync time after the first run.

Bug: [google internal] b/179090734
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707

Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-11 18:59:29 +00:00
Mike Frysinger
38867fb6d3 git_config: add SetBoolean helper
A little sugar simplifies the code a bit.

Change-Id: Ie2b8a965faa9f9ca05c7be479d03e8e073cd816d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296522
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-11 01:48:12 +00:00
Raman Tenneti
8d43dea6ea sync: pass --bare option when doing git clone of superproject.
Changed "git pull" to "git fetch" as we are using --bare option. Used the
following command to fetch:
  git fetch origin +refs/heads/*:refs/heads/* --prune

Pass --branch argument to Superproject's UpdateProjectsRevisionId function.

Returned False/None when directories don't exist instead of raise
GitError exception from _Fetch and _LsTree functions. The caller of Fetch
does Clone if Fetch fails.

Tested the code with the following commands.

$ ./run_tests -v

Tested the init and sync code by copying all the repo changes into my Android
AOSP checkout and running repo sync with --use-superproject option.

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-02-08 17:34:55 +00:00
Raman Tenneti
1fd7bc2438 sync: superproject performance changes.
After updating all project’s revsionIds with the SHAs from superproject,
write the updated manifest into superproject_override.xml file. Reload
that file for future Reloads. This file is created in exp-superproject
directory.

Moved most of the code that is superproject specific into
git_superproject.py and wrote test code.

If git pull fails, did a git clone of the superproject.

We saw performance gains for consecutive repo sync's. The time to sync
went down from around 120 secs to 40 secs when repo sync is executed
consecutively.

Tested the code with the following commands.

$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.

First run
$ time repo sync --use-superproject
...
real	21m3.745s
user	97m59.380s
sys	19m11.286s

After two consecutive sync runs
$ time repo sync -c -j8 --use-superproject
real	0m39.626s
user	0m29.937s
sys	0m38.155s

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>

Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-02-07 22:25:38 +00:00
Raman Tenneti
b5c5a5e068 manifest: set revisionId as revision attribute it it is not being set in ToXml.
As we were testing superproject setting revisionId attribute to SHA and
reloading the manifest, we found out revisionId attribute is not being
saved. Made the change to save the revisionId if it is not being saved.

Tested the code with the following commands.

$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I95fdf655b19648ad3e9aba10b9bed8bb9439deb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296182
Reviewed-by: Ian Kasprzak <iankaz@google.com>
2021-02-07 17:13:35 +00:00
Raman Tenneti
7caa3658b2 sync: Do a git pull with --use-superproject if superproject tree already exists.
Tested the code with the following commands.

$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.

.../WORKING_DIRECTORY$ repo sync --use-superproject

Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I7e4b1e51ca1d18b836a5fa8d139a0765262ba500
2021-02-01 12:24:51 -08:00
Raman Tenneti
6a872c9dae sync: Added --use-superproject option and support for superproject.
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v

Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,

.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-21 19:41:52 +00:00
Ian Kasprzak
febe73ff16 Update "evt" field to be logged as a string type.
Testing:
- Unit tests
- Verified git trace log has "evt": "2" (vs "evt": 2 previously)

Bug: https://crbug.com/gerrit/13966
Change-Id: I2e0c98dda0cccdd5cb6328105c11b93cd42676eb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294123
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Ian Kasprzak <iankaz@google.com>
2021-01-19 20:35:56 +00:00
Mike Frysinger
e5670c8812 launcher: add a requirements framework to declare version dependencies
Currently we don't have a way for the checked out repo version to
declare the version of tools it needs before we start running it.
For somethings, like git, it's not a big deal as it can handle all
the asserts itself.  But for things like Python, it's impossible
to reliably check before executing.

We're in this state now:
- we've been allowing Python 3.4, so the launcher accepts it
- the repo codebase starts using Python 3.6 features
- launcher tries to import us but hits syntax errors
- user is left confused and assuming new repo is broken because
  they're seeing syntax errors

This scenario is playing out with old launchers that still accept
Python 2, and will continue to play out as time goes on and we want
to require newer versions of Python 3.

Lets create a JSON file to declare all these system requirements.
That file format is extremely stable, so loading & parsing from
even ancient versions of Python shouldn't be a problem.  Then the
launcher can read these settings and check the system state before
attempting to execute any code.  If the tools are too old, it can
clearly diagnose & display information to the user as to the real
problem (and not emit tracebacks or syntax errors).

We have a couple of different tool version checks already (git,
python, ssh) and can harmonize them in a single place.

This also allows us to assert a reverse dependency if the need
ever comes up: force the user to upgrade their `repo` launcher
before we'll let them run us.  Even though the launcher warns
whenever a newer release is available, some users seem to ignore
that, or they don't use repo that often (on the scale of years),
and their upgrade jump is so dramatic that they fall back into
the syntax error pit.

Hopefully by the end of the year we can assume enough people
have upgraded their launcher such that we can delete all of the
duplicate version checks in the codebase.  But until then, we'll
keep them to maintain coverage.

Change-Id: I5c12bbffdfd0a8ce978f39aa7f4674026fe9f4f8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293003
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-19 16:48:21 +00:00
Raman Tenneti
48b2d10d8f manifest_xml: - Added doc and testing of unknown tags/elements.
Added this test to verify that older versions of repo can handle
"<superproject" element. Tested by adding "<iankaz" unknown element.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I858d56f38cefcfcd14474efdd631a5a940c3ce47
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293482
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-12 21:49:13 +00:00
Raman Tenneti
1bb4fb222d manifest_xml: initial support for <superproject>
At most one superproject may be specified. It will be used
to specify the URL of superproject.

It would have 3 attributes: remote, name, and default.
Only "name" is required while the others have reasonable defaults.

<remote name="superproject-url" review="<url>" />
<superproject remote="superproject-url" name="platform/superproject"/>

TODO: This CL only implements the parsing logic and further work
will be in followup CLs.

Tested the code with the following commands.

$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v

Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I5b4bba02c8b59601c754cf6b5e4d07a1e16ce167
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292982
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-08 19:49:52 +00:00
Ian Kasprzak
343d585ff9 Fix bug in git trace2 event Write() function when no config present.
See https://bugs.chromium.org/p/gerrit/issues/detail?id=13706#c9

Added additional unit tests for Write() for additional test coverage.

Testing:
- Unit tests
- Verified repo works with:
  - Valid trace2.eventtarget
  - Invalid trace2.eventtarget

Bug: https://crbug.com/gerrit/13706
Tested-by: Ian Kasprzak <iankaz@google.com>
Change-Id: I6b027cb2399bd03e453a132ad82e022a1f48476e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292762
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-01-07 14:31:51 +00:00
Mike Frysinger
acf63b2892 drop pyversion & is_python3 checking
We're committed to Python 3 at this point, so purge all the
is_python3 related dynamic checks.

Bug: https://crbug.com/gerrit/10418
Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:58 +00:00
Mike Frysinger
784ccfc040 strip python2-only coding:utf-8 & print_function settings
We're committed to Python 3 at this point, so clean up boilerplate.

Bug: https://crbug.com/gerrit/10418
Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:05 +00:00
Mike Frysinger
1379a9b185 launcher: add test for version requirements
Make sure the modules stay in sync in case one is updated but we
forgot to update the other.

Bug: https://crbug.com/gerrit/13795
Change-Id: I6de9533d45c083e5f7ad792ee6d541e23647de3f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292444
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 17:56:51 +00:00
Ian Kasprzak
30bc354e25 Enable git trace2 event format logging.
Ways to enable logging:
1) Set git's trace2.eventtarget config variable to desired logging path
2) Specify path via --git_trace2_event_log option

A unique logfile name is generated per repo execution (based on the repo session-id).

Testing:
1) Verified git config 'trace2.eventtarget' and flag enable logging.
2) Verified version/start/end events are expected format:
  https://git-scm.com/docs/api-trace2#_event_format
3) Unit tests

Bug: https://crbug.com/gerrit/13706
Change-Id: I335eba68124055321c4149979bec36ac16ef81eb
Tested-by: Ian Kasprzak <iankaz@google.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292262
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2021-01-06 17:49:41 +00:00
Mike Frysinger
51e39d536d manifest_xml: harmonize list fields
We allow project.groups to be whitespace or comma delimited, but
repo-hooks.enabled-list is only whitespace delimited.  This hasn't
been a big deal as it's only ever had one valid value, but if we
want to add more, we should harmonize these a bit.

Refactor the groups method to be more generic, and run the enabled-
list attribute through it.  Then add missing docs for it.

Change-Id: Iaa96a0faa9c4a68b313b49336751831b73bf855d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290743
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-12-04 17:27:11 +00:00
Fredrik de Groot
6342d56914 Fix tests after "use new main branch"
Tests worked fine if init.defaultBranch main was used,
but failed due to git branch reasons if master was still used.

Since we can only use init.defaultBranch if git version >= 2.28,
I also went with a template dir HEAD main tweak if lower so tests
now pass regardless of client git default branch and version.

Test: Ran tests with ~/.gitconfig:init.defaultBranch=master
Test: Ran tests with ~/.gitconfig:init.defaultBranch=main
Test: Ran tests for both code branches of git require

Change-Id: I49fa1e4ae45b8aec16a093132ee9fa466cbc11ec
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290404
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-12-03 07:29:59 +00:00
Fredrik de Groot
352c93b680 manifest: add support for groups in include
Attrib groups can now be added to manifest include, thus
all projects in an included manifest file can easily be tagged
with a group without modifying all projects in that manifest file.

Include groups will add and recurse, meaning included manifest
projects will carry all parent includes. Intentionally, no support
added for group remove, to keep complexity down.

Group handling for projects is untouched, meaning a group set on
a project will still append to whatever was or was not inherited
in parent manifest includes, resulting in union of groups inherited
and set for the project itself.

Test: manual multi-level manifest include structure, in serial and parallel,
      with different groups set on init
Test: added unit tests to cover the inheritance

Change-Id: Id2229aa6fd78d355ba598cc15c701b2ee71e5c6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/283587
Tested-by: Fredrik de Groot <fredrik.de.groot@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2020-11-26 09:13:14 +00:00
Mike Frysinger
8c1e9cbef1 manifest_xml: refactor manifest parsing from client management
We conflate the manifest & parsing logic with the management of the
repo client checkout in a single class.  This makes testing just one
part (the manifest parsing) hard as it requires a full checkout too.

Start splitting the two apart into separate classes to make it easy
to reason about & test.

Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-18 19:10:57 +00:00
Mike Frysinger
e283b95cf2 tests: use new main branch
Now that we clone "main" by default, use that for our local test.

Change-Id: Ib8420074bdfabfcb9d5252a3a0ecd3d852ca36e8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288422
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-11-17 04:29:09 +00:00
Remy Bohmer
16c1328fec Move RepoHook class from project.py file to dedicated file
The project.py file is huge and contains multiple
classes. By moving it to seperate class files the code
becomes more readable and maintainable.

Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: Ida9d99d31751d627ae1ea0373418080696d2e14b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/281293
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Remy Bohmer <linux@bohmer.net>
2020-11-03 22:08:08 +00:00
Mike Frysinger
0578ebf61a init: reject unknown args
If you pass args to `repo init` when first creating a checkout, the
repo launcher throws an error.  But the init subcommand that runs in
an existing checkout silently ignores them.  Throw a proper error.

Change-Id: I433bfcc73902d25f6b6a2974e77f6a977a75ed16
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279696
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-09-02 07:53:16 +00:00
Anders Björklund
8e0fe1920e Use hash for ControlPath instead of full variables
The generated socket path can be too long, if your FQDN is very long...

Typical error message from ssh client:
unix_listener: path "/tmp/ssh-fqduawon/master-USER@HOST:PORT.qfCZ51OAZgTzVLbg" too long for Unix domain socket

Use a hashed version instead, to keep within the socket file path limit.

This requires OpenSSH_6.7p1, or later.

Change-Id: Ia4bb9ae8aac6c4ee31d5a458f917f3753f40001b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255632
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Anders Björklund <anders.bjorklund.2@volvocars.com>
2020-04-15 06:51:22 +00:00
Marcos Marado
2735bfc5ff tests: fix SetupGnuPG test
The SetupGnuPG test tries to test the full setup, including the
creation of the directories. In order to do that, it create a
temporary directory, and redefines the home_dot_repo to point there.

When a home_dot_repo directory does not exist, it should be created.
The gpg_dir, which should exist inside home_dot_repo, also needs to be
created if it does not exist. However, since the gpg_dir path is
relative to home_dot_repo, once we redefine one, we need to redifine
the other.

The failure of this test might have gone unnoticed so far, since in
only fails if you do not have a ~/.repoconfig/gnupg/ on the
environment you are running the tests on.

Change-Id: Ic69d59e56137eea43349a61b5cf81f215c6a7f9a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/262573
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Marcos Marado <mindboosternoori@gmail.com>
2020-04-12 17:12:14 +00:00
Mike Frysinger
3599cc3975 init: respect --repo-rev changes
We respect this option when running the first `repo init`, but then
silently ignore it once the initial sync is done.  Make sure users
are able to change things on the fly.

We refactor the wrapper API to allow reuse between the two init's.

Bug: https://crbug.com/gerrit/11045
Change-Id: Icb89a8cddca32f39a760a6283152457810b2392d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260032
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-03-25 04:55:50 +00:00
Mike Frysinger
cfc8111f5e init: allow REPO_REV/--repo-rev to specify commits/tags
While the help/usage suggested that revisions would work, they never
actually did, and just throw confusing errors.  Now that we warn if
the checkout isn't tracking a branch, allow people to specify commits
or tags explicitly.  Hopefully our nags will be sufficient to keep
most people on the right path.

Bug: https://crbug.com/gerrit/11045
Change-Id: I6ea32c677912185f55ab20faaa23c6c0a4c483b3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259492
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
2020-03-24 05:01:23 +00:00