Commit Graph

138 Commits

Author SHA1 Message Date
Gavin Mak
a56e0e17e2 tests: Change docstring for CopyLinkTestCase
Change-Id: Ic31b8073090abffe4e90cd208b684e99b83d7ef2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/358455
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2023-02-14 21:48:41 +00:00
Joanna Wang
016a25447f git_superproject: Log actual error fmt instead of the entire error message.
Bug: b/258492341
Change-Id: I00678d572712791190ae1ad4e1bcf3cbe04cc1c0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357114
Tested-by: Joanna Wang <jojwang@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2023-02-01 22:32:56 +00:00
Daniel Kutik
035f22abec pylint: remove unused imports
Removed unused imports accross multiple files.

Change-Id: Ib5ae4cebf9660e7339b11e3fa592d99f8d51e8d8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354700
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-12-13 16:23:19 +00:00
Mike Frysinger
e0728a5ecd update-manpages: clean up symlink in checkout
We don't want symlinks in the git tree as it causes pain for Windows
users.  We also don't really need it as we can refactor the code we
want to import slightly.

Change-Id: I4537c07c50ee9449e9f53e0f132a386e8ffe16ec
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354356
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-12-12 23:04:40 +00:00
Joanna Wang
0324e43242 repo_trace: Avoid race conditions with trace_file updating.
Change-Id: I0bc1bb3c8f60465dc6bee5081688a9f163dd8cf8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354515
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Joanna Wang <jojwang@google.com>
2022-12-09 22:49:31 +00:00
Joanna Wang
0e4f1e7fba Use --negotiation-tip in superproject fetches.
Bug: b/260645739
Change-Id: Ib0cdbb13f130b91ab14df9c60a510f1e27cca8e0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354354
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Joanna Wang <jojwang@google.com>
2022-12-09 14:25:15 +00:00
Mike Frysinger
e815286492 tests: clean up repo_trace._TRACE_FILE patching
Patch this automatically for all tests rather than duplicating the
boilerplate in diff testcases.

Change-Id: I391d5c859974cda3d5680d34ede2ce6e9e925838
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354358
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-12-08 22:22:39 +00:00
Mike Frysinger
0ab6b11688 wrapper: switch to functools.lru_cache
No need to implement our own caching logic with newer Python.

Change-Id: Idc3243b8e22ff020817b0a4f18c9b86b1222d631
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354357
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2022-12-08 22:22:36 +00:00
Mike Frysinger
a621254b26 tests: drop old unittest.main logic
We use pytest now which doesn't need this boilerplate.

Change-Id: Ib71d90b3f1669897814ee768927b5b595ca8d789
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354355
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-12-08 17:17:30 +00:00
Mike Frysinger
f159ce0f9e sync: fix manifest sync-j handling
Since --jobs defaults to 0, not None, we never pull the value out
of the manifest.  Treat values of 0 and None the same to fix.

Bug: http://b/239712300
Bug: http://b/260908907
Change-Id: I9b1026682072366616825fd72f90bd90c10a252f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354254
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
2022-12-08 15:06:24 +00:00
Daniel Kutik
8051cdb629 test_manifest_config_properties: use assertEqual
The method assertEquals is an deprecated alias for
assertEqual.
See: https://docs.python.org/3/library/unittest.html#deprecated-aliases

Change-Id: Id94ba6d6055bdc18b87c53e8729902bb278855aa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354035
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
2022-12-05 13:09:09 +00:00
Joanna Wang
ea5239ddd9 Fix ManifestProject.partial_clone_exclude property.
Bug: b/256358360

Change-Id: Ic6e3a049aa38827123d0324c8b14157562c5986e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353574
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Joanna Wang <jojwang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2022-12-02 14:57:56 +00:00
Daniel Kutik
35af2f8daf Fixed wrapper related warnings in tests
Multiple "Could not find reference" warnings in test_wrapper.py
and test_git_command.py resolved.

Test: tox
Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: Ic254c378bbdae6bc3f8f29682ababb37db76adfe
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353235
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-11-28 13:00:34 +00:00
Daniel Kutik
e287fa760b test_capture: allow both Unix and Windows line sep
On Linux/macOS we allow \n in the end of the line.
On Windows we allow both \r\n and \n. Here we also allow Unix line
seperators as tests might be excuted in for example git-shell.

Change-Id: I3975b563cf95407da92e5479980e670eb748b30e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353181
Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-11-28 06:39:31 +00:00
Daniel Kutik
3593a10643 test_bad_path_name_checks: allow Windows path sep
With this change if a path ends with '/' on Linux/macOS
and ends with either '/' or '\' on Windows, the test will pass.

Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: Id7d1b134f9c0bdf7ceaf149af304bbf90cbd7b21
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353180
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-11-28 04:07:17 +00:00
Daniel Kutik
003684b6e5 test: Fix char encoding issues on windows
Some tests were failing due to Windows not using utf-8
by default when executing the tests. Enforcing usage of utf-8
resolves these issues.

Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: If42f6be2a2b688a6105ecf4fcdb541aade24519a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353179
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-11-27 17:03:41 +00:00
Daniel Kutik
0297f8312c test: fix path seperator errors on windows
Fixing multiple errors when running tests on Windows related
to path seperator being different ('\' instead of '/').

Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: I26b44d092b925edecab46a4d88e77dd9dcb8df28
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353178
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-11-27 17:03:01 +00:00
Daniel Kutik
eda6b1ead7 trace: make test timeout after 2min
Before this commit, the test was hanging forever when
run on a Windows host. This should resolve that issue.

Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: Id9ea6d54926b797db3d2978a2ae2930088201eec
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353125
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-11-26 23:30:43 +00:00
Sam Saccone
d686365449 Extract env building into a testable helper.
Previously env dict building was untested and mixed with other mutative
actions. Extract the dict building into a dedicated function and author
tests to ensure the functionality is working as expected.

BUG: b/255376186
BUG: https://crbug.com/gerrit/16247
Change-Id: I0c88e53eb285c5c3fb27f8e6b3a903aedb8e02a8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/351874
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Sam Saccone <samccone@google.com>
2022-11-16 18:26:49 +00:00
LaMont Jones
fa90f7a36f tests: Fix update-manpages test.
Change-Id: I58d85e06edeb9208a782957acc982e996c026ed2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/351854
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-11-16 16:53:49 +00:00
LaMont Jones
f8af33c9f0 update-manpages: explicitly strip color codes
On some systems, help2man produces color codes in the output.  Remove
them to avoid manpage churn.

Also begin adding unit tests.

Change-Id: I3f0204b19d9cae524d3cb5fcfb61ee309b0931fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349655
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2022-11-14 23:46:43 +00:00
LaMont Jones
fa8d939c8f sync: clear preciousObjects when set in error.
If this is a project that is not using object sharing (there is only one
copy of the remote project) then clear preciousObjects.

To override this for a project, run:

  git config --replace-all repo.preservePreciousObjects true

Change-Id: If3ea061c631c5ecd44ead84f68576012e2c7405c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350235
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-11-03 23:01:16 +00:00
Joanna Wang
a6c52f566a Set tracing to always on and save to .repo/TRACE_FILE.
- add `--trace_to_stderr` option so stderr will include trace outputs and any other errors that get sent to stderr
- while TRACE_FILE will only include trace outputs

piggy-backing on: https://gerrit-review.googlesource.com/c/git-repo/+/349154

Change-Id: I3895a84de4b2784f17fac4325521cd5e72e645e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350114
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Joanna Wang <jojwang@google.com>
2022-11-03 21:07:07 +00:00
Peter Kjellerstedt
0d130d2da0 tests: Make the tests pass for Python < 3.8
Before Python 3.8, xml.dom.minidom sorted the attributes of an element
when writing it to a file, while later versions output the attributes
in the order they were created. Avoid these differences by sorting the
attributes for each element before comparing the generated manifests
with the expected ones.

This corresponds to commit 5d58c18, but for new tests introduced since
it was integrated.

Change-Id: I5c360656a0968e6e8d57eb068c8e87da7dfa61c1
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349917
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-10-28 17:26:48 +00:00
Erik Elmeke
4cdfdb7734 manifest: allow extend-project to override dest-branch and upstream
Bug: https://crbug.com/gerrit/16238
Change-Id: Id6eff34791525b3df690e160c911c0286331984b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/345144
Tested-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-09-20 04:25:02 +00:00
LaMont Jones
a8cf575d68 Omit local_manifest groups from superproject override.
When we create superproject_override.xml, do not include projects that
are present from local_manifests/*.  Such projects are fully under the
control of the local_manifests/ file.

Bug: b/238934278
Test: manual, ./run_tests
Change-Id: I40382ceb82d9cf7b8dc7b5f2abed3f6d4d80017e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340877
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Sam Saccone 🐐 <samccone@google.com>
2022-07-15 23:32:24 +00:00
Mike Frysinger
74737da1ab tests: switch to tempfile.TemporaryDirectory
Now that we don't need to support Python 2, we can switch to this
API for better contextmanager logic.

Change-Id: I2d03e391121886547e7808a3b5c3b470c411533f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337515
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-05-20 11:38:10 +00:00
LaMont Jones
a46047a822 sync: refactor use of self.manifest
We need to iterate over multiple manifests, and generally use the
outer_client.manifest for multi-manifest support.  This refactors the
use of self.manifest into a chosen manifest.

Change-Id: I992f21d610c929675e99555ece9c38df4b635839
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334699
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-14 22:24:04 +00:00
LaMont Jones
d56e2eb421 manifest_xml: use Superproject to hold XML content
Always create Superproject when there is a <superproject> tag, and have
it hold the XML content, similar to how other manifest elements are
handled.

This also adds SetQuiet and SetPrintMessages to Superproject
consistent with manifest.SetUseLocalManifests.

Change-Id: I522bf3da542006575799f0640c67f7052704f266
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334641
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-04-12 15:46:23 +00:00
Daniel Andersson
d52ca421d5 sync: respect sync-c manifest option
The documentation states that a `sync-c` attribute in the manifest file
can set a default for whether only the current branch should be fetched
or all branches. This seems to have been broken for some time.

Commit 7356114 introduced the `--no-current-branch` CLI option and
relied on getting `None` via `optparse` if neither `--current-branch`
nor `--no-current-branch` was set to distinguish it from a boolean
value. If `None` was received, it would read the value from the manifest
option `sync-c`. The parsing went through the utility function
`_GetCurrentBranchOnly` which returned `True` if `--current-branch` had
been given on the command-line, or fell back on the "superproject"
setting, which would either return `True` or `None`. This would
incorrectly make `repo` fall back to the manifest setting even if the
user had given `--no-current-branch` if no superproject was requested --
the manifest became "too powerful":

Command-line         Using superproject  → `current_branch_only`
------------         ------------------  -----------------------
                     No                  From manifest
                     Yes                 True
--current-branch     No                  True
--current-branch     Yes                 True
--no-current-branch  No                  From manifest ← wrong
--no-current-branch  Yes                 True

In commit 0cb6e92 the superproject configuration value reading changed
from something that could return `None` to something that always
returned a boolean. If it returned `False`, this would then incorrectly
make `repo` ignore the manifest option even if neither
`--current-branch` nor `--no-current-branch` had been given. The
manifest default became useless:

Command-line         Using superproject  → `current_branch_only`
------------         ------------------  -----------------------
                     No                  False ← wrong
                     Yes                 True
--current-branch     No                  True
--current-branch     Yes                 True
--no-current-branch  No                  False
--no-current-branch  Yes                 True

By swapping the order in which the command-line option target and the
superproject setting is evaluated, things should work as documented:

Command-line         Using superproject  → `current_branch_only`
------------         ------------------  -----------------------
                     No                  From manifest
                     Yes                 True
--current-branch     No                  True
--current-branch     Yes                 True
--no-current-branch  No                  False
--no-current-branch  Yes                 True

Change-Id: I933c232d2fbecc6b9bdc364ebac181798bce9175
Tested-by: Daniel Andersson <daniel.r.andersson@volvocars.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334270
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-08 21:06:37 +00:00
Josh Steadmon
244c9a71a6 trace: allow writing traces to a socket
Git can write trace2 events to a Unix domain socket [1]. This can be
specified via Git's `trace2.eventTarget` config option, which we read to
determine where to log our own trace2 events. Currently, if the Git
config specifies a socket as the trace2 target, we fail to log any
traces.

Fix this by adding support for writing to a Unix domain socket,
following the same specification that Git supports.

[1]: https://git-scm.com/docs/api-trace2#_enabling_a_target

Change-Id: I928bc22ba04fba603a9132eb055141845fa48ab2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/332339
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Josh Steadmon <steadmon@google.com>
2022-03-16 17:33:07 +00:00
Sebastian Wagner
a3ac816278 test_project: use os.readlink instead of Path.readlink
Path.readlink is only available on Python 3.9, breaking compatibility
with all python versions below. os.readlink is already used in other
places of this file, so use it here as well.

Change-Id: I5acf8f5334a3e7c8de9cea1939d7e2b9af5f30ae
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327844
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sebastian Wagner <sebix@sebix.at>
2022-01-11 20:16:42 +00:00
Mike Frysinger
89ed8acdbe project: abort a bit earlier before migrating .git/
Verify all the .git/ paths will be handled by the migration logic before
starting the migration.  This way we still abort & log an error, but the
user gets to see it before we put the tree into a state that they have to
manually recover.  Also add a few more known-safe-to-clobber paths.

Bug: https://crbug.com/gerrit/15273
Change-Id: If49d69b341bc960ddcafa30da333fb5ec7145b51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327557
Reviewed-by: Colin Cross <ccross@android.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-07 20:17:14 +00:00
Mike Frysinger
2a089cfee4 project: migrate worktree .git/ dirs to symlinks
Historically we created a .git/ subdir in each source checkout and
symlinked individual files to the .repo/projects/ paths.  This layer
of indirection isn't actually needed: the .repo/projects/ paths are
guaranteed to only ever have a 1-to-1 mapping with the actual git
checkout.  So we don't need to worry about having files in .git/ be
isolated.

To that end, change how we manage the actual project checkouts from
a dir full of symlinks (and a few files) to a symlink to the internal
.repo/projects/ dir.  This makes the code simpler & faster.

The directory structure we have today is:
.repo/
  project-objects/chromiumos/third_party/kernel.git/
    <paths omitted as not relevant to this change>
  projects/src/third_party/kernel/
    v3.8.git/
      config
      description   -> …/project-objects/…/config
      FETCH_HEAD
      HEAD
      hooks/        -> …/project-objects/…/hooks/
      info/         -> …/project-objects/…/info/
      logs/
      objects/      -> …/project-objects/…/objects/
      packed-refs
      refs/
      rr-cache/     -> …/project-objects/…/rr-cache/
src/third_party/kernel/
  v3.8/
    .git/
      config        -> …/projects/…/v3.8.git/config
      description   -> …/project-objects/…/v3.8.git/description
      HEAD
      hooks/        -> …/project-objects/…/v3.8.git/hooks/
      index
      info/         -> …/project-objects/…/v3.8.git/info/
      logs/         -> …/projects/…/v3.8.git/logs/
      objects/      -> …/project-objects/…/v3.8.git/objects/
      packed-refs   -> …/projects/…/v3.8.git/packed-refs
      refs/         -> …/projects/…/v3.8.git/refs/
      rr-cache/     -> …/project-objects/…/v3.8.git/rr-cache/

The directory structure we have after this commit:
.repo/
  <nothing changes>
src/third_party/kernel/
  v3.8/
    .git            -> …/projects/…/v3.8.git

Bug: https://crbug.com/gerrit/15273
Change-Id: I9dd8def23fbfb2f4cb209a93f8b1b2b24002a444
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323695
Reviewed-by: Mike Nichols <mikenichols@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-12-01 15:27:16 +00:00
Raman Tenneti
a0f6006ae7 git_config: Fixed test.gitconfig getting updated when running tests.
Moved test_GetSyncAnalysisStateData to GitConfigReadWriteTests class.

Deleted [repo "syncstate*..] data from tests/fixtures/test.gitconfig.

Tested:
./run_tests
...
tests/test_git_config.py::GitConfigReadWriteTests::test_GetSyncAnalysisStateData PASSED [ 84%]
...

Bug: https://crbug.com/gerrit/15103
Change-Id: I8cb89ce10b025994a045106c9c66dd243ae8ba50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319557
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-30 21:45:09 +00:00
Raman Tenneti
4f88206178 trace2_event: Add remove_prefix to fix failing tests on Linux & macOS.
removeprefix is available i python 3.9. Mac and Linux are running in
a version below 3.9. Thus tests are failing with the following error:
  "AttributeError: 'str' object has no attribute 'removeprefix' "

Replaced the removeprefix with custom function which we will delete
once Linux and macOS versions are updated.

Tested:
$ ./run_tests

Bug: [google internal] b/201453085
Change-Id: I9b4d564ff1176e1b4471805ef05472c1914cd9f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319375
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-29 21:24:59 +00:00
Michael Kelly
37c21c268b Add 'dest-path' to extend-project to support changing path
This allows us to move the repository to a new location in the source
tree without having to remove-project + add a new project tag.

Change-Id: I4dba6151842e57f6f2b8fe60cda260ecea68b7b4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310962
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
2021-09-28 20:12:00 +00:00
Raman Tenneti
bbe8836494 superproject: Log syncstate's parameter as data-json it it is an array.
All the values of syncstate are strings, check the first byte and last
byte to see if it is an array. For syncstate data, there were no false
positives.

Tested:
$ repo_dev sync

Verified event logged for argv is "data-json".

$./run_tests

Bug: [google internal] b/201102002
Change-Id: Id56adb532b80267f08d09147ac663cdd5987ce87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319075
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-09-28 18:22:49 +00:00
Mike Frysinger
9d96f58f5f make file removal a bit more robust
Some of the file removal calls are subject to race conditions (if
something else deletes the file), so extend our remove API to have
an option to ignore ENOENT errors.  Then update a bunch of random
call sites to use this new functionality.

Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195
Reviewed-by: Sean McAllister <smcallis@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-09-28 16:06:50 +00:00
Raman Tenneti
956f7363d1 superproject: Log argv parameter of syncstate as 'data-json'.
Fixed: "we need to make a special case for logging the argv; it
should probably be a "data-json" event so that we log this directly as
an array rather than an encoded string.

Tested:
$ repo_dev sync

Verified event logged for argv is "data-json".

$./run_tests

Bug: [google internal] b/201102002
Change-Id: I18ccec79c73c8dc931cb8afc472b2361db8aea4c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319055
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-09-27 19:11:14 +00:00
Xin Li
e0b16a22a0 superproject: support a new revision attribute.
Tested:
$ ./run_tests

Verified that a manifest that specified superproject revision would use
the specified revision, and superproject will use the default revision.

Note that this is a slight behavior change from earlier repo versions,
which would always use the branch name of the manifest itself. However,
the new behavior would be more consisitent with regular "project"
element and would allow superproject be used even if it is not enabled
for the particular manifest branch, so we have decided to make the
change as it would provide more flexibility and better matches what
other elements would do.

Bug: [google internal] b/187868160
Change-Id: I35255ee347aff6e65179f7879d52931f168b477e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317643
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-27 06:36:05 +00:00
Jack Neus
a84f43a006 manifest: make repo-hooks more robust wrt element ordering
Currently, repo will fail to sync to a manifest if the definition
of the repo-hooks project comes after the repo-hooks element.

BUG=none
TEST=new test, run_tests

Change-Id: I0bf85625173492af6c6404d4b67543e96e670562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318520
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
2021-09-23 21:17:38 +00:00
Raman Tenneti
6448a4f2af sync: Log repo sync state events as 'data' events.
git_trace2_event_log.py:
+ Added LogDataConfigEvents method to log 'data' events.
  Sync's current_sync_state and previous_sync_state are logged
  as 'data' events in the current log.

  It logs are key/value in the |config| argument. Each key is
  prefixed with |prefix| argument.

  The following are sample events that are logged during repo sync.

   {"event":"data",
   "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
   "thread":"MainThread",
   "time":"2021-09-14T18:16:19.935846Z",
   "key":"previous_sync_state/repo.syncstate.main.synctime",
   "value":"2021-09-14T17:27:11.573717Z"}

   {"event":"data",
   "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
   "thread":"MainThread",
   "time":"2021-09-14T18:16:19.955546Z",
   "key":"current_sync_state/repo.syncstate.main.synctime",
   "value":"2021-09-14T18:16:19.935979Z"}

tests/test_git_trace2_event_log.py:
+ Added unit tests

sync.py:
+ Changed logging calls to LogDataConfigEvents.

Tested:
$ ./run_tests

Tested it by running the following command multiple times.
$ repo_dev sync -j 20
  repo sync has finished successfully

  Verified config data is looged in trace2 event logs.

Bug: [google internal] b/199758376
Change-Id: I75fd830e90c1811ec28510538c99a2632b104e85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317823
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-09-14 21:36:12 +00:00
Raman Tenneti
b55769a5c9 superproject: print messages if the manifest has superproject tag.
1) If the manifest has superproject tag (git_master, etc), then
   display error/warning messages (as it is doing today)
2) If the manifest doesn't have superproject tag (nest, chromeos
   manifests), then don't display any error/warning messages about
   superrproject (behave as though user has specified
   --no-use-superproject).
3) Print error/warning messages if --use-superproject passed as
   argument to repo sync.
4) No change in behavior for the repo init command.

git_superproject.py:
+ Fixed typo in _WriteManifestFile method name
+ Superproject accepts print_message  as an argument and it defaults
  to True. All messages that are printed to stderr are controlled by
  this flag. If it is True, then messages get printed.
+ Added PrintMessages function which return true if either
  --use-superproject is specified on the command line or if the
  manifest has a superproject tag.

sync.py:
+ Displays the warning message if PrintMessgages are enabled and
  passes that as argument to superproject object.
+ Added 'hassuperprojecttag' trace2 log entry for analysis. We can
  find users/branches that are using superproject, but the manifest is
  missing the superproject tag.

Tested:
$ ./run_tests

+ Verified printing of messages with and without superproject tag, with
  with --use-superproject option.

+ aosp-master
  $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
  $ repo_dev sync

+ A manifest without superproject tag.
  $ repo_dev init -m $(pwd)/manifest_7482982.xml
  $ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml

Bug: [google internal] b/196411099
Change-Id: I92166dcad15a4129fab82edcf869e7c8db3efd4b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314982
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-08-13 20:07:40 +00:00
Raman Tenneti
5637afcc60 superproject: prepend messages with - "repo superproject"
Changed _LogError method to _LogWarning.

Replaced 'repo error:' with "repo superproject warning:"(except IOError
message, which is still an "repo superproject error:" message)

Tested:
$ ./run_tests

Tested the errors and warnings by forcing the error/warning.
$ repo_dev sync -j 20 --use-superproject platform/packages/apps/Music
  ...
  repo superproject warning: please file a bug using go/repo-bug to report missing commit_ids for: []
  ...
  repo superproject error: cannot write manifest to : /sdc/android/src/aosp/.repo/exp-superproject/superproject_override.xml
  ...

Bug: [google internal] b/193711236

Change-Id: Ia0b6c830e04cf18dfc1a2ce325181a5b1160e054
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314642
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2021-08-12 16:30:26 +00:00
Raman Tenneti
9122bfc3a8 sync: Remove '_' from the repo.syncstate.* keys when saved to config.
GitConfig doesn't save keys if the keys contain "_" characters. Some
of the options like mp_update, use_superproject have underscores.

This fixes issue with previous_sync_state missing some of the options.

Tested:
$ ./run_tests

$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest

Tested it by running the sync command multiple times and verifing
previous_sync_state and current_sync_state have the same keys.

$ repo_dev sync -j 20
  repo sync has finished successfully

  Verified config file has [syncstate ...] data saved.

Bug: [google internal] b/188573450
Change-Id: I16b52a164f9dd1633d7dad1d8cf6b151c629fcb1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313242
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-07-29 22:41:57 +00:00
Raman Tenneti
7954de13b7 sync: Added logging of repo sync state and config options for analysis.
git_config.py:
+ Added SyncAnalysisState class, which saves the following data
  into the config object.
  ++ sys.argv, options, superproject's logging data.
  ++ repo.*, branch.* and remote.* parameters from config object.
  ++ current time as synctime.
  ++ Version number of the object.
+ All the keys for the above data are prepended with 'repo.syncstate.'
+ Added GetSyncAnalysisStateData and UpdateSyncAnalysisState methods
  to GitConfig object to save/get the above data.

git_trace2_event_log.py:
+ Added LogConfigEvents method with code from DefParamRepoEvents
  to log events.

sync.py:
+ superproject_logging_data is a dictionary that collects all the
  superproject data that is to be logged as trace2 event.
+ Sync at the end logs the previously saved syncstate.* parameters
  as previous_sync_state. Then it calls config's UpdateSyncAnalysisState
  to save and log all the current options, superproject logged data.

docs/internal-fs-layout.md:
+ Added doc string explaining [repo.syncstate ...] sections of
  .repo/manifests.git/config file.

test_git_config.py:
+ Added unit test for the new methods of GitConfig object.

Tested:
$ ./run_tests

$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest

Tested it by running the following command multiple times.
$ repo_dev sync -j 20
  repo sync has finished successfully

  Verified config file has [syncstate ...] data saved.

Bug: [google internal] b/188573450
Change-Id: I1f914ce50f3382111b72940ca56de7c41b53d460
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313123
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2021-07-29 19:20:57 +00:00
Jack Neus
6ea0caea86 repo: properly handle remote annotations in manifest_xml
BUG=b:192664812
TEST=tests/

Change-Id: I1aa50260f4a00d3cebbd531141e1626825e70127
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312643
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-07-23 18:03:11 +00:00
Michael Kelly
06da9987f6 Gracefully ignore bad remove-project line
Sometimes, we don't care if the remove project is referring to a
non-existing project and we can just ignore it.  This change allows us
to ignore remove-project entries if the project that they refer to
doesn't exist, making them effectively a no-op.

Because this change breaks existing configuration, we allow this to be
configuration controlled using the `optional` attribute in the
remove-project tag.

Change-Id: I6313a02983e81344eadcb4e47d7d6b037ee7420e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310964
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-07-08 16:48:21 +00:00
Xin Li
0e776a5837 Fix an issue when syncing with --use-superproject and clone bundles.
It is possible that a clone bundle contained the object referenced by
the branch in the manifest and in the superproject, but not the branch
itself (for example, the branch may be newly created from an existing
branch, or is not vislble to the user downloading the clone bundle).

When --use-superproject is enabled, because we are overriding
revisionExpr with the SHA1 revision provided by the superproject, the
verification step would succeed, but because the expected branch do not
exist, it would confuse git-repo at a later time, as it is expecting the
remote branch to exist in the local clone.

In project.py, fix this by making SetRevisionId() to always remember
the actual branch name and verify it in _CheckForImmutableRevision()
so that we only skip the fetch step when both objects exists locally.

Bug: [google internal] b/191974277
Change-Id: I49d3ca0667f524c8c45f416492faf95b1dd822fb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
2021-06-30 15:31:15 +00:00