Commit Graph

1912 Commits

Author SHA1 Message Date
Mike Frysinger
0531a623e1 sync: make --prune the default
If a remote deletes a ref, and it points to an object that doesn't
exist locally, we can get into a bad state, and the only way for the
user to recover is to run `repo sync --prune` (and to know that is
the option they need).  The error message is not helpful:

fatal: bad object refs/remotes/cros/firmware-zork-13421.B-master
error: https://chromium.googlesource.com/chromiumos/platform/ec did not send all necessary objects

This situation can also come up when the remote renames refs in a
UNIX FS incompatible way.  For example, replacing refs/heads/foo
with refs/heads/foo/bar.

Also add a --no-prune option for users to disable the behavior.

Bug: https://issuetracker.google.com/203366450
Change-Id: Icf45d838a10938feb091d29800f7e49240830ec3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322956
Reviewed-by: Andrew Lamb <andrewlamb@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-11-05 20:13:30 +00:00
Mike Frysinger
2273f46cb3 sync: fix --tags option
This has been broken since it was added where --tags was actually
the same as --no-tags.  Oddly, it was copied from init where the
logic is correct.

Bug: https://crbug.com/gerrit/12401
Change-Id: I15b89da1a655176a11bebc22573b25c728055328
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322955
Reviewed-by: Andrew Lamb <andrewlamb@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-11-05 20:13:02 +00:00
LaMont Jones
7b9b251a5e project: fix format string in error message
BUG=None

Change-Id: I0b195fd919c6db8cb3547e8d6f4c733f2bd4a535
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322735
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2021-11-05 16:59:14 +00:00
Raman Tenneti
6251729cb4 superproject: added 'implies -c' in the help of --use-superproject option.
sync.py: deleted unused import errno.

Tested:
$ ./run_tests
$ flake8 repo subcmds/sync.py

Bug: https://crbug.com/gerrit/15208
Change-Id: I2bb3098f5602ded3861e000100766041ad93b53d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322555
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-11-01 22:15:53 +00:00
Matt Story
11b30b91df Support more url schemes for getting standalone manifest
urllib.requests.urlopen also supports file, so call it unless the
scheme is 'gs'.  This adds http, https, and ftp support.

Change-Id: I3f215c3ebd8e6dee29ba14c7e79ed99d37287109
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322095
Reviewed-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Matt Story <mstory@arista.com>
2021-10-27 13:20:35 +00:00
Jack Neus
198838599c fetch: Fix stderr handling for gsutil
Previously gsutil stderr was getting piped into stdout, which
yields bad results if there are non-fatal warnings in stderr.

Additionally, we should fail outright if gsutil fails (by adding
`check = True`) rather than fail later on when we try to sync to
a manifest that is in fact just a stderr dump.

BUG=none
TEST=manual runs with bad gs urls

Change-Id: Id71791d0c3f180bd0601ef2c783a8e8e4afa8f59
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/321935
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-26 22:18:28 +00:00
Mike Frysinger
282d0cae89 ssh: handle FileNotFoundError errors
If ssh isn't installed, it throws a distinct error we have to catch.

Bug: https://crbug.com/gerrit/15196
Change-Id: I0660e842c304ce7575f5cb100894d05fd65f9454
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322055
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-10-26 16:18:45 +00:00
Jack Neus
03ff276cd7 sync: properly handle standalone manifests for sync command
sync should not attempt to sync the manifest project if it was
created from a standalone manifest. The current work around is to
run sync with --nmu.

BUG=none
TEST=manual runs

Change-Id: I2e121af0badf9642143e77c7af89d1c2d993b0f3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/321195
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-15 17:20:00 +00:00
Allen Webb
4ee4a45d03 subcmds/sync: Use pack-refs instead of gc for redundant gitdirs.
Previously `git gc` was being run on every gitdir even when they shared
the same objects. Instead only call it once and use pack-refs for the
gitdirs that were not gc'ed.

Bug: https://crbug.com/gerrit/15113
Test: repo sync -j # and check that git pack-refs is called
Change-Id: Icff37ab3ec78cfb44391d8cc7f2d875991532320
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/320275
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-14 12:27:12 +00:00
Jack Neus
0f6f16ed17 repo: more arg checking for --standalone-manifest re-inits
`repo init` doesn't do anything on re-init when the checkout has
been initialized using --standalone manifest. Rather than let the
tool run through its existing flows (which happen to noop), check
the args and explicitly quit if a bare `repo init` is run on a
standalone checkout.

BUG=none
TEST=manual tests

Change-Id: Ie4346ef6df1282ec3e3f8045a08138c93653fece
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/320735
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-11 18:58:11 +00:00
Jack Neus
76491590b8 repo: fix bug with --standalone-manifest
We were accidentally always setting manifest.standlone in config,
which was messing up behavior for standard use cases.

BUG=gerrit:15160
TEST=manual runs

Change-Id: Ic80f084ae97de5721aced3bb52d3ea9115f8d833
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/320715
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-11 18:57:57 +00:00
Mike Frysinger
6a74c91f50 sign-launcher: make the help text more automatic
Rather than display "3.0" all the time and confuse people, extract
the version from the launcher we're signing and display that.

Also reformat the text to follow our current practice: upload the
versioned launcher by itself first, and then later copy that over
the default.

And while we're here, add tips for rollbacks.

Change-Id: I1654425c88e5c67d78879f2f33ad685c59be14dc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319637
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-10-06 17:02:56 +00:00
Allen Webb
669efd0fd7 subcmds/sync: Disable autoDetach for git gc.
gc.autoDetach is enabled by default which makes 'git gc --auto' return
immediately and run in background. This can lead to a pile up of
operations all using large amounts of memory at the same time. To avoid
this set gc.autoDetach to false so that the garbage collect task waits
for instances to finish before spawning more.

Bug: https://crbug.com/gerrit/15113
Test: repo sync -j # and check the number of 'git gc' processes
Change-Id: Ic0815156ba3db03972968f33f6f9f51e4928f23b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319835
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-10-05 14:12:01 +00:00
Raman Tenneti
a0f6006ae7 git_config: Fixed test.gitconfig getting updated when running tests.
Moved test_GetSyncAnalysisStateData to GitConfigReadWriteTests class.

Deleted [repo "syncstate*..] data from tests/fixtures/test.gitconfig.

Tested:
./run_tests
...
tests/test_git_config.py::GitConfigReadWriteTests::test_GetSyncAnalysisStateData PASSED [ 84%]
...

Bug: https://crbug.com/gerrit/15103
Change-Id: I8cb89ce10b025994a045106c9c66dd243ae8ba50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319557
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-30 21:45:09 +00:00
Mike Frysinger
2ddbf8a8bf Merge "Merge history of v2.14.5." into main 2021-09-30 21:37:09 +00:00
Mike Frysinger
445723fd37 Merge history of v2.14.5.
The v2.14.[345] releases were cut on a branch based on v2.14.2.  We
had some regression fixes we wanted in v2.14, but too many risky
changes landed in main since to cut another v2.14.x directly, and
we didn't want to destabilize even more by pushing a v2.15 right
away.  So we branched to keep things healthy.

But people with old checkouts trying to upgrade from those versions
run into an old repo bug where it only selfupdates with fast-forwards,
and repo can't fast-forward from those divergent histories.  So let's
do a merge commit to stitch the history back together.

There's no actual changes in here.

Change-Id: I05a96048e3846321e57c5f5224fb8dcf3c191d35
2021-09-30 21:36:56 +00:00
Xin Li
436bde5137 Merge history of v1.13.11.
For older versions of repo, this would make it easier for it to perform
a self update by making it a fast-forward from the following tags:

  v1.13.9.2, v1.13.9.3, v1.13.9.4, v1.13.10, v1.13.11

Change-Id: Ia75776312eaf802a150db8bd7c0a6dce57914580
2021-09-30 11:49:08 -07:00
Raman Tenneti
4f88206178 trace2_event: Add remove_prefix to fix failing tests on Linux & macOS.
removeprefix is available i python 3.9. Mac and Linux are running in
a version below 3.9. Thus tests are failing with the following error:
  "AttributeError: 'str' object has no attribute 'removeprefix' "

Replaced the removeprefix with custom function which we will delete
once Linux and macOS versions are updated.

Tested:
$ ./run_tests

Bug: [google internal] b/201453085
Change-Id: I9b4d564ff1176e1b4471805ef05472c1914cd9f9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319375
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-29 21:24:59 +00:00
Mike Frysinger
f88282ccc2 git_config: update error handling with no config file
Now that _do throws an exception when `git` fails, update the logic
that tries to read config files but the file doesn't exist.

Bug: b/192664812
Change-Id: I6417ecd70891b8f2d5f2bdb819f91df69ac4b70c
Test: `repo upload` no longer crashes when .repo/config doesn't exist
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319295
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-09-29 01:02:47 +00:00
Mike Frysinger
8967a5aec6 launcher: bump version for new release
Change-Id: I9812185c9dfc11289547f5956c0cbe567d720f7f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319335
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-09-28 20:54:56 +00:00
Michael Kelly
2f3c3316e4 Update revisionId if required when using extend-project
When a hard revision ID is provided in a regular project tag then the
revisionId is updated as well if it is a commit hash.  The difference
is that if the revisionExpr is a commit, git-repo needs to update
refs/remotes/m/master with update-ref not symbolic-ref, as the latter
must refer to another ref, not to a specific commit.

Change-Id: I215a62dabb30225e480ad2c731416d775fc0c750
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310963
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-28 20:12:00 +00:00
Michael Kelly
37c21c268b Add 'dest-path' to extend-project to support changing path
This allows us to move the repository to a new location in the source
tree without having to remove-project + add a new project tag.

Change-Id: I4dba6151842e57f6f2b8fe60cda260ecea68b7b4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310962
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
2021-09-28 20:12:00 +00:00
Xin Li
b12c369e0b superproject: Only trigger enrollment logic when manifest have it.
The current code would check for enrollment status when the user did not
explicitly specify --[no-]use-superproject and do not have a remembered
value in their repo client. However, because superproject only makes
sense for manifests that have one specified, we should skip the
enrollment logic in that case.

Address this by checking manifest.superproject prior to proceeding. This
would avoid showing the greeting message of superproject enrollment
which can be confusing for developers.

Tested:
  For manifest without superproject:
   - repo sync --use-superproject will still show message for
     superproject;
   - repo sync will not show message regardless of enrollment state
  For manifest with superproject:
   - repo sync will show message and perform enrollment if not
     previously enrolled

Bug: https://crbug.com/gerrit/15039
Change-Id: Ic2be9f9d037f0e7cf3446da474a5a0d0e4bd88da
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319255
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-28 19:42:01 +00:00
Raman Tenneti
bbe8836494 superproject: Log syncstate's parameter as data-json it it is an array.
All the values of syncstate are strings, check the first byte and last
byte to see if it is an array. For syncstate data, there were no false
positives.

Tested:
$ repo_dev sync

Verified event logged for argv is "data-json".

$./run_tests

Bug: [google internal] b/201102002
Change-Id: Id56adb532b80267f08d09147ac663cdd5987ce87
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319075
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-09-28 18:22:49 +00:00
Mike Frysinger
9d96f58f5f make file removal a bit more robust
Some of the file removal calls are subject to race conditions (if
something else deletes the file), so extend our remove API to have
an option to ignore ENOENT errors.  Then update a bunch of random
call sites to use this new functionality.

Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195
Reviewed-by: Sean McAllister <smcallis@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-09-28 16:06:50 +00:00
Jack Neus
7a1e7e772f repo: add support for reading standalone manifests from disk
BUG=b:192664812
TEST=existing tests (no coverage), manual runs

Change-Id: Ic032417ecfca77d5e0de1b1ff62b30ce8205bfc5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318715
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-28 16:03:21 +00:00
Jack Neus
c474c9cba1 repo: Add support for standalone manifests
Added --standalone_manifest to repo tool. If set, the
manifest is downloaded directly from the appropriate source
(currently, we only support GS) and used instead of creating
a manifest git checkout. The manifests.git repo is still created to
keep track of various config but is marked as being for a standalone
manifest so that the repo tool doesn't try to run networked git
commands in it.

BUG=b:192664812
TEST=existing tests (no coverage), manual runs

Change-Id: I84378cbc7f8e515eabeccdde9665efc8cd2a9d21
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312942
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-28 15:40:46 +00:00
Raman Tenneti
956f7363d1 superproject: Log argv parameter of syncstate as 'data-json'.
Fixed: "we need to make a special case for logging the argv; it
should probably be a "data-json" event so that we log this directly as
an array rather than an encoded string.

Tested:
$ repo_dev sync

Verified event logged for argv is "data-json".

$./run_tests

Bug: [google internal] b/201102002
Change-Id: I18ccec79c73c8dc931cb8afc472b2361db8aea4c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319055
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-09-27 19:11:14 +00:00
Xin Li
6f8c1bf4ff Fix indent error which would have prevented choice expiration to work.
Change-Id: I077e05eea23ad58d1dde2c9fe5608660a56d03e5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318815
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Amith Dsouza <amithds@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
2021-09-27 18:59:21 +00:00
Xin Li
e0b16a22a0 superproject: support a new revision attribute.
Tested:
$ ./run_tests

Verified that a manifest that specified superproject revision would use
the specified revision, and superproject will use the default revision.

Note that this is a slight behavior change from earlier repo versions,
which would always use the branch name of the manifest itself. However,
the new behavior would be more consisitent with regular "project"
element and would allow superproject be used even if it is not enabled
for the particular manifest branch, so we have decided to make the
change as it would provide more flexibility and better matches what
other elements would do.

Bug: [google internal] b/187868160
Change-Id: I35255ee347aff6e65179f7879d52931f168b477e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317643
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-27 06:36:05 +00:00
Mike Frysinger
d669d2dee5 release-process: update distro baseline & add OpenSSH
Stop tracking Ubuntu Trusty & Xenial and Debian Jessie & Stretch
as they only had Python 3.5 available which we've dropped.

Backfill OpenSSH versions since we've started testing for it.

Change-Id: I03183ed97f6e43dce8a00e36cce2956544a26afc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318835
Reviewed-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-09-24 16:42:56 +00:00
wenchiching
366824937c platform_utils: os.rename exception when src and des on different file system
Symptom: repo sync exception
Root Cause: os.rename only works when source and destination are on the same file system
Solution: using shutil.move

to save disk usage, I create links for projects and project-objects, link to folder on another disk
lrwxrwxrwx  1 owenwen owenwen   47 Jun  9 16:40 project-objects -> /disk3/AndroidLocalRepos/.repo/project-objects/
lrwxrwxrwx  1 owenwen owenwen   40 Jun  9 16:40 projects -> /disk3/AndroidLocalRepos/.repo/projects/

below are exception I met:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/disk2/Android11/.repo/repo/subcmds/sync.py", line 550, in _CheckoutOne
    project.Sync_LocalHalf(syncbuf, force_sync=force_sync)
  File "/disk2/Android11/.repo/repo/project.py", line 1251, in Sync_LocalHalf
    self._InitWorkTree(force_sync=force_sync, submodules=submodules)
  File "/disk2/Android11/.repo/repo/project.py", line 2801, in _InitWorkTree
    self._CheckDirReference(self.gitdir, dotgit, share_refs=True)
  File "/disk2/Android11/.repo/repo/project.py", line 2674, in _CheckDirReference
    platform_utils.rename(dst_path, src_path)
  File "/disk2/Android11/.repo/repo/platform_utils.py", line 127, in rename
    os.rename(src, dst)
OSError: [Errno 18] Invalid cross-device link: '/disk2/Android11/system/libhidl/.git/packed-refs' -> '/disk2/Android11/.repo/projects/system/libhidl.git/packed-refs'
"""

Change-Id: Ifda2f16530cc5a8f280169f482ee858f9e5241d3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/316002
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-24 08:20:06 +00:00
Jack Neus
a84f43a006 manifest: make repo-hooks more robust wrt element ordering
Currently, repo will fail to sync to a manifest if the definition
of the repo-hooks project comes after the repo-hooks element.

BUG=none
TEST=new test, run_tests

Change-Id: I0bf85625173492af6c6404d4b67543e96e670562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318520
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jack Neus <jackneus@google.com>
2021-09-23 21:17:38 +00:00
Mike Frysinger
0468feac39 update-manpages: avoid regen just for datestamp update
To avoid noise due to the passage of time, don't regenerate man pages
if the only thing different is the datestamp in the header.

Change-Id: Ic8d7b08d12e59c66994c0cc2d4ec2d2ed3eb6e6d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318575
Reviewed-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-09-22 19:37:35 +00:00
Xin Li
0ec2029833 superproject: Move enrollment to opt-out when enabled globally
Our internal experiments was a success so far and we are enrolling 100%
users now.  Instead of asking every two weeks, simply consider a lack of
unexpired choice as accepting the system default.

With this change the user would still be able to override the system
default with --no-use-superproject, or to permanently set the choice in
user's profile with git config --global repo.superprojectchoice.

Bug: [google internal] b/190688390
Change-Id: Idc77a9cbf88a169d90304169e91f0d722dc4ac8b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317975
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
2021-09-20 07:21:22 +00:00
Raman Tenneti
d8e8ae8990 superproject: Log branch and remote url with every log message.
Saved superproject's remote URL in _remote_url data and used it
in the _Fecth function.

Tested:
$ ./run_tests
$ flake8 git_superproject.py
$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
$ repo_dev sync

   Verified the all log messages have the following format.
   repo superproject branch: <branch> url: <url> warning: <message>

Bug: [google internal] b/200072098
Change-Id: Iac6af7c99225479fd50bc6909396b22e0ce5f76b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318177
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-09-16 15:46:19 +00:00
Raman Tenneti
6448a4f2af sync: Log repo sync state events as 'data' events.
git_trace2_event_log.py:
+ Added LogDataConfigEvents method to log 'data' events.
  Sync's current_sync_state and previous_sync_state are logged
  as 'data' events in the current log.

  It logs are key/value in the |config| argument. Each key is
  prefixed with |prefix| argument.

  The following are sample events that are logged during repo sync.

   {"event":"data",
   "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
   "thread":"MainThread",
   "time":"2021-09-14T18:16:19.935846Z",
   "key":"previous_sync_state/repo.syncstate.main.synctime",
   "value":"2021-09-14T17:27:11.573717Z"}

   {"event":"data",
   "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0",
   "thread":"MainThread",
   "time":"2021-09-14T18:16:19.955546Z",
   "key":"current_sync_state/repo.syncstate.main.synctime",
   "value":"2021-09-14T18:16:19.935979Z"}

tests/test_git_trace2_event_log.py:
+ Added unit tests

sync.py:
+ Changed logging calls to LogDataConfigEvents.

Tested:
$ ./run_tests

Tested it by running the following command multiple times.
$ repo_dev sync -j 20
  repo sync has finished successfully

  Verified config data is looged in trace2 event logs.

Bug: [google internal] b/199758376
Change-Id: I75fd830e90c1811ec28510538c99a2632b104e85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317823
Reviewed-by: Josh Steadmon <steadmon@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-09-14 21:36:12 +00:00
Xin Li
1328c35a4d superproject: Provide accurate feedback for user choice
Currently the code would give a message that would appear like the user
have enrolled the experiment regardless of the actual choice. For users
who choose to not enroll in the experiment, we should give them
instructions to override (enable) superproject once instead of how to
disable it, which is what the code already behave.

Bug: [google internal] b/199167992
Change-Id: Iba3314cb510aedf024375a26baa8bc1d5e2846cf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317382
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
2021-09-08 20:35:42 +00:00
Raman Tenneti
7f8bd85184 superoject: log error message in the 'fmt' field also.
Tested:
+ Verified error messages are being collected.

Bug: [google internal] b/193711236
Change-Id: I6c608a2af332ccd38722b7f83a82e5ac8fa143db
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317162
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-09-03 00:32:04 +00:00
Roger Shimizu
c63328e5ff docs: Add version for Ubuntu 21.04 Hirsute and Debian 11 Bullseye
* Add footer to the version table, so easier to read and maintain.
* Add version entry for Ubuntu 21.04 Hirsute (non-LTS).
* Add version entry for Debian 11 Bullseye (LTS).

Change-Id: Ic72f911e616b1a13901e56074004f05cdc2c7633
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313322
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Xin Li <delphij@google.com>
2021-09-03 00:16:43 +00:00
Raman Tenneti
b55769a5c9 superproject: print messages if the manifest has superproject tag.
1) If the manifest has superproject tag (git_master, etc), then
   display error/warning messages (as it is doing today)
2) If the manifest doesn't have superproject tag (nest, chromeos
   manifests), then don't display any error/warning messages about
   superrproject (behave as though user has specified
   --no-use-superproject).
3) Print error/warning messages if --use-superproject passed as
   argument to repo sync.
4) No change in behavior for the repo init command.

git_superproject.py:
+ Fixed typo in _WriteManifestFile method name
+ Superproject accepts print_message  as an argument and it defaults
  to True. All messages that are printed to stderr are controlled by
  this flag. If it is True, then messages get printed.
+ Added PrintMessages function which return true if either
  --use-superproject is specified on the command line or if the
  manifest has a superproject tag.

sync.py:
+ Displays the warning message if PrintMessgages are enabled and
  passes that as argument to superproject object.
+ Added 'hassuperprojecttag' trace2 log entry for analysis. We can
  find users/branches that are using superproject, but the manifest is
  missing the superproject tag.

Tested:
$ ./run_tests

+ Verified printing of messages with and without superproject tag, with
  with --use-superproject option.

+ aosp-master
  $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest
  $ repo_dev sync

+ A manifest without superproject tag.
  $ repo_dev init -m $(pwd)/manifest_7482982.xml
  $ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml

Bug: [google internal] b/196411099
Change-Id: I92166dcad15a4129fab82edcf869e7c8db3efd4b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314982
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-08-13 20:07:40 +00:00
Raman Tenneti
5637afcc60 superproject: prepend messages with - "repo superproject"
Changed _LogError method to _LogWarning.

Replaced 'repo error:' with "repo superproject warning:"(except IOError
message, which is still an "repo superproject error:" message)

Tested:
$ ./run_tests

Tested the errors and warnings by forcing the error/warning.
$ repo_dev sync -j 20 --use-superproject platform/packages/apps/Music
  ...
  repo superproject warning: please file a bug using go/repo-bug to report missing commit_ids for: []
  ...
  repo superproject error: cannot write manifest to : /sdc/android/src/aosp/.repo/exp-superproject/superproject_override.xml
  ...

Bug: [google internal] b/193711236

Change-Id: Ia0b6c830e04cf18dfc1a2ce325181a5b1160e054
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314642
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2021-08-12 16:30:26 +00:00
Mike Frysinger
df8b1cba47 man: make output system independent
The current help output might change based on the number of CPU cores
available (since it reflects the dynamic --jobs logic).  This is good
for users running repo locally, but not good for shipping static man
pages.  Hook the help output to have it generate the same output all
the time.

Change-Id: I3098ceddc0ad914b0b8e3b25d660b5a264cb41ee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312882
Reviewed-by: Roger Shimizu <rosh@debian.org>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-07-31 11:39:35 +00:00
Raman Tenneti
9122bfc3a8 sync: Remove '_' from the repo.syncstate.* keys when saved to config.
GitConfig doesn't save keys if the keys contain "_" characters. Some
of the options like mp_update, use_superproject have underscores.

This fixes issue with previous_sync_state missing some of the options.

Tested:
$ ./run_tests

$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest

Tested it by running the sync command multiple times and verifing
previous_sync_state and current_sync_state have the same keys.

$ repo_dev sync -j 20
  repo sync has finished successfully

  Verified config file has [syncstate ...] data saved.

Bug: [google internal] b/188573450
Change-Id: I16b52a164f9dd1633d7dad1d8cf6b151c629fcb1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313242
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-07-29 22:41:57 +00:00
Raman Tenneti
7954de13b7 sync: Added logging of repo sync state and config options for analysis.
git_config.py:
+ Added SyncAnalysisState class, which saves the following data
  into the config object.
  ++ sys.argv, options, superproject's logging data.
  ++ repo.*, branch.* and remote.* parameters from config object.
  ++ current time as synctime.
  ++ Version number of the object.
+ All the keys for the above data are prepended with 'repo.syncstate.'
+ Added GetSyncAnalysisStateData and UpdateSyncAnalysisState methods
  to GitConfig object to save/get the above data.

git_trace2_event_log.py:
+ Added LogConfigEvents method with code from DefParamRepoEvents
  to log events.

sync.py:
+ superproject_logging_data is a dictionary that collects all the
  superproject data that is to be logged as trace2 event.
+ Sync at the end logs the previously saved syncstate.* parameters
  as previous_sync_state. Then it calls config's UpdateSyncAnalysisState
  to save and log all the current options, superproject logged data.

docs/internal-fs-layout.md:
+ Added doc string explaining [repo.syncstate ...] sections of
  .repo/manifests.git/config file.

test_git_config.py:
+ Added unit test for the new methods of GitConfig object.

Tested:
$ ./run_tests

$ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest

Tested it by running the following command multiple times.
$ repo_dev sync -j 20
  repo sync has finished successfully

  Verified config file has [syncstate ...] data saved.

Bug: [google internal] b/188573450
Change-Id: I1f914ce50f3382111b72940ca56de7c41b53d460
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313123
Tested-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2021-07-29 19:20:57 +00:00
Raman Tenneti
ae86a46022 superproject: Skip updating of superproject when -l is used with sync.
Skip updating the superproject when -l is present and use the existing
superproject, if available (this would make sync -l work as it's
intended to do), and fall back to sync without superproject when not
(this would catch the case when superproject is enabled by automatic
rollout).

Tested:
$ repo sync -j 20 -n
NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version`
/usr/local/google/home/rtenneti/work/android/src/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed.
Fetching: 100% (1032/1032), done in 41.184s
...

$ repo_dev sync -j 20 -l
prebuilts/asuite/: discarding 1 commits
prebuilts/runtime/: discarding 1 commits
...
repo sync has finished successfully.

+ With superproject-override.xml and test it.

  $ ls -l .repo/exp-superproject/
  total 176
  drwxr-xr-x 7 rtenneti primarygroup   4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
  -rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 superproject_override.xml
  rtenneti@rtenneti:~/work/android/src/aosp$ repo_dev sync -j 20 -l
  ...
  repo sync has finished successfully.

+ Rename the file superproject-override.xml and test it.
  $ ls -l .repo/exp-superproject/
  total 176
  drwxr-xr-x 7 rtenneti primarygroup   4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git
  -rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 temp.xml

  $ repo_dev sync -j 20 -l
  Checking out:  1% (12/1031) platform/external/rust/crates/fallible-streaming-iteexternal/linux-kselftest/: discarding 1 commits
  prebuilts/remoteexecution-client/: discarding 1 commits
  Checking out: 51% (536/1031) platform/prebuilts/gcc/darwin-x86/aarch64/....
  ....
  Checking out: 100% (1031/1031), done in 5.478s
  repo sync has finished successfully.

Bug: [google internal] b/184368268
Change-Id: I3aba5872e4f7c299977b92c2a39847ef28698c5a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312962
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-07-28 16:12:53 +00:00
Mike Frysinger
73c43b839f repo: add --show-toplevel akin to git
Simple API to make it easy to find the top of the repo client checkout
for users.  This mirrors the `git rev-parse --show-toplevel` API.

Change-Id: I0c3f98def089d0fc9ebcfa50aa3dc02091c1c273
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312909
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-07-28 05:38:53 +00:00
Mike Frysinger
56345c345b repo: refactor help output handling
Currently we have the behavior:
* `repo`: Equivalent to `repo help` -- only shows common subcommands
  (with short description), and then exits 0.
* `repo --help`: Shows repo's core options, lists all commands (no
  specific info), and then exits 0.

The first case is not behaving well:
* If you run `repo` without a specific subcommand, that's an error,
  so we should be exiting 1 instead.
* Showing only subcommands and no actual option summary makes it seem
  like repo itself doesn't take any options.  This confuses users.

Let's rework things a bit.  Now we have the behavior:
* `repo`: Shows repo's core options, lists all commands (no specific
  info), and then exits 1.
* `repo --help`: Shows repo's core options, shows common subcommands
  (with short description), and then exits 0.
* `repo --help-all`: Shows repo's core options, shows all subcommands
  (with short description), and then exits 0.

Basically we swap the behavior of `repo` and `repo --help`, and fix
the exit status when the subcommand is missing.

The addition of --help-all is mostly for the man pages.  We were
relying on `repo help --all` to generate the repo(1) man page, but
that too omitted the core repo options.  Now the man page includes
all the core repo options and provides a summary of all commands.

Change-Id: I1f99b99d5b8af2591f96a078d0647a3d76d6b0fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312908
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-07-28 05:38:34 +00:00
Mike Frysinger
a024bd33b8 repo: make --version always work
We don't really care what the subcommand is set to when --version
output is requested, so stop enforcing it.  This fixes some weird
behavior like `repo --version version` fails, but `repo --version
help` works.

The new logic skips subcommand validation, so `repo --version asdf`
will still display the version output.  This matches git behavior,
and makes a bit of sense when we consider that the user really wants
to see the tool version, and probably doesn't care about anything
else on the command line.

Change-Id: I87454d473c2c8869344b3888a7affaa2e03f5b0f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312907
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-07-28 05:37:52 +00:00
Mike Frysinger
968d646f04 repo: refactor internal --help/--version parsing
The _ParseArgs method parses the arguments and processes some of
the options, with the rest left to the _Run method.  Simplify the
_ParseArgs method to only parse arguments and have _Run handle all
actual processing.

This will make it easier to add more terminal options (ones that
exit immediately without a subcommand), and makes it easier to
understand the overall code flow.

Change-Id: I47f7274c3f2b59378fd479e403e70fb24b681536
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312906
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-07-28 05:37:27 +00:00