Commit Graph

1912 Commits

Author SHA1 Message Date
Fredrik de Groot
e8a7b9d596 Add smoke test for subcmd forall
After some refactoring earlier, the forall command was
broken briefly, returning after only one run instead
of after all projects.

This test, albeit simple in nature, would have caught that.

Due to the somewhat demanding nature of forall,
a lot more setup was needed than expected but seems
to do its job now so hopefully it catches similar stuff
in the future.

Change-Id: I51e161ff0e7e31a65401211c376f319bda504532
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445461
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Fredrik de Groot <fredrik.de.groot@haleytek.com>
2024-12-11 00:30:15 -08:00
Josip Sokcevic
cf411b3f03 Remove gitc support from repo
gitc is no longer available.

Change-Id: I0cbfdf936832f2cdd4876104ae3cc5a6e26154e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/444841
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
2024-12-03 22:27:56 +00:00
Josip Sokcevic
1feecbd91e branches: Escape percent signs in branch names
If a branch name contains a percent sign, it will be interpreted as a placeholder and color.py will fail to format it.

To avoid this, escape the percent signs prior to calling Coloring
method.

Bug: b/379090488
Change-Id: Id019c776bbf8cbed5c101f2773606f1d32c9e057
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443801
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-12-03 19:02:20 +00:00
Peter Kjellerstedt
616e314902 sync: Do not fail to sync a manifest with no projects
Since commit 454fdaf119 (v2.48), syncing a
manifest without any projects would result in:

  Repo command failed: RepoUnhandledExceptionError
          Number of processes must be at least 1

Bug: 377546300
Change-Id: Iaa2f6a3ac64542ad65a19c0eef449f53c09cae67
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443442
Reviewed-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Tested-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
2024-11-26 10:16:03 +00:00
Josip Sokcevic
fafd1ec23e Fix event log command event hierarchy.
command should be cmd_name, to match what git is emitting. This also
fixes arguments, so that only relevant arguments are passed instead
of the entire sys.args, which will contain wrapper information

Change-Id: Id436accfff511292ec2c56798fffb2306dda38fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443741
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
2024-11-22 18:39:41 +00:00
Josip Sokcevic
b1613d741e Make repo installation work without .git
Some tools like jj and cog will not have .git. This change
makes it possible to run all repo commands in such setups.

Change-Id: I7f3845dc970fbaa731c31e0aa48355a4b56ed3a6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/442821
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-11-18 19:36:14 +00:00
Kuang-che Wu
ab2d321104 sync: fix connection error on macOS
With a large number of sync workers, the sync process may fail on
macOS due to connection errors. The root cause is that multiple
workers may attempt to connect to the multiprocessing manager server
at the same time when handling the first job. This can lead to
connection failures if there are too many pending connections, exceeding
the socket listening backlog.

Bug: 377538810
Change-Id: I1924d318d076ca3be61d75daa37bfa8d7dc23ed7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441541
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2024-11-06 16:33:17 +00:00
Josip Sokcevic
aada468916 upload: Return correct tuple values in _ProcessResults
Incorrect tuple values were returned with http://go/grev/440221 -
instead of returning (Project, ReviewableBranch), _ProcessResults was
returning (int, ReviewableBranch).

R=jojwang@google.com

Bug: 376731172
Change-Id: I75205f42fd23f5ee6bd8d0c15b18066189b42bd9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441121
Reviewed-by: Sam Saccone <samccone@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
2024-10-31 21:18:53 +00:00
Allen Webb
1d5098617e worktree: Do not try to fix relative paths
--worktree was broken with incorrect paths in the .git files
whenever the local copy of git populated gitdir with relative paths
instead of absoulte paths.

Bug: 376251410
Change-Id: Id32dc1576315218967de2a9bfe43bf7a5a0e7aa6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440801
Commit-Queue: Allen Webb <allenwebb@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Allen Webb <allenwebb@google.com>
2024-10-30 17:03:57 +00:00
Josip Sokcevic
e219c78fe5 forall: Fix returning results early
rc should be returned only after all results are processed.

R=jojwang@google.com

Bug: b/376454189
Change-Id: I8200b9954240dd3e8e9f2ab82494779a3cb38627
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440901
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>
2024-10-30 16:11:04 +00:00
joehsu
f9f4df62e0 Use full name of the revision when checking dest-branch
The manifest usually doesn't sepecify the revision with the full name
(e.g. refs/heads/REV).
However, when checking if the name of the merge branch, full name is
used on the merge branch.

The CL use full name of revision when comparing it with the merge
branch.

Bug: b/370919047
Test: repo upload on a project with `dest-branch` set
Change-Id: Ib6fa2f7246beb5bae0a26a70048a7ac03b6c5a2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438401
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Joe Hsu <joehsu@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-10-28 23:47:08 +00:00
Fredrik de Groot
ebdf0409d2 Add REPO_SKIP_SELF_UPDATE check in sync
The command _PostRepoFetch will try to self update
during repo sync. That is beneficial but adds
version uncertainty, fail potential and slow downs
in non-interactive scenarios.

Conditionally skip the update if env variable
REPO_SKIP_SELF_UPDATE is defined.

A call to selfupdate works as before, meaning even
with the variable set, it will run the update.

Change-Id: Iab0ef55dc3d3db3cbf1ba1f506c57fbb58a504c3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439967
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2024-10-28 17:46:25 +00:00
Fredrik de Groot
303bd963d5 manifest: add optional base check on remove and extend
This adds an optional, built-in checker for
guarding against patches hanging on wrong
base revisions, which is useful if a lower layer of
the manifest changes after a patch was done.

When adding a patch with a new revision using
extend-project or remove-project/project:

          C---D---E patches in project bla
         /
    A---B project bla in manifest state 1

<extend-project name="bla" revision="E" base-rev="B">

If project bla gets updated, in a new snap ID
or by a supplier or similar, to a new state:

          C---D---E patches in project bla
         /
    A---B---F---G project bla in manifest state 2

Parsing will fail because revision of bla is now G,
giving the choice to create a new patch branch
from G and updating base-rev, or keeping previous
branch for some reason and only updating base-rev.

Intended for use in a layered manifest with
hashed revisions. Named refs like branches and tags
also work fine when comparing, but will be misleading
if a branch is used as base-rev.

Change-Id: Ic6211550a7d3cc9656057f6a2087c505b40cad2b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436777
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-10-28 16:55:10 +00:00
Josip Sokcevic
ae384f8623 [event_log] Stop leaking semaphore resources
With the global state and fork, we are left with uncleaned resources.
Isolate mulitprocessing.Value in a function so we stop the leak.

Bug: 353656374
Change-Id: If50bb544bda12b72f00c02bc1d2c0d19de000b88
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440261
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
2024-10-24 16:58:17 +00:00
Kuang-che Wu
70a4e643e6 progress: always show done message
The done message was omitted if the task is shorter than 0.5s. This
might confuse users.

Bug: b/371638995
Change-Id: I3fdd2cd8daea16d34fba88457d09397fff71af15
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440222
Tested-by: Kuang-che Wu <kcwu@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2024-10-24 16:21:28 +00:00
Kuang-che Wu
8da4861b38 subcmds: reduce multiprocessing serialization overhead
Follow the same approach as 39ffd9977e to reduce serialization overhead.

Below benchmarks are tested with 2.7k projects on my workstation
(warm cache). git tracing is disabled for benchmark.

(seconds)              | v2.48 | v2.48 | this CL | this CL
	               |       |  -j32 |         |    -j32
-----------------------------------------------------------
with clean tree state:
branches (none)        |   5.6 |   5.9 |    1.0  |    0.9
status (clean)         |  21.3 |   9.4 |   19.4  |    4.7
diff (none)            |   7.6 |   7.2 |    5.7  |    2.2
prune (none)           |   5.7 |   6.1 |    1.3  |    1.2
abandon (none)         |  19.4 |  18.6 |    0.9  |    0.8
upload (none)          |  19.7 |  18.7 |    0.9  |    0.8
forall -c true         |   7.5 |   7.6 |    0.6  |    0.6
forall -c "git log -1" |  11.3 |  11.1 |    0.6  |    0.6

with branches:
start BRANCH --all     |  21.9 |  20.3 |   13.6  |    2.6
checkout BRANCH        |  29.1 |  27.8 |    1.1  |    1.0
branches (2)           |  28.0 |  28.6 |    1.5  |    1.3
abandon BRANCH         |  29.2 |  27.5 |    9.7  |    2.2

Bug: b/371638995
Change-Id: I53989a3d1e43063587b3f52f852b1c2c56b49412
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440221
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Kuang-che Wu <kcwu@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>
2024-10-23 23:34:34 +00:00
Kuang-che Wu
39ffd9977e sync: reduce multiprocessing serialization overhead
Background:
 - Manifest object is large (for projects like Android) in terms of
   serialization cost and size (more than 1mb).
 - Lots of Project objects usually share only a few manifest objects.

Before this CL, Project objects were passed to workers via function
parameters. Function parameters are pickled separately (in chunk). In
other words, manifests are serialized again and again. The major
serialization overhead of repo sync was
  O(manifest_size * projects / chunksize)

This CL uses following tricks to reduce serialization overhead.
 - All projects are pickled in one invocation. Because Project objects
   share manifests, pickle library remembers which objects are already
   seen and avoid the serialization cost.
 - Pass the Project objects to workers at worker intialization time.
   And pass project index as function parameters instead. The number of
   workers is much smaller than the number of projects.
 - Worker init state are shared on Linux (fork based). So it requires
   zero serialization for Project objects.

On Linux (fork based), the serialization overhead is
  O(projects)  --- one int per project
On Windows (spawn based), the serialization overhead is
  O(manifest_size * min(workers, projects))

Moreover, use chunksize=1 to avoid the chance that some workers are idle
while other workers still have more than one job in their chunk queue.

Using 2.7k projects as the baseline, originally "repo sync" no-op
sync takes 31s for fetch and 25s for checkout on my Linux workstation.
With this CL, it takes 12s for fetch and 1s for checkout.

Bug: b/371638995
Change-Id: Ifa22072ea54eacb4a5c525c050d84de371e87caa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439921
Tested-by: Kuang-che Wu <kcwu@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>
2024-10-23 02:58:45 +00:00
Kaushik Lingarkar
584863fb5e Fix incremental syncs for prjs with submodules
When performing an incremental sync (re-running repo init with an
updated manifest revision) with --fetch-submodules or sync-s=true,
there is an attempt to get a list of all projects (including
submodules) before projects are actually fetched. However, we can
only list submodules of a project if we have already fetched its
revision. Instead of throwing an error when we don't have the
revision, assume there are no submodules for that project. In the
sync cmd, we already update the list of projects to include
submodules after fetching superprojects.

Change-Id: I48bc68c48b5b10117356b18f5375d17f9a89ec05
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439761
Commit-Queue: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Tested-by: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org>
2024-10-18 03:55:10 +00:00
Josip Sokcevic
454fdaf119 sync: Always use WORKER_BATCH_SIZE
With 551285fa35, the comment about number
of workers no longer stands - dict is shared among multiprocesses and
real time information is available.

Using 2.7k projects as the baseline, using chunk size of 4 takes close
to 5 minutes. A chunk size of 32 takes this down to 40s - a reduction of
rougly 8 times which matches the increase.

R=gavinmak@google.com

Bug: b/371638995
Change-Id: Ida5fd8f7abc44b3b82c02aa0f7f7ae01dff5eb07
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438523
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2024-10-07 18:44:19 +00:00
Josip Sokcevic
f7f9dd4deb project: Handle git sso auth failures as repo exit
If a user is not authenticated, repo continues execution and it will
likely result in more of the same errors being printed. A user is also
likely to SIGTERM the process resulting in more errors.

This change stops repo sync if any of repositories can't be fetched to
Git authentcation using sso helper. We could extend this to all Git
authentication

Change-Id: I9e471e063450c0a51f25a5e7f12a83064dfb170c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438522
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-10-03 20:47:50 +00:00
Josip Sokcevic
70ee4dd313 superproject: Remove notice about beta
It's been the default for Android for over a year now, and it's no
longer useful notice.

Change-Id: I53c6f1e7cee8c1b2f408e67d3a6732db3b272bee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438521
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-10-03 20:37:18 +00:00
Josip Sokcevic
cfe3095e50 project: run fetch --refetch on unable to not parse commit
Similarly to e59e2ae757, handle missing
gc'ed commits by running `git fetch --refetch`.

R=jojwang@google.com

Bug: b/360889369
Bug: b/371000949
Change-Id: I108b870b855d3b9f23665afa134c6e35f7cd2830
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438461
Reviewed-by: Joanna Wang <jojwang@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-10-03 17:40:37 +00:00
Josip Sokcevic
621de7ed12 Disable git terminal prompt during fetch/clone
git fetch operation may prompt user to enter username and password.
This won't be visible to user when repo sync operation since stdout and
stderr are redirected. If that happens, user may think repo is doing
work and likely won't realize it's stuck on user's input.

This patch disables prompt for clone and fetch operations, and repo will
fail fast.

R=gavinmak@google.com

Bug: b/368644181
Change-Id: I2efa88ae66067587a00678eda155d861034b9127
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438001
Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2024-09-26 22:10:36 +00:00
Kaushik Lingarkar
d7ebdf56be init: add --manifest-upstream-branch
When a sha1 is provided to '--manifest-branch', the ref which
is expected to contain that sha1 can be provided using the new
'--manifest-upstream-branch' option. This is useful with
'--current-branch' to avoid having to sync all heads and tags,
or with a commit that comes from a non-head/tag ref (like a
Gerrit change ref).

Change-Id: I46a3e255ca69ed9e809039e58b0c163e02af94ef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436717
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
Tested-by: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
2024-09-26 00:52:28 +00:00
Kaushik Lingarkar
fabab4e245 man: regenerate man pages
Change-Id: Icf697eda7d5dcdc87854ad6adf607353c7ba5ac2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/437941
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2024-09-25 20:57:42 +00:00
Brian Norris
b577444a90 project: Copy and link files even with local branches
In the winding maze that constitutes Sync_LocalHalf(), there are paths
in which we don't copy-and-link files. Examples include something like:

  cd some/project/
  repo start head .
  # do some work, make some commit, upload that commit to Gerrit

  [[ ... in the meantime, someone addes a <linkfile ...> for
     some/project/ in the manifest ... ]]

  cd some/project/
  git pull --rebase
  repo sync

In this case, we never hit a `repo rebase` case, which might have saved
us. Instead, the developer is left confused why some/project/ never had
its <linkfile>s created.

Notably, this opens up one more corner case in which <linkfile ... /> or
<copyfile ... /> could potentially clobber existing work in the
destination directory, but there are existing cases where that's true,
and frankly, those seem like bigger holes than this new one.

Change-Id: I394b0e4529023a8ee319dc25d03d513a19251a4a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/437421
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Brian Norris <briannorris@google.com>
Commit-Queue: Brian Norris <briannorris@google.com>
2024-09-19 00:11:52 +00:00
Navil
1e19f7dd61 sync: include TARGET_RELEASE when constructing smart sync target.
When using the smart sync option, we try to construct the target that
was "lunched" from the TARGET_PRODUCT and TARGET_BUILD_VARIANT envvars.

However, an android target is now made of three parts,
{TARGET_PRODUCT}-{TARGET_RELEASE}-{TARGET_BUILD_VARIANT}.

I am leaving the option of creating a target if a TARGET_RELEASE is not
specified in case there are other consumers who depend on that option.

BUG=b:358101714
TEST=./run_tests
TEST=smart sync on android repo and manually inspecting
smart_sync_override.xml

Change-Id: I556137e33558783a86a0631f29756910b4a93d92
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436977
Tested-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
2024-09-12 16:15:50 +00:00
bright
d8b4101eae color: fix have_fg not re assign to true
In method _parse the value of this variable 'have_fg ' is always
False, Maybe reassign it to True is lost.
I guess the author’s original intention was:
if set some value in gitconfig file(for ex: text = black red ul),
the first is bg color, the second is fg color, and the last one is attr.



Change-Id: I372698fe625db4c1fdaa94ea7f193a80a850ecb9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/425997
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Bright Ma <mmh1989@foxmail.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-09-12 16:15:06 +00:00
Peter Kjellerstedt
1c53b0fa44 tox.ini: Make the lint and format environments run black for all code
This matches the extra files specified in run_tests.

Change-Id: Ic8999383a17b3ec7ae27322323ea44eeaa40c968
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/434998
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-09-12 16:09:24 +00:00
Peter Kjellerstedt
e5ae870a2f tox.ini, constraints.txt: Lock the version of black to <24
The formatting produced by black versions before 24 matches the current
formatting of the code.

Change-Id: I045f22d2f32a09d4683867293e81512f2abd1036
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/434997
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
2024-09-12 16:05:35 +00:00
Josip Sokcevic
e59e2ae757 project: run fetch --refetch onacould not parse commit
git may gc reachable objects in partial clone repository due to a bug
(report:
https://lore.kernel.org/git/20240802073143.56731-1-hanyang.tony@bytedance.com/
). Until git is properly patched and released, force --refetch iff
"could not parse commit" is part of git output. --refetch will will
ensure that gc'ed objects are retrieved.

Bug: b/360889369
Change-Id: I0fc911c591060f859235dcd8d019881106f0858e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/437017
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
2024-09-12 15:27:12 +00:00
Jeroen Dhollander
c44ad09309 Add a --rebase option to sync command
Previously repo would abort a sync if there were published changes not
merged upstream. The --rebase option allows the modification of
published commits.

This is a copy of http://go/grev/369694 with the merge conflicts
resolved.

Bug: 40014610
Change-Id: Idac8199400346327b530abea33f1ed794e5bb4c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/435838
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Jeroen Dhollander <jeroendh@google.com>
Commit-Queue: Jeroen Dhollander <jeroendh@google.com>
2024-08-30 09:08:29 +00:00
Erik Elmeke
4592a63de5 sync: Fix git command for aborting rebase being called incorrectly.
The argument list was incorrectly destructured so only the first
element of the list was considered a git-cmd, split up by each
character in the string.

Change-Id: Idee8a95a89a7da8b8addde07135354fc506c2758
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/435839
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Erik Elmeke <erik@haleytek.corp-partner.google.com>
Tested-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
2024-08-28 08:56:35 +00:00
Mike Frysinger
0444ddf78e project: ignore more curl failure modes
Current clone bundle fetches from Google storage results HTTP/404
and curl exiting 56.  This is basically WAI, so stop emitting
verbose error output whenever that happens.  Also add a few more
curl exit statuses based on chromite history, and document them.

Change-Id: I3109f8a8a19109ba9bbd62780b40bbcd4fce9b76
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/432197
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2024-07-02 19:03:54 +00:00
Shik Chen
9bf8236c24 logging: Fix log formatting with colored output
The log message is already formatted before being passed to the colorer.
To avoid the exception "TypeError: not enough arguments for format
string", we should use the `nofmt_colorer` instead.

This bug occurs only when the formatted string still contains '%'
character. The following snippet can reproduce the bug:

```
from repo_logging import RepoLogger
RepoLogger(__name__).error("%s", "100% failed")
```

Change-Id: I4e3977b3d21aec4e0deb95fc1c6dd1e59272d695
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/432017
Tested-by: Shik Chen <shik@google.com>
Commit-Queue: Shik Chen <shik@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2024-07-02 06:24:31 +00:00
Mike Frysinger
87f52f308c upload: add a --topic option for setting topic explicitly
Let people specify the exact topic when uploading CLs.  The existing
-t option only supports setting the topic to the current local branch.

Add a --topic-branch long option to the existing -t to align it a bit
better with --hashtag & --hashtag-branch.

Change-Id: I010abc4a7f3c685021cae776dd1e597c22b79135
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/431997
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2024-07-01 17:54:19 +00:00
Erik Elmeke
562cea7758 sync: Abort rebase in progress if force-checkout is set
This will make "repo sync -d --force-checkout" more reliable
in CI automation, as there are fewer things in the way that may
need manual intervention.

Bug: b/40015382
Change-Id: I8a79971724a3d9a8e2d682b7a0c04deda9e34177
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/423317
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
Commit-Queue: Erik Elmeke <erik@haleytek.corp-partner.google.com>
2024-05-23 14:14:18 +00:00
Erik Elmeke
eede374e3e ssh: Set git protocol version 2 on SSH ControlMaster
According to https://git-scm.com/docs/protocol-v2#_ssh_and_file_transport,
when using SSH, the environment variable GIT_PROTOCOL must be set
when establishing the connection to the git server.

Normally git does this by itself. But in repo-tool where the SSH
connection is managed by the repo-tool, it must be passed in
explicitly instead.

Under some circumstances of environment configuration, this
caused all repo sync commands over ssh to always use
git protocol version 1. Even when git was configured to use
version 2.

Using git protocol v2 can significantly improve fetch speeds,
since it uses server side filtering of refs, reducing the
amount of unneccessary objects to send.

Change-Id: I6d4c3b7300a6090d707480b1a638ed03622fa71a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/411362
Tested-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Erik Elmeke <erik@haleytek.corp-partner.google.com>
2024-05-16 13:26:46 +00:00
Mike Frysinger
2c5fb84d35 upload: drop check for uncommitted local changes
git push, like most git commands, does not warn or otherwise prompt
users when there are local uncommitted changes.  To simplify the
upload logic, and to harmonize repo upload with git push as a more
git-esque flow, stop checking/warning/prompting the user here too.

Change-Id: Iee18132f0faad0881f1a796cb58821328e04b694
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/425337
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2024-05-14 02:32:27 +00:00
Mike Frysinger
12f6dc49e9 git: raise hard version to 1.9.1
Debian 7 Wheezy went EOL in May 2018.  We don't need to carry support
for that anymore as there have been 5 major releases since.  Ubuntu
Precise went EOL in Apr 2019 (including the extended support phase).
That means we can bump the required git version from 1.7.9 to 1.9.1.

git-1.7.9 was released in 2012 while git-1.9.1 was released in 2014.
So that shouldn't be a problem either.  And we've been warning people
using git versions older than 1.9.1 for 3 years now that they need to
upgrade.

Change-Id: Ifbbf72f51010b0a944c2785895d1b605333f9146
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415637
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2024-05-01 15:23:50 +00:00
Mike Frysinger
5591d99ee2 release: update-hooks: helper for automatically syncing hooks
These hooks are maintained in other projects.  Add a script to automate
their import so people don't send us changes directly, and we can try to
steer them to the correct place.

Change-Id: Iac0bdb3aae84dda43a1600e73107555b513ce82b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/422177
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2024-04-23 18:31:51 +00:00
Mike Frysinger
9d865454aa gitc: delete a few more dead references
Change-Id: I1da6f2ee799c735a63ac3ca6e5abd1211af10433
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/419217
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2024-04-18 02:30:06 +00:00
Mike Frysinger
cbd78a9194 man: regenerate man pages
Change-Id: I8d9dcb37f315d4208b7c8005206ae939dad79a3e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/419197
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2024-04-18 02:28:33 +00:00
Kaiyi Li
46819a78a1 Remove platform_utils.realpath
... since it's just a simple wrapper of os.path.realpath now.

Change-Id: I7433e5fe09c64b130f06e2541151dce1961772c9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/416637
Tested-by: Kaiyi Li <kaiyili@google.com>
Reviewed-by: Greg Edelston <gredelston@google.com>
Commit-Queue: Kaiyi Li <kaiyili@google.com>
2024-03-27 17:13:58 +00:00
Kaiyi Li
159389f0da Fix drive mounted directory on Windows
On my Windows machine, I mount drive D: to the directory C:\src.

The old implementation returns the incorrect 'C:\\??\\Volume{ad2eb15e-f293-4d48-a448-54757d95a97c}' result, which breaks the repo init command.

With the use of os.path.realpath, it can return 'D:\\' correctly.

Change-Id: Ia5f53989055125cb282d4123cf55d060718aa1ff
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/416580
Reviewed-by: Greg Edelston <gredelston@google.com>
Tested-by: Kaiyi Li <kaiyili@google.com>
Commit-Queue: Kaiyi Li <kaiyili@google.com>
2024-03-27 14:00:47 +00:00
Mike Frysinger
4406642e20 git_command: unify soft/hard versions with requirements.json
Use the requirements logic in the wrapper to load versions out of the
requirements.json file to avoid duplicating them in git_command.py.

Change-Id: Ib479049fc54ebc6f52c2c30d1315cf1734ff1990
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415617
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2024-03-21 21:20:50 +00:00
Josip Sokcevic
73356f1d5c project: Check if dotgit exists w/out symlink check
os.path.exists returns false on a broken symlink. This is not what repo
needs when checking if a project is setup properly.

For example, if src/foo/.git can't be resolved, repo tries to create
symlink and that results in FileExistsError.

Use lexists which returns True even if symlink is broken. That will
force path where repo checks where symlink is pointing to and will fix
it to the correct location.

Bug: b/281746795
Change-Id: Id3f7dc3a3cb6499d02ce7335eca992ddc7deb645
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415197
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: George Engelbrecht <engeg@google.com>
Reviewed-by: Greg Edelston <gredelston@google.com>
2024-03-20 22:09:14 +00:00
Mike Frysinger
09fc214a79 git: raise soft version to 2.7.4
git-1.9.1 was released in 2014 while git-2.7.4 was released in 2016.
Debian Stretch was released in 2017 and Ubuntu Xenial was released in
2016 which are plenty old at this point.  Both of those include at
least git-2.7.4.

We will start warning users of Debian Jessie (released in 2015 & EOL
in 2020) and Ubuntu Trusty (released in 2014 & EOL Apr 2024) that
they will need to upgrade.

Change-Id: I6be3809bc45968fdcb02cad3f7daf75ded1bb5b1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415137
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2024-03-20 21:11:26 +00:00
Mike Frysinger
3762b17e98 git: raise hard version to 1.7.9
Debian 6 Squeeze went EOL in Feb 2016.  We don't need to carry support
for that anymore as there have been 6 major releases since.  That means
we can bump the required git version from 1.7.2 to 1.7.9.  Ubuntu Precise
shipped with the latter.

git-1.7.2 was released in 2010 while git-1.7.9 was released in 2012.
So that shouldn't be a problem either.  And we've been warning people
using git versions older than 1.9.1 for 3 years now that they need to
upgrade.

Change-Id: I7712f110ea158297b489b8379b112c6700b21a46
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415097
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2024-03-20 19:49:44 +00:00
Mike Frysinger
ae419e1e01 docs: release: add recent git/python/ssh/debian info
Change-Id: I744360b1bfc816e94b3511f0130abb2c08dedd42
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/415117
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2024-03-20 19:49:01 +00:00