Commit Graph

456 Commits

Author SHA1 Message Date
William Escande
ac76fd3e3a upload: Add ready flag to remove wip
The `--wip` allow to bulk push changed as work-in-progress. This CL
intend to allow the opposite opperation by removing the wip mark on the
CL and set it to be ready to review

Change-Id: If0743c5b14829f77be2def5a8547060d06a5648c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/342214
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: William Escande <wescande@google.com>
2022-08-03 20:17:06 +00:00
LaMont Jones
8501d4602a status, diff: display correct path for multi-manifest
Display the project path relative to the outermost manifest by default,
and relative to the sub manifest only when --this-manifest-only is
specified.

For project-related diagnostic messages, use the outermost manifest for
messages.

Change-Id: I4537d7dd412a2c182e77d6720e95c1b0ef70eb0e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340754
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-07-14 16:00:18 +00:00
Martin Geisler
8db78c7d4d project: simplify if-statement
Change-Id: I05e4505b45963fe6e85cf74a669afafd00fc83c0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340457
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Martin Geisler <mgeisler@google.com>
2022-07-11 17:58:06 +00:00
Martin Geisler
9fb64ae29c upload: add ‘--ignore-untracked-files’ option
This option will suppress the

    Uncommitted changes in ... (did you forget to amend?)

prompt when there are untracked (unknown) files in the working copy.
The prompt is still shown if tracked files are modified.

Change-Id: Ia3fcc82989b7fad09b69214eda31e2d0dfc14600
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340456
Tested-by: Martin Geisler <mgeisler@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-07-11 17:57:43 +00:00
LaMont Jones
68d69635c7 Fix Projects.shareable_dirs
If this tree is not using alternates for object sharing, then we need to
continue to call it a shared directory.

Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=15982
Test: manual
Change-Id: I1750f10b192504ac67f552222f8ddb9809d344fe
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/338974
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-06-08 16:49:08 +00:00
LaMont Jones
ff6b1dae1e Only sync superproject if it will be used.
If the user says `--no-use-superproject`, then do not bother syncing the
superproject.

Also add/update docstrings and comments throughout.

Change-Id: I9cdad706130501bab9a22d3099a1dae605e9c194
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/338975
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-06-08 16:49:08 +00:00
LaMont Jones
bdcba7dc36 sync: add multi-manifest support
With this change, partial syncs (sync with a project list) are again
supported.

If the updated manifest includes new sub manifests, download them
inheriting options from the parent manifestProject.

Change-Id: Id952f85df2e26d34e38b251973be26434443ff56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334819
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-05-26 00:03:37 +00:00
Mike Frysinger
1d00a7e2ae project: initial separation of shared project objects
For now, this is opt-in via environment variables:
  - export REPO_USE_ALTERNATES=1

The shared project logic that shares the internal .git/objects/ dir
directly between multiple projects via the project-objects/ tree has
a lot of KI with random corruption.  It all boils down to projects
sharing objects/ but not refs/.  Git operations that use refs to see
what objects are reachable and discard the rest can easily discard
objects that are used by other projects.

Consider this project layout:
<show fs layout>

There are unique refs in each of these trees that are not visible in
the others.  This means it's not safe to run basic operations like
git prune or git gc.

Since we can't share refs (each project needs to have unique refs
like HEAD in order to function), let's change how we share objects.
The old way involved symlinking .git/objects/ to the project-objects
tree.  The new way shares objects using git's info/alternates.

This means project-objects/ will only contain objects that exist in
the remote project.  Local per-project objects (like when creating
branches and making changes) will never be shared.  When running a
prune or gc operation in the per-project state, it will only ever
repack or discard those per-project objects.  The common shared
objects would only be cleaned up when running a common operation
(i.e. by repo itself).

One downside to this for users is if they try blending unrelated
upstream projects.  For example, in CrOS we have multiple kernel
projects (for diff versions) checked out.  If a dev fetched the
upstream Linus tree into one of them, the objects & tags would
not be shared with the others, so they would have to fetch the
upstream state for each project.  Annoying, but better than the
current corruption situation we're in now.

Also if the dev runs a manual `git fetch` in the per-project to
sync it up to newer state than the last `repo sync` they ran,
the objects would get duplicated.  However, git operations later
on should eventually dedupe this.

Bug: https://crbug.com/gerrit/15553
Change-Id: I313a9b8962f9d439ef98ac0ed37ecfb9e0b3864e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328101
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-05-26 00:02:18 +00:00
Mike Frysinger
3a0a145b0e upload: move label validation to core function
This way we know we don't need to encode the labels.

Change-Id: Ib83ed8f4ed05f00b9d2d06a9dd3f304e4443430e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337518
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
2022-05-21 19:19:44 +00:00
LaMont Jones
0ddb677611 project: fix --use-superproject logic for init.
If init was run with --use-superproject, init failed.

If init was run without --{no,}use-superproject option then manifests
with <superproject/> elements were mishandled.

Bug: b/233226285
Test: manual
Change-Id: I737e71c89d2d7c324114f58bf2dc82b40e5beba7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337534
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-05-20 11:01:28 +00:00
LaMont Jones
501733c2ab manifest: add submanifest.default_groups attribute
When the user does not specify any manifest groups, this allows the
parent manifest to indicate which manifest groups should be used for
syncing the submanifest.

Change-Id: I88806ed35013d13dd2ab3cd245fcd4f9061112c4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335474
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-29 18:42:23 +00:00
LaMont Jones
0165e20fcc project: Do not exit early on --standalone-manifest.
After we successfully download the standalone manifest file, we cannot
exit early.

Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=15861
Change-Id: Ic47c9f7e9921851f94c6f24fd82b896eff524037
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335974
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-29 17:13:49 +00:00
LaMont Jones
0de4fc3001 project: Add missing imports
Some imports were missed when moving manifestProject to project.py

Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=15861
Change-Id: Id8fffeaa3f88f344a13b5ab44e5403c7edd98f31
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335554
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
2022-04-21 18:44:26 +00:00
LaMont Jones
b90a422ab6 Override the manifest for the entire command
When a manifest file is overridden, remember that and keep using the
override for the remainder of the process.  If we need to revert it,
make the override name evaluate False.

Change-Id: I1eee05fec6988c1ee4a3c751c4b540d5b5d11797
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335136
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-19 21:28:20 +00:00
LaMont Jones
4ada043dc0 ManifestProject: add manifest_platform
And fix most of the other attributes to return the value instead of
None.

Change-Id: Iddcbbeb56238ee082bb1cae30adbd27a2f551f3d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335134
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2022-04-14 20:56:45 +00:00
LaMont Jones
d56e2eb421 manifest_xml: use Superproject to hold XML content
Always create Superproject when there is a <superproject> tag, and have
it hold the XML content, similar to how other manifest elements are
handled.

This also adds SetQuiet and SetPrintMessages to Superproject
consistent with manifest.SetUseLocalManifests.

Change-Id: I522bf3da542006575799f0640c67f7052704f266
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334641
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-04-12 15:46:23 +00:00
LaMont Jones
a2ff20dd20 manifest_xml: Add Load and Unload methods
- do not call the internal method from subcmds/sync.py.
- use the correct default groups for submanifests.
- only sync the superproject when we are told to.

Change-Id: I81e4025058f1ee564732b9e17aecc522f6b5f626
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334639
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-04-08 19:52:04 +00:00
LaMont Jones
55ee304304 Fix sub manifest handling
Also fixes some typos

Change-Id: Id2ba5834ba3a74ed3f29c36d2c0030737dc63e35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334579
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-06 21:04:46 +00:00
LaMont Jones
409407a731 init: add multi-manifest support
This moves more of the manifest project handling into ManifestProject.

Change-Id: Iecdafbec18cccdfd8e625753c3bd1bcddf2b227f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334520
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-06 17:02:40 +00:00
LaMont Jones
d82be3e672 Move manifest config logic into ManifestProject
Use ManifestProject properties for config values.

Change-Id: Ib4ad90b0d9a089916e35615b8058942e6d01dc04
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334519
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-06 16:59:45 +00:00
LaMont Jones
9b03f15e8e project: add ManifestProject.Sync()
Move the logic to sync a ManifestProject out of subcmds/init.py

Change-Id: Ia9d00f3da1dc3c5dada84c4d19cf9802c2346cb0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334140
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-04-01 15:48:04 +00:00
LaMont Jones
9b72cf2ba5 project: Isolate ManifestProject from RepoProject
Create RepoProject and ManifestProject, inheriting from MetaProject,
  with methods separated for isolation and clarity.

Change-Id: Ic1d6efc65c99470290fea612e2abaf8670d199f4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334139
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-03-31 21:02:52 +00:00
LaMont Jones
cc879a97c3 Add multi-manifest support with <submanifest> element
To be addressed in another change:
 - a partial `repo sync` (with a list of projects/paths to sync)
   requires `--this-tree-only`.

Change-Id: I6c7400bf001540e9d7694fa70934f8f204cb5f57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322657
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-02-17 21:57:55 +00:00
LaMont Jones
adaa1d8734 project.py: pass --recurse-submodules={value}
If submodules is False, explicitly pass '=no'.  Uninitialized submodules
may cause the default option to fail.

Change-Id: Ia00bcba5b69c4b65195f4c469c686a3ef9a4a3ad
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/330159
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
2022-02-10 23:57:45 +00:00
Mike Frysinger
8e91248655 project: mark gc.log as safe to discard when migrating .git/
This is just a log file that, while useful for humans when gc aborts,
doesn't contain any data, so it's safe to throw away.

Bug: https://crbug.com/gerrit/15619
Change-Id: Ia95e0e281f52260668f7a80b5d5f990e32a8597a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328999
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-01-26 16:46:03 +00:00
XD Trol
630876f9e4 init: add an option --enable-git-lfs-filter
It was reported that git-lfs did not work with git-repo. Specifically,
`git read-tree -u` run by `repo sync` would fail git-lfs's smudge
filter. See https://github.com/github/git-lfs/issues/1422.

In fact, by the time `git read-tree -u` is run, the repository is not
bare. It is just that, the working directory is not the same as the
.git directory. git-lfs's filter should work. No one seems to have
delved into that issue.

Today, with newer versions of git-repo and git-lfs, that issue will
not reproduce. Tested with
- git 2.33, git-lfs 2.13 on macOS
- git 2.17, git-lfs 2.3 on ubuntu

So, it seems fine to add an option --enable-git-lfs-filter, default to
false, and stat that it may not work with older versions of git and
git-lfs in the help doc.

Bug: https://crbug.com/gerrit/14516
Change-Id: I8d21854eeeea541e072f63d6b10ad1253b1a9826
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328359
Tested-by: XD Trol <milestonejxd@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-01-26 01:47:20 +00:00
Peter Kjellerstedt
b550501254 project: Ignore failure to remove the sample hooks
Removing the sample hooks is just clean up, so if repo cannot remove a
sample hook that should not cause it to fail.

Change-Id: I716b977da091c22b8f53e134f4fbc114116f9a65
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328635
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-01-22 06:13:10 +00:00
Mike Frysinger
67d6cdf2bc project: store objects in project-objects directly
In order to stop sharing objects/ directly between shared projects,
we have to fetch the remote objects into project-objects/ manually.
So instead of running git operations in the individual project dirs
and relying on .git/objects being symlinked to project-objects/,
tell git to store any objects it fetches in project-objects/.

We do this by leveraging the GIT_OBJECT_DIRECTORY override.  This
has been in git forever, or at least since v1.7.2 which is what we
already hard require.  This tells git to save new objects to the
specified path no matter where it's being run otherwise.

We still otherwise run git in the project-specific dir so that it
can find the right set of refs that it wants to compare against,
including local refs.  For that reason, we also have to leverage
GIT_ALTERNATE_OBJECT_DIRECTORIES to tell git where to find objects
that are not in the upstream remote.  This way git doesn't blow up
when it can't find objects only associated with local commits.

As it stands right now, the practical result is the same: since we
symlink the project objects/ dir to the project-objects/ tree, the
default objects dir, the one we set $GIT_OBJECT_DIRECTORY to, and
the one we set $GIT_ALTERNATE_OBJECT_DIRECTORIES to are actually
all the same.  So this commit by itself should be safe.  But in a
follow up commit, we can replace the symlink with a separate dir
and git will keep working.

Bug: https://crbug.com/gerrit/15553
Change-Id: Ie4e654aec3e1ee307eee925a54908a2db6a5869f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328100
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-19 17:24:51 +00:00
Mike Frysinger
152032cca2 project: move --reference handling to project-objects
When using --reference, the path is written to objects/info/alternates.
The path is accessed inconsistently -- sometimes through projects/ (via
self.gitdir) and sometimes through project-objects/ (via self.objdir).
This works because projects/.../objects is a symlink to the objects dir
under project-objects/.  Change all accesses to go through self.objdir.
This will allow us to stop symlinking projects/.../objects without the
reference dir logic breaking.  The projects/ path is going to use its
alternates file for its own needs.

Bug: https://crbug.com/gerrit/15553
Change-Id: I6b452ad1aaffec74ecb7ac1bb9baa3a3a52e076c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328099
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jack Neus <jackneus@google.com>
2022-01-13 18:27:28 +00:00
Mike Frysinger
98bb76577d project: prune sample hooks
These hooks are never used and often get stale, so just trim them.
Users rarely look in these dirs to begin with.

Change-Id: Ic785aa55fb7ec84a61376df101127d0018882030
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327538
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-10 17:41:45 +00:00
Mike Frysinger
d33dce0b77 project: drop support for symlinking internal .git files
Since we don't do this anymore, and there prob won't be a need to
bring it back, drop support for it.

Bug: https://crbug.com/gerrit/15460
Change-Id: I7d86706f108c797a5c7962cb1578693d49430367
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327537
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-10 17:41:40 +00:00
Mike Frysinger
89ed8acdbe project: abort a bit earlier before migrating .git/
Verify all the .git/ paths will be handled by the migration logic before
starting the migration.  This way we still abort & log an error, but the
user gets to see it before we put the tree into a state that they have to
manually recover.  Also add a few more known-safe-to-clobber paths.

Bug: https://crbug.com/gerrit/15273
Change-Id: If49d69b341bc960ddcafa30da333fb5ec7145b51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327557
Reviewed-by: Colin Cross <ccross@android.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-07 20:17:14 +00:00
Mike Frysinger
13576a8caf project: stop symlinking info dir under .git/
Unsharing this directory shouldn't be a problem.  The current repo code
treated it as a file, and while that's actually incorrect, files & dirs
are basically treated the same, so it's practically the same.

Let's enumerate each subpath since there aren't that many.

info/refs:
Only used when the project is exported over git dumb transports (i.e.
a http:// server).  Repo never does this, and it's extremely unlikely
any user has ever done this.  Plus, this proposal talks about unsharing
project refs, so this file should get unshared too.

info/grafts:
A user-configurable file that repo never touches.  Might be useful to
share across projects, but probably rarely (if ever) used by developers,
and forcing them to configure it for each project isn't that big of a
deal.

info/exclude:
info/attributes:
User-configurable files that repo never touches.  Doesn't seem like
most users ever touch these, and if they do, having them do it for
each shared project isn't a big deal.

info/sparse-checkout:
Repo doesn't use sparse checkouts, and it's extremely unlikely to even
work if a user tried doing something themselves.

Bug: https://crbug.com/gerrit/15460
Change-Id: I53e44d73a6d7a92da615b46600d8ea51cb46e3ac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327519
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-06 08:31:45 +00:00
Mike Frysinger
2345906d04 project: stop symlinking description file under .git/
Nothing uses this path.  It’s only for exporting git dirs e.g. for
online gitweb use which probably no one does.  It is not the same
description file as exists on servers we cloned from.  Leaving it
as the default plain text file will simplify code.

We don't undo any existing symlinks if they exist since repo does
not care about them, and their existence doesn't hurt.

Bug: https://crbug.com/gerrit/15460
Change-Id: Ic34fe7c3cfb8f6da844de5be30158f59382b1cc8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327518
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-06 08:29:06 +00:00
Mike Frysinger
41289c62b4 project: stop symlinking svn under .git/
This path only matters to users of `git svn` who manually run it in
local projects after they get a full repo client checkout.  With svn
usage falling in general, and with the fact that the source checkout
now symlinks its .git/ state to the internal projects/ path, we don't
need to manage this anymore.

It means the path won't be shared among multiple local projects that
have the same remote, but so it goes.  It was an optimization only,
not functionality required for correctness.  We want to simplify the
internals to stop messing with git state, and this particular path
doesn't seem worth the effort to maintain.

We don't undo any existing svn symlinks if they exist since repo does
not care about them, and their existence doesn't hurt anything.

Bug: https://crbug.com/gerrit/15460
Change-Id: Ie8496b275bcc589771aa9f4ee874ed2ee6d5241d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327517
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-06 08:28:37 +00:00
Mike Frysinger
c72bd8486a project: clean up now unused code
Now that we symlink worktree .git/ paths to .repo/projects/, we never
set share_refs=True anywhere, which means all of this logic is dead
code.  Throw it all away.  Do it as a separate commit to make the
parent commit easier to review.

Bug: https://crbug.com/gerrit/15273
Change-Id: If496d39029d3d3bd523ba24c603ce47a63ad9b51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/326817
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jack Neus <jackneus@google.com>
2022-01-06 04:08:05 +00:00
Mike Frysinger
2a089cfee4 project: migrate worktree .git/ dirs to symlinks
Historically we created a .git/ subdir in each source checkout and
symlinked individual files to the .repo/projects/ paths.  This layer
of indirection isn't actually needed: the .repo/projects/ paths are
guaranteed to only ever have a 1-to-1 mapping with the actual git
checkout.  So we don't need to worry about having files in .git/ be
isolated.

To that end, change how we manage the actual project checkouts from
a dir full of symlinks (and a few files) to a symlink to the internal
.repo/projects/ dir.  This makes the code simpler & faster.

The directory structure we have today is:
.repo/
  project-objects/chromiumos/third_party/kernel.git/
    <paths omitted as not relevant to this change>
  projects/src/third_party/kernel/
    v3.8.git/
      config
      description   -> …/project-objects/…/config
      FETCH_HEAD
      HEAD
      hooks/        -> …/project-objects/…/hooks/
      info/         -> …/project-objects/…/info/
      logs/
      objects/      -> …/project-objects/…/objects/
      packed-refs
      refs/
      rr-cache/     -> …/project-objects/…/rr-cache/
src/third_party/kernel/
  v3.8/
    .git/
      config        -> …/projects/…/v3.8.git/config
      description   -> …/project-objects/…/v3.8.git/description
      HEAD
      hooks/        -> …/project-objects/…/v3.8.git/hooks/
      index
      info/         -> …/project-objects/…/v3.8.git/info/
      logs/         -> …/projects/…/v3.8.git/logs/
      objects/      -> …/project-objects/…/v3.8.git/objects/
      packed-refs   -> …/projects/…/v3.8.git/packed-refs
      refs/         -> …/projects/…/v3.8.git/refs/
      rr-cache/     -> …/project-objects/…/v3.8.git/rr-cache/

The directory structure we have after this commit:
.repo/
  <nothing changes>
src/third_party/kernel/
  v3.8/
    .git            -> …/projects/…/v3.8.git

Bug: https://crbug.com/gerrit/15273
Change-Id: I9dd8def23fbfb2f4cb209a93f8b1b2b24002a444
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323695
Reviewed-by: Mike Nichols <mikenichols@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-12-01 15:27:16 +00:00
Robin Schneider
9c1fc5bc5d sync: Handle tag ref in "upstream" field
repo sync only handles a git tag properly when it is in the "revision"
field. However, "revision locked manifests" (`repo manifest
--revision-as-HEAD`) specifies the tag in the "upstream" field. The
issue is that this tag is not fetched. Only the commit that the tag
points to is fetched. This cases issues as
self._CheckForImmutableRevision() runs and comes to the conclusion that
the tag was changed while in fact, it was just not fetched. This causes
a full sync.

File docs/manifest-format.md, section Element-project:
> Attribute upstream: Name of the Git ref in which a sha1 can be found.
Used when syncing a revision locked manifest in -c mode to avoid having
to sync the entire ref space. Project elements not setting their own
upstream will inherit this value.

Change-Id: I0507d3a5f30aee8920a9f820bafedb48dd5db554
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323620
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Robin Schneider <ypid@riseup.net>
2021-11-16 20:43:57 +00:00
Mike Frysinger
333c0a499b project: init hooks in objdir only
objdir is the .repo/project-objects/ dir based on the remote path.
gitdir is the .repo/projects/ dir based on the local source checkout
path.  When we setup the gitdir, we symlink "hooks" to the one in the
objdir.  But when we go to initialize the hooks, we do it via gitdir.
There is a 1-to-many mapping from project-objects to projects, so
initializing via gitdir can be repetitive.  Collapse the hook init
logic to the objdir init path.

Change-Id: I828fca60ce6e125d6706c709cdb2797faa40aa50
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323815
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-11-15 19:50:18 +00:00
LaMont Jones
7b9b251a5e project: fix format string in error message
BUG=None

Change-Id: I0b195fd919c6db8cb3547e8d6f4c733f2bd4a535
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322735
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2021-11-05 16:59:14 +00:00
Michael Kelly
2f3c3316e4 Update revisionId if required when using extend-project
When a hard revision ID is provided in a regular project tag then the
revisionId is updated as well if it is a commit hash.  The difference
is that if the revisionExpr is a commit, git-repo needs to update
refs/remotes/m/master with update-ref not symbolic-ref, as the latter
must refer to another ref, not to a specific commit.

Change-Id: I215a62dabb30225e480ad2c731416d775fc0c750
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310963
Tested-by: Michael Kelly <mkelly@arista.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-09-28 20:12:00 +00:00
Michael Kelly
37c21c268b Add 'dest-path' to extend-project to support changing path
This allows us to move the repository to a new location in the source
tree without having to remove-project + add a new project tag.

Change-Id: I4dba6151842e57f6f2b8fe60cda260ecea68b7b4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310962
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Michael Kelly <mkelly@arista.com>
2021-09-28 20:12:00 +00:00
Mike Frysinger
9d96f58f5f make file removal a bit more robust
Some of the file removal calls are subject to race conditions (if
something else deletes the file), so extend our remove API to have
an option to ignore ENOENT errors.  Then update a bunch of random
call sites to use this new functionality.

Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195
Reviewed-by: Sean McAllister <smcallis@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-09-28 16:06:50 +00:00
Jack Neus
6ea0caea86 repo: properly handle remote annotations in manifest_xml
BUG=b:192664812
TEST=tests/

Change-Id: I1aa50260f4a00d3cebbd531141e1626825e70127
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312643
Tested-by: Jack Neus <jackneus@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2021-07-23 18:03:11 +00:00
Xin Li
8e983bbc0f Force a fetch when superproject has a newer SHA1 for remote branch.
For older git-repo versions, we might have only fetched the SHA1
revision that was provided by the project, but have remote branch left
intact as long as they exist. When the remote branch become stale,
some repo operations like rebase would fail, and repo sync would not
correct this situation.

Fix this by tightening the requirement to also require the superproject
provided SHA1 be an ancestor or equal to the tip-of-tree of the remote
branch.

Bug: [google internal] b/193798453
Change-Id: Ie34c5d860dabb1cbd9f822da929088ec69c79cf6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312642
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
2021-07-20 23:26:01 +00:00
Xin Li
0e776a5837 Fix an issue when syncing with --use-superproject and clone bundles.
It is possible that a clone bundle contained the object referenced by
the branch in the manifest and in the superproject, but not the branch
itself (for example, the branch may be newly created from an existing
branch, or is not vislble to the user downloading the clone bundle).

When --use-superproject is enabled, because we are overriding
revisionExpr with the SHA1 revision provided by the superproject, the
verification step would succeed, but because the expected branch do not
exist, it would confuse git-repo at a later time, as it is expecting the
remote branch to exist in the local clone.

In project.py, fix this by making SetRevisionId() to always remember
the actual branch name and verify it in _CheckForImmutableRevision()
so that we only skip the fetch step when both objects exists locally.

Bug: [google internal] b/191974277
Change-Id: I49d3ca0667f524c8c45f416492faf95b1dd822fb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310802
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
2021-06-30 15:31:15 +00:00
Mike Frysinger
b16b9d26bd project: fix error display when output_redir is disabled
We always pass in output_redir when syncing, but that's the common case:
there are a few situations (like `repo init`) where we don't pass in a
buffer, and if any errors show up in that case, we'd crash.  Rely on the
print function to handle this logic for us.

Bug: https://crbug.com/gerrit/14568
Change-Id: I8cd47e82329797ffc42534418a3dfbd8429205be
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/307222
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-20 06:15:38 +00:00
Mike Frysinger
339f2df1dd ssh: rewrite proxy management for multiprocessing usage
We changed sync to use multiprocessing for parallel work.  This broke
the ssh proxy code as it's all based on threads.  Rewrite the logic to
be multiprocessing safe.

Now instead of the module acting as a stateful object, callers have to
instantiate a new ProxyManager class that holds all the state, an pass
that down to any users.

Bug: https://crbug.com/gerrit/12389
Change-Id: I4b1af116f7306b91e825d3c56fb4274c9b033562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305486
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
2021-05-10 21:16:06 +00:00
Mike Frysinger
19e409c818 ssh: move proxy usage to the sync subcommand
The only time we really need ssh proxies is when we want to run many
connections and reuse them.  That only happens when running sync.
Every other command makes at most two connections, and even then it's
only one or none.  So the effort of setting up & tearing down ssh
proxies isn't worth it most of the time.

The big reason we want to move this logic to sync is that it's now
using multiprocessing for parallel work.  The current ssh proxy code
is all based on threads, which means none of the logic is working
correctly.  The current ssh design makes it hard to fix when all of
the state lives in the global/module scope.

So the first step to fixing this is top move the setup & teardown to
the one place that really needs it: sync.  No other commands will use
proxies anymore, just direct connections.

Bug: https://crbug.com/gerrit/12389
Change-Id: Ibd351acdec39a87562b3013637c5df4ea34e03c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305485
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-10 21:10:29 +00:00
Mike Frysinger
aedd1e5ef0 sync: fix print error when handling server error
When converting this logic from print() to the output buffer, this
error codepath should have dropped the use of the file= redirect.

Bug: https://crbug.com/gerrit/14482
Change-Id: Ib484924a2031ba3295c1c1a5b9a2d816b9912279
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305142
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-05-04 16:40:28 +00:00