Replace tuple returns with namedtuples, to simplify adding new fields.
Extend the Sync_NetworkHalf return value to:
- success: True if successful (the former return value)
- remote_fetched: True if we called `git fetch`
Change-Id: If63c24c2f849523f77fa19c05bbf23a5e9a20ba9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344534
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
By default there are 4 categories in the diffmanifests
api puts the diffs in to - added, removed, changed and unreachable
Example of command - repo diffmanifests 1.xml 2.xml
added - list down the projects present in second manifest but not in
first
removed - list down the projects present in first but not in
second
changed - list down the changes and the differences for each project
unreachable - when it encounters revision value in a project is incorrect
But, when there are projects present in both manifests and could not
find in local workspace where we have cloned the repo(because of
different/subset manifest xml) - this will create unhandled exception
Now we have added a 5th category called 'missing' - where in such
cases it will handle the scenario and print the log for user
Example:
added projects :
project_2 at revision e6c8a59832c05dc4b6a68cee6bc0feb832181725
removed projects :
project_1 at revision e6c8a59832c05dc4b6a68cee6bc0feb832181725
changed projects :
project_3 changed from 3bb890e1286f04e84d505e5db48e0ada89892331 to e434b3736f11537c67590fefadfe4495895e9785
missing projects :
project_4
Change-Id: I244e8389bff7e95664c29d3dcb61e22308e3a573
Signed-off-by: Shashank Devaraj <shashankkarthik@gmail.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344774
Reviewed-by: Mike Frysinger <vapier@google.com>
Since the repo wrapper uses #!/usr/bin/python, use the python3 that
this wrapper is actively using.
Change-Id: I03d1e54418d18a504eec628e549b4cc233621c45
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/345294
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This is ENABLED BY DEFAULT due to data corruption potential. To disable
it, set REPO_BACKUP_OBJECTS=0 in the environment.
While the workspace will grow over time, this provides a recovery path
for an issue where objects are erroneously deleted from the workspace,
resulting in lost work. Once the root cause is determined, we will be
able to stop saving backups again.
Backups are kept in .git/objects/.repo/pack.bak
Bug: https://crbug.com/gerrit/16247
Change-Id: Ib8b5c9b4bf0dfa9e29606e0f5c881d65996b2a40
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/345114
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
This reverts commit 07d21e6bde.
Reason for revert: crbug.com/gerrit/16230, b/244467766 - breaks aosp-master-with-phones case
Change-Id: Id967d92f8622c2c13356b09e46ece9f20040aabc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344314
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
We don't want to keep showing the user config notice when reinitializing
existing checkouts, so hide it.
Change-Id: Id40610bd683396cbff7e1aefc092c8b77c397783
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343536
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
Make it clear to users when we're reinitializing an existing checkout
in case they weren't expecting it.
Bug: https://crbug.com/gerrit/12396
Change-Id: I22e89ae041a8e7b147c9d06a82f1302dd5807ae0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343535
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the specified repo dir doesn't actually have a `repo` program,
we only show a warning before continuing on, and then we fail in
weird ways. Since we really need the repo dir to contain repo,
have this be fatal and delete the results.
Bug: https://crbug.com/gerrit/13526
Change-Id: Icee4cba96136d470cbb459a81918c40205078f98
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343538
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
In case something goes wrong in the initial setup of the repo dir,
clone it into a temporary .repo/repo.tmp/ directory first, and then
rename it only when things have finished fully.
Bug: https://crbug.com/gerrit/13526
Change-Id: Ib0f5a975e4d436b0fb616fac70f5789c4e02a61a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343537
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If stdout or stderr are empty, don't print empty lines.
Also trim any trailing lines so we don't show excess ones.
Change-Id: I65fcea191e1f725be03c064cb510578632466013
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343516
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
After reworking this function to use subprocess for output capturing
in commit c87c1863b1 ("git_command:
switch process capturing over to subprocess"), passing input via
stdin write no longer works. We have to pass it via communicate(),
and we have to pass it a string instead of bytes (since we always
use encoding='utf-8' now). This is fine since the only user of the
input= setting today is already passing in a string.
Bug: https://crbug.com/gerrit/16151
Change-Id: Ic58db1e568b8f8aa840a6d62c5a157c14aa6d9bc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343515
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
If initializing the manifest fails for any reason, don't leave it in
a half complete state. This can cause problems if/when the user tries
to reinit because different codepaths will be taken. For example, if
we initialize manifests.git and don't finish probing the remote to see
what default branch it uses, we end up always using "master" even if
that isn't what the remote uses.
To avoid all of this, use .tmp dirs when initializing, and rename to
the final path only after we complete all the right steps.
We should roll this out to all projects we clone, but start with the
manifest project for now.
Bug: https://crbug.com/gerrit/13526
Bug: https://crbug.com/gerrit/15805
Change-Id: I0214338de69ee11e090285c6b0b211052804af06
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343539
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
People rarely care about the history of the manifest repo. Change
the default to 1 to speed up initial setup. If people really want
the full history, they can pass --manifest-depth=0.
We can't reuse the existing --depth option because that applies to
all projects we clone, not just the manifest repo.
Bug: https://crbug.com/gerrit/16193
Change-Id: Ideab1712e9ffc743b9bae97903e074d7d429b134
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343435
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
People rarely care about having all manifest branches locally. Change
the default to only pull down the selected branch. If people want other
branches, the -b option will fetch it automatically, or people can use
--no-current-branch.
This only applies to the manifest project syncing, not the rest of the
projects that are in the checkout.
Bug: https://crbug.com/gerrit/16194
Change-Id: Ia9e7e2f23b8028d82772db72dcc7d6c32260be79
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343434
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
Normally git produces no output when creating or switching branches.
If there's a problem though, we want to show that to the user. So
switch from capturing all output to running in quiet mode.
Bug: https://crbug.com/gerrit/15819
Change-Id: I7873ecc7c3bacce591899cc9471cb0244eb74541
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343454
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If the user's pager settings are broken, display an error message
rather than crash to avoid confusing them.
Bug: https://crbug.com/gerrit/16173
Change-Id: Idc0891da783c68f3a96ac53a82781e34e40421fb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343437
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We almost always use self.remote.name when calling self.GetRemote,
so make that the default to simplify the code a bit.
Change-Id: Ifdf6e1370d6b8963b44e6d384b0fac8fa5c4f2ba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343184
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
The `--wip` allow to bulk push changed as work-in-progress. This CL
intend to allow the opposite opperation by removing the wip mark on the
CL and set it to be ready to review
Change-Id: If0743c5b14829f77be2def5a8547060d06a5648c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/342214
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: William Escande <wescande@google.com>
If the server responds with an HTML page, we should show that to the
user instead of crashing with XML errors.
Bug: https://crbug.com/gerrit/15936
Change-Id: I52e6b781c3bb6a6c9f6ecbe2e0907044876cdc8d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337519
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
For projects that no longer share their per-project objects directly, we
no longer have to disable the git settings that disable pruning. See
commit "project: stop directly sharing objects/ between shared projects"
for more details.
Bug: https://crbug.com/gerrit/15553
Change-Id: Ica0e83c3002716424c2bc9111b3b3d3a76c30973
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337535
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
For --jobs-network, the logic is now:
* If the user specifies --jobs-network, use that.
* Else, if the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to 1.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.
For --jobs-checkout, the logic is now:
* If the user specifies --jobs-checkout, use that.
* Else, if the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to DEFAULT_LOCAL_JOBS which is based on user's ncpus.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.
For garbage collecting, the logic is now:
* If the user specifies --jobs, use that.
* Else, if the manifest specifies sync-j, use that.
* Else, default to the user's ncpus.
Then we limit the jobs count based on the softlimit RLIMIT_NOFILE.
Having to factor in the manifest settings makes this more complicated
which is why we delay processing of defaults until after we've synced
the manifest projects.
Bug: http://b/239712300
Change-Id: Id27cda63c76c156f1d63f6a20cb2c4ceeb3d547c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/341394
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
Apparently, manifest.topdir already contains a trailing slash in some
cases, so a simple string concatenation may or not lead to double
slashes. It is safer to use os.path.join. See
https://screenshot.googleplex.com/6pSY3QewAeCdAqk
Change-Id: I2411452296b7e78fc975787b675273a48d6b3d85
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/341574
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mateus Azis <azis@google.com>
When we create superproject_override.xml, do not include projects that
are present from local_manifests/*. Such projects are fully under the
control of the local_manifests/ file.
Bug: b/238934278
Test: manual, ./run_tests
Change-Id: I40382ceb82d9cf7b8dc7b5f2abed3f6d4d80017e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340877
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Reviewed-by: Sam Saccone 🐐 <samccone@google.com>
Display the project path relative to the outermost manifest by default,
and relative to the sub manifest only when --this-manifest-only is
specified.
For project-related diagnostic messages, use the outermost manifest for
messages.
Change-Id: I4537d7dd412a2c182e77d6720e95c1b0ef70eb0e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340754
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
This option will suppress the
Uncommitted changes in ... (did you forget to amend?)
prompt when there are untracked (unknown) files in the working copy.
The prompt is still shown if tracked files are modified.
Change-Id: Ia3fcc82989b7fad09b69214eda31e2d0dfc14600
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340456
Tested-by: Martin Geisler <mgeisler@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If the user says `--no-use-superproject`, then do not bother syncing the
superproject.
Also add/update docstrings and comments throughout.
Change-Id: I9cdad706130501bab9a22d3099a1dae605e9c194
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/338975
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
With this change, partial syncs (sync with a project list) are again
supported.
If the updated manifest includes new sub manifests, download them
inheriting options from the parent manifestProject.
Change-Id: Id952f85df2e26d34e38b251973be26434443ff56
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334819
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
For now, this is opt-in via environment variables:
- export REPO_USE_ALTERNATES=1
The shared project logic that shares the internal .git/objects/ dir
directly between multiple projects via the project-objects/ tree has
a lot of KI with random corruption. It all boils down to projects
sharing objects/ but not refs/. Git operations that use refs to see
what objects are reachable and discard the rest can easily discard
objects that are used by other projects.
Consider this project layout:
<show fs layout>
There are unique refs in each of these trees that are not visible in
the others. This means it's not safe to run basic operations like
git prune or git gc.
Since we can't share refs (each project needs to have unique refs
like HEAD in order to function), let's change how we share objects.
The old way involved symlinking .git/objects/ to the project-objects
tree. The new way shares objects using git's info/alternates.
This means project-objects/ will only contain objects that exist in
the remote project. Local per-project objects (like when creating
branches and making changes) will never be shared. When running a
prune or gc operation in the per-project state, it will only ever
repack or discard those per-project objects. The common shared
objects would only be cleaned up when running a common operation
(i.e. by repo itself).
One downside to this for users is if they try blending unrelated
upstream projects. For example, in CrOS we have multiple kernel
projects (for diff versions) checked out. If a dev fetched the
upstream Linus tree into one of them, the objects & tags would
not be shared with the others, so they would have to fetch the
upstream state for each project. Annoying, but better than the
current corruption situation we're in now.
Also if the dev runs a manual `git fetch` in the per-project to
sync it up to newer state than the last `repo sync` they ran,
the objects would get duplicated. However, git operations later
on should eventually dedupe this.
Bug: https://crbug.com/gerrit/15553
Change-Id: I313a9b8962f9d439ef98ac0ed37ecfb9e0b3864e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328101
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
This way we know we don't need to encode the labels.
Change-Id: Ib83ed8f4ed05f00b9d2d06a9dd3f304e4443430e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337518
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: LaMont Jones <lamontjones@google.com>
Now that we don't need to support Python 2, we can switch to this
API for better contextmanager logic.
Change-Id: I2d03e391121886547e7808a3b5c3b470c411533f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337515
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
If init was run with --use-superproject, init failed.
If init was run without --{no,}use-superproject option then manifests
with <superproject/> elements were mishandled.
Bug: b/233226285
Test: manual
Change-Id: I737e71c89d2d7c324114f58bf2dc82b40e5beba7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337534
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
When the user does not specify any manifest groups, this allows the
parent manifest to indicate which manifest groups should be used for
syncing the submanifest.
Change-Id: I88806ed35013d13dd2ab3cd245fcd4f9061112c4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335474
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Rather than erase the entire line first then print out the new content,
print out the new content on top of the old and then erase anything we
didn't update. This should result in a lot less flashing with faster
terminals.
Bug: https://crbug.com/gerrit/11293
Change-Id: Ie2920b0bf3d5e6f920b8631a1c406444b23cd12d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335214
Reviewed-by: LaMont Jones <lamontjones@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
When a manifest file is overridden, remember that and keep using the
override for the remainder of the process. If we need to revert it,
make the override name evaluate False.
Change-Id: I1eee05fec6988c1ee4a3c751c4b540d5b5d11797
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335136
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
We need to iterate over multiple manifests, and generally use the
outer_client.manifest for multi-manifest support. This refactors the
use of self.manifest into a chosen manifest.
Change-Id: I992f21d610c929675e99555ece9c38df4b635839
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334699
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Make git_superproject independent of the command line by passing
the specific value instead of requiring the caller to have an
optparse.Values object to pass in.
Flag --use-superproject and --archive as incompatible in subcmds/init.py
Change-Id: Ied7c874b312e151038df903c8af4328f070f387c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335135
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Raman Tenneti <rtenneti@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
