From dfdf577e98f6e9b13a3236767316863b3a995c01 Mon Sep 17 00:00:00 2001 From: Mike Frysinger Date: Thu, 30 Jan 2025 19:11:36 -0500 Subject: [PATCH] docs: smart-sync: split out & expand details The existing documentation on smart-sync behavior is a bit light on details, and out of date wrt what the code actually does. Start a dedicated document and fill it out more. Change-Id: I1a8a3ac6edf9291d72182ad55db865035d9b683e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/450002 Commit-Queue: Mike Frysinger Tested-by: Mike Frysinger Reviewed-by: Josip Sokcevic --- docs/manifest-format.md | 21 +------ docs/smart-sync.md | 129 ++++++++++++++++++++++++++++++++++++++++ subcmds/sync.py | 1 + 3 files changed, 131 insertions(+), 20 deletions(-) create mode 100644 docs/smart-sync.md diff --git a/docs/manifest-format.md b/docs/manifest-format.md index cfb80164..71fa04c5 100644 --- a/docs/manifest-format.md +++ b/docs/manifest-format.md @@ -231,26 +231,7 @@ At most one manifest-server may be specified. The url attribute is used to specify the URL of a manifest server, which is an XML RPC service. -The manifest server should implement the following RPC methods: - - GetApprovedManifest(branch, target) - -Return a manifest in which each project is pegged to a known good revision -for the current branch and target. This is used by repo sync when the ---smart-sync option is given. - -The target to use is defined by environment variables TARGET_PRODUCT -and TARGET_BUILD_VARIANT. These variables are used to create a string -of the form $TARGET_PRODUCT-$TARGET_BUILD_VARIANT, e.g. passion-userdebug. -If one of those variables or both are not present, the program will call -GetApprovedManifest without the target parameter and the manifest server -should choose a reasonable default target. - - GetManifest(tag) - -Return a manifest in which each project is pegged to the revision at -the specified tag. This is used by repo sync when the --smart-tag option -is given. +See the [smart sync documentation](./smart-sync.md) for more details. ### Element submanifest diff --git a/docs/smart-sync.md b/docs/smart-sync.md new file mode 100644 index 00000000..1769572f --- /dev/null +++ b/docs/smart-sync.md @@ -0,0 +1,129 @@ +# repo Smart Syncing + +Repo normally fetches & syncs manifests from the same URL specified during +`repo init`, and that often fetches the latest revisions of all projects in +the manifest. This flow works well for tracking and developing with the +latest code, but often it's desirable to sync to other points. For example, +to get a local build matching a specific release or build to reproduce bugs +reported by other people. + +Repo's sync subcommand has support for fetching manifests from a server over +an XML-RPC connection. The local configuration and network API are defined by +repo, but individual projects have to host their own server for the client to +communicate with. + +This process is called "smart syncing" -- instead of blindly fetching the latest +revision of all projects and getting an unknown state to develop against, the +client passes a request to the server and is given a matching manifest that +typically specifies specific commits for every project to fetch a known source +state. + +[TOC] + +## Manifest Configuration + +The manifest specifies the server to communicate with via the +the [`` element](manifest-format.md#Element-manifest_server) +element. This is how the client knows what service to talk to. + +```xml + +``` + +If the URL starts with `persistent-`, then the +[`git-remote-persistent-https` helper](https://github.com/git/git/blob/HEAD/contrib/persistent-https/README) +is used to communicate with the server. + +## Credentials + +Credentials may be specified directly in typical `username:password` +[URI syntax](https://en.wikipedia.org/wiki/URI#Syntax) in the +`` element directly in the manifest. + +If they are not specified, `repo sync` has `--manifest-server-username=USERNAME` +and `--manifest-server-password=PASSWORD` options. + +If those are not used, then repo will look up the host in your +[`~/.netrc`](https://docs.python.org/3/library/netrc.html) database. + +When making the connection, cookies matching the host are automatically loaded +from the cookiejar specified in +[Git's `http.cookiefile` setting](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpcookieFile). + +## Manifest Server + +Unfortunately, there are no public reference implementations. Google has an +internal one for Android, but it is written using Google's internal systems, +so wouldn't be that helpful as a reference. + +That said, the XML-RPC API is pretty simple, so any standard XML-RPC server +example would do. Google's internal server uses Python's +[xmlrpc.server.SimpleXMLRPCDispatcher](https://docs.python.org/3/library/xmlrpc.server.html). + +## Network API + +The manifest server should implement the following RPC methods. + +### GetApprovedManifest + +> `GetApprovedManifest(branch: str, target: Optional[str]) -> str` + +The meaning of `branch` and `target` is not strictly defined. The server may +interpret them however it wants. The recommended interpretation is that the +`branch` matches the manifest branch, and `target` is an identifier for your +project that matches something users would build. + +See the client section below for how repo typically generates these values. + +The server will return a manifest or an error. If it's an error, repo will +show the output directly to the user to provide a limited feedback channel. + +If the user's request is ambiguous and could match multiple manifests, the +server has to decide whether to pick one automatically (and silently such that +the user won't know there were multiple matches), or return an error and force +the user to be more specific. + +### GetManifest + +> `GetManifest(tag: str) -> str` + +The meaning of `tag` is not strictly defined. Projects are encouraged to use +a system where the tag matches a unique source state. + +See the client section below for how repo typically generates these values. + +The server will return a manifest or an error. If it's an error, repo will +show the output directly to the user to provide a limited feedback channel. + +If the user's request is ambiguous and could match multiple manifests, the +server has to decide whether to pick one automatically (and silently such that +the user won't know there were multiple matches), or return an error and force +the user to be more specific. + +## Client Options + +Once repo has successfully downloaded the manifest from the server, it saves a +copy into `.repo/manifests/smart_sync_override.xml` so users can examine it. +The next time `repo sync` is run, this file is automatically replaced or removed +based on the current set of options. + +### --smart-sync + +Repo will call `GetApprovedManifest(branch[, target])`. + +The `branch` is determined by the current manifest branch as specified by +`--manifest-branch=BRANCH` when running `repo init`. + +The `target` is defined by environment variables in the order below. If none +of them match, then `target` is omitted. These variables were decided as they +match the settings Android build environments automatically setup. + +1. `${SYNC_TARGET}`: If defined, the value is used directly. +2. `${TARGET_PRODUCT}-${TARGET_RELEASE}-${TARGET_BUILD_VARIANT}`: If these + variables are all defined, then they are merged with `-` and used. +3. `${TARGET_PRODUCT}-${TARGET_BUILD_VARIANT}`: If these variables are all + defined, then they are merged with `-` and used. + +### --smart-tag=TAG + +Repo will call `GetManifest(TAG)`. diff --git a/subcmds/sync.py b/subcmds/sync.py index 3f1faa9f..f7ed49e4 100644 --- a/subcmds/sync.py +++ b/subcmds/sync.py @@ -1502,6 +1502,7 @@ later is required to fix a server side protocol bug. if manifest_server.startswith("persistent-"): manifest_server = manifest_server[len("persistent-") :] + # Changes in behavior should update docs/smart-sync.md accordingly. try: server = xmlrpc.client.Server(manifest_server, transport=transport) if opt.smart_sync: