We were perhaps a bit too hasty to jump to git-2.10. Existing LTS
releases of Ubuntu are quite old still: Trusty has 1.9 while Xenial
has 2.5. While we plan on dropping support for those eventually as
we migrate to Python 3.6, we don't need to be so strict just yet on
the git versions.
We also want to disconnect the version the repo launcher requires
from the version the rest of the source tree requires. The repo
launcher doesn't need as many features, and being flexible there
allows us more freedom to upgrade & rollback as needed.
So we'll allow git-1.7 again, but start warning on any users older
than git-1.9. This aligns better with existing LTS releases, and
gives users a chance to start upgrading before we cut them off.
Change-Id: I140305dd8e42c9719c84e2aee0dc6a5c5b18da25
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254573
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The git-2.10 series was released in 2016. Since we're moving to
require Python 3.6 which was also released in 2016, bumping up the
git version seems reasonable. Also we don't really test any git
versions close to as old as 1.7.2 which was released in 2010.
Change-Id: Ib71b714de6cd0b7dd50d0b300b108a560ee27331
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253134
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
We've been setting the User-Agent header when making connections
from repo itself, but not when running git (as the latter will set
up User-Agent itself). Our Gerrit/Git admins say it'll be helpful
if we pass through the repo version settings even when running git.
We currently set GIT_HTTP_USER_AGENT and not GIT_USER_AGENT as it's
unclear if the extended form works over all protocols. We can wait
for a user request.
Bug: https://crbug.com/gerrit/11144
Change-Id: I21d293f49534058dbc23225152451df26c5b7bef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239233
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Convert the RepoUserAgent function into a UserAgent class. This
makes it cleaner to hold internal state, and will make it easier
to add a separate git User-Agent, although we don't do it here.
We make the RepoSourceVersion independent of GitCommand so that
it can be called by the class (later).
Bug: https://crbug.com/gerrit/11144
Change-Id: Iab4e1f974b8733a36b243b2d03f5085a96effa19
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239232
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We can't import the main module, so move the UserAgent helper out of
it and into the git_command module so it can be used in more places.
Bug: https://crbug.com/gerrit/11144
Change-Id: I8093c8a20bd1dc7d612d0e2a85180341817c0d86
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231057
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
There is a standard Python "trace" module, so having a local trace.py
prevents us being able to import that. Rename the module to avoid.
Change-Id: I23e29ec95a2204bb168a641323d05e76968d9b57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234832
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
A new option, --partial-clone is added to 'repo init' which tells repo
to utilize git's partial clone functionality, which reduces disk and
bandwidth usage when downloading by omitting blob downloads initially.
Different from restricting clone-depth, the user will have full access
to change history, etc., as the objects are downloaded on demand.
Change-Id: I60326744875eac16521a007bd7d5481112a98749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
Since ParseGitVersion can call `git --version` automatically, we don't
need this duplicate version() helper anymore. The only other user is
the `repo version` code, so convert that to version_tuple().full.
Bug: https://crbug.com/gerrit/11144
Change-Id: I9d77822fc39f4ba28884d9183359169cabf5f17d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231055
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.
Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
Make it explicit that the ssh wrapper we use for control master
support accepts OpenSSH-compatible command line arguments instead of
asking Git to guess.
The GIT_SSH_VARIANT setting was introduced in Git v2.13.0-rc0~3^2~2
(2017-02-01) as a more reliable detection method than relying on the
ssh command name. Fortunately the default variant was 'ssh' (i.e.,
OpenSSH-compatible) so this wasn't initially required.
Now Git wants to start using more OpenSSH features
(-o SendEnv=GIT_PROTOCOL), and in order to do so its ssh variant
detection will need to be tweaked. Set GIT_SSH_VARIANT explicitly
so this helper can continue to work regardless of how Git modifies
its autodetection.
Reported-by: William Yan <wyan@google.com>
Change-Id: I6bf2c53b4eb5303a429eae6cb68e0a5ccce89064
Python on Windows does not support non blocking file operations.
To workaround this issue, we instead use Threads and a Queue to
simulate non-blocking calls. This is happens only when running
with the native Windows version of Python, meaning Linux and Cygwin
are not affected by this change.
Change-Id: I4ce23827b096c5138f67a85c721f58a12279bb6f
See git commit 33cfccbbf35a -- some protocols allow arbitrary command
execution as part of the URL. Instead of blindly allowing those,
whitelist the allowed URL protocols unless the user has already done so.
Bug: Issue 210
Change-Id: I6bd8e721aa5e3dab53ef28cfdc8fde33eb74ef76
output from a process is in bytes in python3. we need
to decode it.
in Python3, bytes don't have an encode attribute. use this
to identify it.
Change-Id: I152f2ec34614131027db680ead98b53f9b321ed5
Switch the GitCommand program to always capture the output for stdout
and stderr. And by default print the output while running.
The options capture_stdout and capture_stderr have effectively become
options to supress the printing of stdout and stderr.
Update the 'git fetch' to use '--progress' so that the progress messages
will be displayed. git checks if the output location isatty() and if it
is not a TTY it will by default not print the progress messages.
Change-Id: Ifdae138e008f80a59195f9f43c911a1a5210ec60
'repo' and 'git_command.py' had their own git version parsing code.
This change shares that code between the modules. DRY is good.
Change-Id: Ic896d2dc08353644bd4ced57e15a91284d97d54a
The last change regarding version parsing lost handling of version
hyphenation, this restores that. In otherwords,
1.1.1-otherstuff is parsed as (1,1,1) instead of (1,1,0)
Change-Id: I3753944e92095606653835ed2bd090b9301c7194
* Add .decode('utf-8') where needed
* Add 'b' to `open` where needed, and remove where unnecessary
Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
Fixing some more pylint warnings:
W1401: Anomalous backslash in string
W0623: Redefining name 'name' from outer scope
W0702: No exception type(s) specified
E0102: name: function already defined line n
Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
"except Exception as e" instead of "except Exception, e"
This is part of a transition to supporting Python 3. Python >= 2.6
support "as" syntax.
Note: this removes Python 2.5 support.
Change-Id: I309599f3981bba2b46111c43102bee38ff132803
Git requires the values in this environment variable to be
single quoted. repo must wrap the expression into '' before
adding it to the environment.
Change-Id: I20a1fb8772f9aa6e9fd5a0516c981c2ca020ef05
Ancient versions of Git don't understand the -c command line flag
that we tried to use to pass http_proxy down into Git on Darwin.
Use the environment variable instead, to more gracefully degrade
with these old versions.
Change-Id: Iffffa32088c1fd803895b990b3377ecfec6a1b14
The system libcurl library seems to ignore http_proxy on Mac OS
X systems. Copy the http_proxy environment variable (if set) as
`git -c http.proxy` whenever running a Git command.
Change-Id: I0ab29336897178f70b85092601f9fcc306dd17e1
Make repo use the standard way in python to work with pipes.
Communication via pipes to sub processes is done by calling
communicate(). This will make repo not hang every now and
then.
Change-Id: Ibe2c4ecbdbcbe72f0b725ca50d54088e5646fc5d
Setting REPO_CURL_VERBOSE=1 in the environment will register a debug
level HTTPHandler on the urllib2 library, showing HTTP requests and
responses on the stderr channel of repo.
During any HTTP or HTTPS request created inside of the repo process,
a custom User-Agent header is now defined:
User-Agent: git-repo/1.7.5 (Linux) git/1.7.7 Python/2.6.5
Change-Id: Ia5026fb1e1500659bd2af27416d85e205048bf26
Signed-off-by: Shawn O. Pearce <sop@google.com>
Windows allows the environment to have unicode values.
This will cause Python to fail to execute the command.
Change-Id: I37d922c3d7ced0d5b4883f0220346ac42defc5e9
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the SSH control master process is killed while an active git
fetch is using its network socket, the underlying SSH client may
not realize the connection was broken. This can lead to both the
client and the server waiting indefinitely for network messages
which will never be sent.
Work around the problem by keeping track of any processes that use
the tunnels we establish. If we are about to kill any of the SSH
control masters that we started, ensure the clients using them are
successfully killed first.
Change-Id: Ida6c124dcb0c6a26bf7dd69cba2fbdc2ecd5b2fc
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way we can use it to detect feature support in the underlying
git, such as new options or commands that have been added in more
recent versions.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Mac OS X sets TMPDIR to a very long path within /var, so long
that a socket created in that location is too big for a struct
sockaddr_un on the platform, resulting in OpenSSH being unable
to create or bind to a socket in that location.
Instead we try to use the very short and very common /tmp, but
fall back to the guessed default if /tmp does not exist.
Signed-off-by: Shawn O. Pearce <sop@google.com>
By creating a background ssh "control master" process which lives
for the duration of our sync cycle we can easily cut the time for
a no-op sync of 132 projects from 60s to 18s.
Bug: REPO-11
Signed-off-by: Shawn O. Pearce <sop@google.com>
The value of the varible TRACE was copied during the import, which
happens before the --trace option can be processed. So instead we
now use a function to determine if the value is set, as the function
can be safely copied early during import.
Signed-off-by: Shawn O. Pearce <sop@google.com>
