Convert the RepoUserAgent function into a UserAgent class. This
makes it cleaner to hold internal state, and will make it easier
to add a separate git User-Agent, although we don't do it here.
We make the RepoSourceVersion independent of GitCommand so that
it can be called by the class (later).
Bug: https://crbug.com/gerrit/11144
Change-Id: Iab4e1f974b8733a36b243b2d03f5085a96effa19
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239232
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
We can't import the main module, so move the UserAgent helper out of
it and into the git_command module so it can be used in more places.
Bug: https://crbug.com/gerrit/11144
Change-Id: I8093c8a20bd1dc7d612d0e2a85180341817c0d86
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231057
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
There is a standard Python "trace" module, so having a local trace.py
prevents us being able to import that. Rename the module to avoid.
Change-Id: I23e29ec95a2204bb168a641323d05e76968d9b57
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234832
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
A new option, --partial-clone is added to 'repo init' which tells repo
to utilize git's partial clone functionality, which reduces disk and
bandwidth usage when downloading by omitting blob downloads initially.
Different from restricting clone-depth, the user will have full access
to change history, etc., as the objects are downloaded on demand.
Change-Id: I60326744875eac16521a007bd7d5481112a98749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
Since ParseGitVersion can call `git --version` automatically, we don't
need this duplicate version() helper anymore. The only other user is
the `repo version` code, so convert that to version_tuple().full.
Bug: https://crbug.com/gerrit/11144
Change-Id: I9d77822fc39f4ba28884d9183359169cabf5f17d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231055
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.
Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
Make it explicit that the ssh wrapper we use for control master
support accepts OpenSSH-compatible command line arguments instead of
asking Git to guess.
The GIT_SSH_VARIANT setting was introduced in Git v2.13.0-rc0~3^2~2
(2017-02-01) as a more reliable detection method than relying on the
ssh command name. Fortunately the default variant was 'ssh' (i.e.,
OpenSSH-compatible) so this wasn't initially required.
Now Git wants to start using more OpenSSH features
(-o SendEnv=GIT_PROTOCOL), and in order to do so its ssh variant
detection will need to be tweaked. Set GIT_SSH_VARIANT explicitly
so this helper can continue to work regardless of how Git modifies
its autodetection.
Reported-by: William Yan <wyan@google.com>
Change-Id: I6bf2c53b4eb5303a429eae6cb68e0a5ccce89064
Python on Windows does not support non blocking file operations.
To workaround this issue, we instead use Threads and a Queue to
simulate non-blocking calls. This is happens only when running
with the native Windows version of Python, meaning Linux and Cygwin
are not affected by this change.
Change-Id: I4ce23827b096c5138f67a85c721f58a12279bb6f
See git commit 33cfccbbf35a -- some protocols allow arbitrary command
execution as part of the URL. Instead of blindly allowing those,
whitelist the allowed URL protocols unless the user has already done so.
Bug: Issue 210
Change-Id: I6bd8e721aa5e3dab53ef28cfdc8fde33eb74ef76
output from a process is in bytes in python3. we need
to decode it.
in Python3, bytes don't have an encode attribute. use this
to identify it.
Change-Id: I152f2ec34614131027db680ead98b53f9b321ed5
Switch the GitCommand program to always capture the output for stdout
and stderr. And by default print the output while running.
The options capture_stdout and capture_stderr have effectively become
options to supress the printing of stdout and stderr.
Update the 'git fetch' to use '--progress' so that the progress messages
will be displayed. git checks if the output location isatty() and if it
is not a TTY it will by default not print the progress messages.
Change-Id: Ifdae138e008f80a59195f9f43c911a1a5210ec60
'repo' and 'git_command.py' had their own git version parsing code.
This change shares that code between the modules. DRY is good.
Change-Id: Ic896d2dc08353644bd4ced57e15a91284d97d54a
The last change regarding version parsing lost handling of version
hyphenation, this restores that. In otherwords,
1.1.1-otherstuff is parsed as (1,1,1) instead of (1,1,0)
Change-Id: I3753944e92095606653835ed2bd090b9301c7194
* Add .decode('utf-8') where needed
* Add 'b' to `open` where needed, and remove where unnecessary
Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
Fixing some more pylint warnings:
W1401: Anomalous backslash in string
W0623: Redefining name 'name' from outer scope
W0702: No exception type(s) specified
E0102: name: function already defined line n
Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
"except Exception as e" instead of "except Exception, e"
This is part of a transition to supporting Python 3. Python >= 2.6
support "as" syntax.
Note: this removes Python 2.5 support.
Change-Id: I309599f3981bba2b46111c43102bee38ff132803
Git requires the values in this environment variable to be
single quoted. repo must wrap the expression into '' before
adding it to the environment.
Change-Id: I20a1fb8772f9aa6e9fd5a0516c981c2ca020ef05
Ancient versions of Git don't understand the -c command line flag
that we tried to use to pass http_proxy down into Git on Darwin.
Use the environment variable instead, to more gracefully degrade
with these old versions.
Change-Id: Iffffa32088c1fd803895b990b3377ecfec6a1b14
The system libcurl library seems to ignore http_proxy on Mac OS
X systems. Copy the http_proxy environment variable (if set) as
`git -c http.proxy` whenever running a Git command.
Change-Id: I0ab29336897178f70b85092601f9fcc306dd17e1
Make repo use the standard way in python to work with pipes.
Communication via pipes to sub processes is done by calling
communicate(). This will make repo not hang every now and
then.
Change-Id: Ibe2c4ecbdbcbe72f0b725ca50d54088e5646fc5d
Setting REPO_CURL_VERBOSE=1 in the environment will register a debug
level HTTPHandler on the urllib2 library, showing HTTP requests and
responses on the stderr channel of repo.
During any HTTP or HTTPS request created inside of the repo process,
a custom User-Agent header is now defined:
User-Agent: git-repo/1.7.5 (Linux) git/1.7.7 Python/2.6.5
Change-Id: Ia5026fb1e1500659bd2af27416d85e205048bf26
Signed-off-by: Shawn O. Pearce <sop@google.com>
Windows allows the environment to have unicode values.
This will cause Python to fail to execute the command.
Change-Id: I37d922c3d7ced0d5b4883f0220346ac42defc5e9
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the SSH control master process is killed while an active git
fetch is using its network socket, the underlying SSH client may
not realize the connection was broken. This can lead to both the
client and the server waiting indefinitely for network messages
which will never be sent.
Work around the problem by keeping track of any processes that use
the tunnels we establish. If we are about to kill any of the SSH
control masters that we started, ensure the clients using them are
successfully killed first.
Change-Id: Ida6c124dcb0c6a26bf7dd69cba2fbdc2ecd5b2fc
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way we can use it to detect feature support in the underlying
git, such as new options or commands that have been added in more
recent versions.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Mac OS X sets TMPDIR to a very long path within /var, so long
that a socket created in that location is too big for a struct
sockaddr_un on the platform, resulting in OpenSSH being unable
to create or bind to a socket in that location.
Instead we try to use the very short and very common /tmp, but
fall back to the guessed default if /tmp does not exist.
Signed-off-by: Shawn O. Pearce <sop@google.com>
By creating a background ssh "control master" process which lives
for the duration of our sync cycle we can easily cut the time for
a no-op sync of 132 projects from 60s to 18s.
Bug: REPO-11
Signed-off-by: Shawn O. Pearce <sop@google.com>
The value of the varible TRACE was copied during the import, which
happens before the --trace option can be processed. So instead we
now use a function to determine if the value is set, as the function
can be safely copied early during import.
Signed-off-by: Shawn O. Pearce <sop@google.com>
