Use JSON as it is shown to be much faster than pickle.
Also clean up the loading and saving functions.
Change-Id: I45b3dee7b4d59a1c0e0d38d4a83b543ac5839390
Gerrit no longer requires 'p/', and this causes unexpected behavior.
In this change we stop appending 'p/' to the urls.
Change-Id: I72c13bf838f4112086141959fb1af249f9213ce6
* Add .decode('utf-8') where needed
* Add 'b' to `open` where needed, and remove where unnecessary
Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
When uploading a change for review, we sometimes request /ssh_info to
get the correct port number for uploading via ssh (regardless of
whether or not we intend to upload over ssh).
If we have trouble accessing /ssh_info (for authentication reasons,
etc), we now press on under the assumption that we will upload via http
instead of aborting.
Change-Id: Ica6bbeac1279e89424a903529649b7f4af0b6937
The call to `urlopen` can raise `HTTPException`, but this is not
caught which results in a python Traceback.
Add handling of the exception. Because `HTTPException` and its
derived classes do not have any message, print the name of the
exception in the error message instead.
Change-Id: Ic90fb4cc0e92702375cd976d4a03876c8ce8bffc
Add a new module with methods for checking the Python version.
Instead of handling Python3 imports with try...except blocks, first
check the python version and then import the relevant modules. This
makes the code a bit cleaner and will result in less diff when/if we
remove support for Python < 3 later.
Use the same mechanism to handle `input` vs. `raw_input` and add
suppression of pylint warnings caused by redefinition of the built-in
method `input`.
Change-Id: Ia403e525b88d77640a741ac50382146e7d635924
Also-by: Chirayu Desai <cdesai@cyanogenmod.org>
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
* Fix imports.
* Use python3 syntax.
* Wrap map() calls with list().
* Use list() only wherever needed.
(Thanks Conley!)
* Fix dictionary iteration methods
(s/iteritems/items/).
* Make use of sorted() in appropriate places
* Use iterators directly in the loop.
* Don't use .keys() wherever it isn't needed.
* Use sys.maxsize instead of sys.maxint
TODO:
* Make repo work fully with python3. :)
Some of this was done by the '2to3' tool [1], by
applying the needed fixes in a way that doesn't
break compatibility with python2.
Links:
[1]: http://docs.python.org/2/library/2to3.html
Change-Id: Ibdf3bf9a530d716db905733cb9bfef83a48820f7
Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
Enable the following Pylint warnings:
C0322: Operator not preceded by a space
C0323: Operator not followed by a space
C0324: Comma not followed by a space
And make the necessary fixes.
Change-Id: I74d74283ad5138cbaf28d492b18614eb355ff9fe
The repo coding style is to indent at 2 characters, but there are
many places where this is not followed.
Enable pylint warning "W0311: Bad indentation" and make sure all
indentation is at multiples of 2 characters.
Change-Id: I68f0f64470789ce2429ab11104d15d380a63e6a8
In Python3, range() creates a generator rather than a list.
None of the parameters in the ranges changed looked large enough
to create an impact in memory in Python2. Note: the only use of
range() was for iteration and did not need to be changed.
This is part of a series of changes to introduce Python3 support.
Change-Id: I50b665f9296ea160a5076c71f36a65f76e47029f
Fixing some more pylint warnings:
W1401: Anomalous backslash in string
W0623: Redefining name 'name' from outer scope
W0702: No exception type(s) specified
E0102: name: function already defined line n
Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
"except Exception as e" instead of "except Exception, e"
This is part of a transition to supporting Python 3. Python >= 2.6
support "as" syntax.
Note: this removes Python 2.5 support.
Change-Id: I309599f3981bba2b46111c43102bee38ff132803
Fix the following issues reported by pylint:
C0321: More than one statement on a single line
W0622: Redefining built-in 'name'
W0612: Unused variable 'name'
W0613: Unused argument 'name'
W0102: Dangerous default value 'value' as argument
W0105: String statement has no effect
Also fixed a few cases of inconsistent indentation.
Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
Clients might be using their own special git-remote-* helper that
has a hypen in its name. Permit - in the scheme part of the URL
when trying to decide if it is an SSH URL and assume it is *not*
SSH if the URL matches "foo-bar://" style.
Change-Id: I7ba2d810a614f6e605a441d5972902c4a14e73fd
If SSH is not available, Gerrit returns NOT_AVAILABLE to the /ssh_info
query made by repo upload. In this case fallback to the /p/$PROJECT URL
that Gerrit also exports and use that for uploads.
Change-Id: I1e3e39ab709ecc0a692614a41a42446426f39c08
REPO_HOST_PORT_INFO can be set to 'host:port' and be used
instead of the review URL given in the manifest.
Change-Id: I440bdecb2c2249fe5285ec5d0c28a937b4053450
Signed-off-by: Shawn O. Pearce <sop@google.com>
Not every version of urllib2 supplies a reason object on the
HTTPError exception that it throws from urlopen(). Work around
this by using str(e) instead and hope the string formatting includes
sufficient information.
Change-Id: I0f4586dba0aa7152691b2371627c951f91fdfc8d
Signed-off-by: Shawn O. Pearce <sop@google.com>
An HTTP (or HTTPS) based remote server may now offer a 'clone.bundle'
file in each repository's Git directory. Over an http:// or https://
remote repo will first ask for '$URL/clone.bundle', and if present
download this to bootstrap the local client, rather than relying
on the native Git transport to initialize the new repository.
Bundles may be hosted elsewhere. The client automatically follows a
HTTP 302 redirect to acquire the bundle file. This allows servers
to direct clients to cached copies residing on content delivery
networks, where the bundle may be closer to the end-user.
Bundle downloads are resumeable from where they last left off,
allowing clients to initialize large repositories even when the
connection gets interrupted.
If a bundle does not exist for a repository (a HTTP 404 response
code is returned for '$URL/clone.bundle'), the native Git transport
is used instead. If the client is performing a shallow sync, the
bundle transport is not used, as there is no way to embed shallow
data into the bundle.
Change-Id: I05dad17792fd6fd20635a0f71589566e557cc743
Signed-off-by: Shawn O. Pearce <sop@google.com>
Teach repo how to resolve URLs using the url.insteadof feature
that C Git natively uses during clone, fetch or push. This will
later allow repo to resolve a URL before accessing it directly.
We do not want to pre-resolve things and store the resolved URL
into individual projects, as this makes it impossible for the
user to undo the insteadof mapping at a later date.
Change-Id: I0f62e811197c53fbc8a8be424e3cabf4ed07b4cb
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the http_proxy environment variable was set, honor it during
the entire repo session for any Python created HTTP connections.
Change-Id: Ib4ae833cb2cdd47ab0126949f6b399d2c142887d
Signed-off-by: Shawn O. Pearce <sop@google.com>
'repo upload' makes http request using urllib2 python library.
Unfortunately this library does not work (by default) in case
if the user behind a proxy.
This change adds proxy handler in case if 'http_proxy' environment
variable is set.
Change-Id: Ic4176ad733fc21bd5b59661b3eacc2f0a7c3c1ff
Instead of giving a Python backtrace when there is a connectivity
problem during repo upload, report that we cannot access the host,
and why, with a halfway decent error message.
Bug: REPO-45
Change-Id: I9a45b387e86e48073a2d99bd6d594c1a7d6d99d4
Signed-off-by: Shawn O. Pearce <sop@google.com>
(cherry picked from commit d2dfac81ad)
This fixes the SSH Control Masters to be managed in a thread-safe
fashion. This is important because "repo sync -jN" uses threads to
sync more than one repository at the same time. The problem didn't
show up earlier because it was masked if all of the threads tried to
connect to the same host that was used on the "repo init" line.
os.remove() raises OSError if the file being removed doesn't exist.
Check before calling to ensure we don't raise a useless exception
on an already deleted file.
Change-Id: I44c1c7dd97a47fcab8afb6c18fdf179158b6dab7
Signed-off-by: Shawn O. Pearce <sop@google.com>
Be more thorough about checking for an existing ssh master by
running a test command first, and only opening up a new master
if the test fails to connect.
Change-Id: I56fe8e7b4dbc123675b7f259e81d359ed0cd55cf
Signed-off-by: Shawn O. Pearce <sop@google.com>
Some users might need to use a different login name than the local
part of their email address for their Gerrit Code Review user
account. Allow it to be overridden with the review.HOST.username
configuration variable.
Change-Id: I714469142ac7feadf09fee9c26680c0e09076b75
Signed-off-by: Shawn O. Pearce <sop@google.com>
This change allows local SSH configuration to choose the port number
to use when not explicitly set in the manifest.
(cherry picked from commit 4c0f670465)
Change-Id: Ibea99cfe46b6a2cc27f754cc3944a2fe10f6fda4
If the SSH control master process is killed while an active git
fetch is using its network socket, the underlying SSH client may
not realize the connection was broken. This can lead to both the
client and the server waiting indefinitely for network messages
which will never be sent.
Work around the problem by keeping track of any processes that use
the tunnels we establish. If we are about to kill any of the SSH
control masters that we started, ensure the clients using them are
successfully killed first.
Change-Id: Ida6c124dcb0c6a26bf7dd69cba2fbdc2ecd5b2fc
Signed-off-by: Shawn O. Pearce <sop@google.com>
Repo can now properly handle url.insteadOf sections in the
user's ~/.gitconfig file. This means that a user can now enjoy
the master-ssh functionality even if he/she uses insteadOf's in
~/.gitconfig to rewrite git:// URLs to ssh:// style URLs.
Change-Id: Ic0f04a9c57206a7b89eb0f10bf188c4c483debe3
Signed-off-by: Shawn O. Pearce <sop@google.com>
If a file (e.g. ~/.gitconfig) does not exist, we get None
here rather than a string. NoneType lacks rstrip() so we
cannot strip it.
Signed-off-by: Shawn O. Pearce <sop@google.com>
A git-config entry with no value was preventing repo
from initializing. This modifies _ReadGit() to handle
config entries with empty values.
Signed-off-by: David Aguilar <davvid@gmail.com>
Reported-by: Josh Guilfoyle <jasta00@gmail.com>
If the SSH client terminated abnormally in the background (e.g. the
server shutdown while we were doing a sync) then the pid won't exist.
Instead of crashing, ignore it, the result we wanted (a non-orphaned
ssh process) is already acheived.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the pickle config file is 0 bytes in length, we may have
crashed (or been aborted) while writing the file out to disk.
Instead of crashing with a backtrace, just treat the file as
though it wasn't present and load off a `git config` fork.
Signed-off-by: Shawn O. Pearce <sop@google.com>
Noticed by users on repo-discuss, we were missing a return False
here to signal that SSH control master was not used to setup the
network connection.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This way we can put it in another directory than the config file
itself, e.g. hide it inside ".git" when parsing a ".gitmodules"
file from the working tree.
Signed-off-by: Shawn O. Pearce <sop@google.com>
This can be useful when pulling apart a configuration file, like
finding all entries which match submodule.*.*.
Signed-off-by: Shawn O. Pearce <sop@google.com>
I only tested this with ssh://hostname/ style URLs, so I failed
to test ssh://user@hostname/ format, which failed if the hostname
portion was longer than 1 character.
Signed-off-by: Shawn O. Pearce <sop@google.com>
If the SSH URL doesn't contain a port number, but uses the ssh://
or git+ssh:// syntax we raised a Python runtime error due to the
'port' local variable not being assigned a value. Default it to
the IANA assigned port for SSH, 22.
Signed-off-by: Shawn O. Pearce <sop@google.com>
By creating a background ssh "control master" process which lives
for the duration of our sync cycle we can easily cut the time for
a no-op sync of 132 projects from 60s to 18s.
Bug: REPO-11
Signed-off-by: Shawn O. Pearce <sop@google.com>
Its quite common for most projects to be matching the current
manifest revision, as most developers only modify one or two projects
at any one time. We can speed up `repo start foo` (that impacts
the entire client) by performing most of the branch creation and
switch operations in pure Python, and thus avoid 4 forks per project.
Signed-off-by: Shawn O. Pearce <sop@google.com>
The value of the varible TRACE was copied during the import, which
happens before the --trace option can be processed. So instead we
now use a function to determine if the value is set, as the function
can be safely copied early during import.
Signed-off-by: Shawn O. Pearce <sop@google.com>
