Commit Graph

20 Commits

Author SHA1 Message Date
Mike Frysinger
f0925c482f platform_utils: delete unused FileDescriptorStreams APIs
Now that we've converted the few users of this over to subprocess APIs,
we don't need this anymore.  It's been a bit hairy to maintain across
different operating systems, so there's no desire to bring it back.

Using multiprocessing Pool to batch things has been working better in
general anyways.

Change-Id: I10769e96f60ecf27a80d8cc2aa0d1b199085252e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-02-24 01:45:57 +00:00
Mike Frysinger
acf63b2892 drop pyversion & is_python3 checking
We're committed to Python 3 at this point, so purge all the
is_python3 related dynamic checks.

Bug: https://crbug.com/gerrit/10418
Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:58 +00:00
Mike Frysinger
784ccfc040 strip python2-only coding:utf-8 & print_function settings
We're committed to Python 3 at this point, so clean up boilerplate.

Bug: https://crbug.com/gerrit/10418
Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-01-06 18:53:05 +00:00
Theodore Dubois
23d7dafd10 Reland "Port _FileDescriptorStreamsNonBlocking to use poll()"
Now that repo 2 requires Python 3, we can reland this.

This reverts commit 91d9587e45.

Change-Id: Id5b178ebb53bdba04bfa79cbb5c698ae5080c957
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258672
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Theodore Dubois <tbodt@google.com>
2020-03-13 17:45:36 +00:00
Mike Frysinger
8f9bf484d8 platform_utils: have Windows select stream return "" at EOF
This matches *NIX behavior where the last read is '', not None.

Bug: https://crbug.com/gerrit/12329
Change-Id: I48b026b4d1b8d7c6abbce198757b970931869e1a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255352
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-18 19:06:02 +00:00
David Pursehouse
819827a42d Fix blank line issues reported by flake8
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition

Fixed automatically with autopep8:

  git ls-files | grep py$ | xargs autopep8 --in-place \
    --select E301,E302,E303,E305,E306

Manually fix issues in project.py caused by misuse of block comments.

Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
2020-02-12 06:36:40 +00:00
Mike Frysinger
91d9587e45 Revert "Port _FileDescriptorStreamsNonBlocking to use poll()"
This reverts commit 1e01a74445.

Not all platforms support select.poll() currently it seems.
At least macOS's Python 2 doesn't (while macOS Python 3 does).

Lets back this out for the existing release series and once we
start repo-2 which is Python 3-only, we can put this back in.

Change-Id: I205206b0fa4fe2d755f4fbc6ec683ad125f27cc2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253072
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2020-02-03 23:01:07 +00:00
Rostislav Krasny
ec0ba2777f Fix method signature of platform_utils.FileDescriptorStreams._create_stream()
Change-Id: Ib80e4ec5e540d97488e7564703ddbcb74350fdfd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/251836
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Rostislav Krasny <rostigm@gmail.com>
2020-01-25 13:29:08 +00:00
Theodore Dubois
1e01a74445 Port _FileDescriptorStreamsNonBlocking to use poll()
select() has a limit of FD_SETSIZE file descriptors. If you run repo
sync -j500 you'll pretty quickly hit this limit and get "file descriptor
out of range for select" errors. poll() has no such limit.

Change-Id: I21f350e472bda1db03dcbcc437645c23dbc7a901
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248852
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Theodore Dubois <tbodt@google.com>
2019-12-18 21:16:23 +00:00
Mike Frysinger
f454512619 sync: make .git init more robust
Hitting Ctrl-C in the middle of this func will leave the .git in a
bad state that requires manual recovery.  The code tries to catch
all exceptions and recover by deleting the incomplete .git dir, but
it omits KeyboardInterrupt which Exception misses.

We could add that to the recovery path, but we can make this more
robust with a different approach: set up everything in .git.tmp/
and only move it to .git/ once we've fully initialized it.

Change-Id: I0f5b97f2e19fc39cffc3e5e23993a2da7220f4e3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244733
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
2019-11-12 03:44:33 +00:00
Mike Frysinger
f601376e13 set default file encoding to utf-8
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.

Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
2019-06-13 14:30:52 +00:00
David Pursehouse
d26146de7f platform_utils: Fix exception handling in _walk_windows_impl
Change-Id: I6b79cbc4c1bbbe17ffe8361fe1544434beaa9059
2018-11-06 09:25:35 +09:00
Renaud Paquay
bed8b62345 Add support for long paths
* Add more file i/o wrappers in platform_utils to allow using
  long paths (length > MAX_PATH) on Windows.

* Paths using the long path syntax ("\\?\" prefix) should never
  escape the platform_utils API surface area, so that this
  specific syntax is not visible to the rest of the repo code base.

* Forward many calls from os.xxx to platform_utils.xxx in various place
  to ensure long paths support, specifically when repo decides to delete
  obsolete directories.

* There are more places that need to be converted to support long paths,
  this commit is an initial effort to unblock a few common use cases.

* Also, fix remove function to handle directory symlinks

Change-Id: If82ccc408e516e96ff7260be25f8fd2fe3f9571a
2018-10-22 08:16:35 -07:00
Dylan Deng
e469a0c741 fix some sync error while using python3
Change-Id: I70925e48756c356d48359679d8ad1b9e33a68595
2018-07-24 22:20:08 +08:00
Renaud Paquay
010fed7711 Replace all os.remove calls
os.remove raises an exception when deleting read-only files on
Windows. Replace all calls with calls to platform_utils.remove,
which deals with deals with that issue.

Change-Id: I4dc9e0c9a36b4238880520c69f5075eca40f3e66
2017-08-31 13:49:36 -07:00
Renaud Paquay
227ad2ef42 Implement islink, readlink and realpath using Win32 api
Change-Id: I18452cbb32d24db73601ad10485dbe6bb278731c
2017-08-31 13:49:01 -07:00
Renaud Paquay
ad1abcb556 Port os.rename calls to work on Windows
os.rename fails on Windows if the destination exists, so replace
os.rename to platform_utils.rename which handles the platform
differences.

Change-Id: I15a86f10f65eedee5b003b80f88a0c28a3e1aa48
2017-05-29 19:33:07 +09:00
Renaud Paquay
a65adf74f9 Workaround shutil.rmtree limitation on Windows
By default, shutil.rmtree raises an exception when deleting readonly
files on Windows.

Replace all shutil.rmtree with platform_utils.rmtree, which adds an
error handler to make files read-write when they can't be deleted.

Change-Id: I9cfea9a7b3703fb16a82cf69331540c2c179ed53
2017-05-29 19:32:31 +09:00
Renaud Paquay
d5cec5e752 Add support for creating symbolic links on Windows
Replace all calls to os.symlink with platform_utils.symlink.

The Windows implementation calls into the CreateSymbolicLinkW Win32
API, as os.symlink is not supported.

Separate the Win32 API definitions into a separate module
platform_utils_win32 for clarity.

Change-Id: I0714c598664c2df93383734e609d948692c17ec5
2017-05-29 19:30:34 +09:00
Renaud Paquay
2e70291162 Make "git command" and "forall" work on Windows
Python on Windows does not support non blocking file operations.
To workaround this issue, we instead use Threads and a Queue to
simulate non-blocking calls. This is happens only when running
with the native Windows version of Python, meaning Linux and Cygwin
are not affected by this change.

Change-Id: I4ce23827b096c5138f67a85c721f58a12279bb6f
2017-05-29 19:29:30 +09:00