Symptom: repo sync exception
Root Cause: os.rename only works when source and destination are on the same file system
Solution: using shutil.move
to save disk usage, I create links for projects and project-objects, link to folder on another disk
lrwxrwxrwx 1 owenwen owenwen 47 Jun 9 16:40 project-objects -> /disk3/AndroidLocalRepos/.repo/project-objects/
lrwxrwxrwx 1 owenwen owenwen 40 Jun 9 16:40 projects -> /disk3/AndroidLocalRepos/.repo/projects/
below are exception I met:
"""
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/disk2/Android11/.repo/repo/subcmds/sync.py", line 550, in _CheckoutOne
project.Sync_LocalHalf(syncbuf, force_sync=force_sync)
File "/disk2/Android11/.repo/repo/project.py", line 1251, in Sync_LocalHalf
self._InitWorkTree(force_sync=force_sync, submodules=submodules)
File "/disk2/Android11/.repo/repo/project.py", line 2801, in _InitWorkTree
self._CheckDirReference(self.gitdir, dotgit, share_refs=True)
File "/disk2/Android11/.repo/repo/project.py", line 2674, in _CheckDirReference
platform_utils.rename(dst_path, src_path)
File "/disk2/Android11/.repo/repo/platform_utils.py", line 127, in rename
os.rename(src, dst)
OSError: [Errno 18] Invalid cross-device link: '/disk2/Android11/system/libhidl/.git/packed-refs' -> '/disk2/Android11/.repo/projects/system/libhidl.git/packed-refs'
"""
Change-Id: Ifda2f16530cc5a8f280169f482ee858f9e5241d3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/316002
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Now that we've converted the few users of this over to subprocess APIs,
we don't need this anymore. It's been a bit hairy to maintain across
different operating systems, so there's no desire to bring it back.
Using multiprocessing Pool to batch things has been working better in
general anyways.
Change-Id: I10769e96f60ecf27a80d8cc2aa0d1b199085252e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
We're committed to Python 3 at this point, so purge all the
is_python3 related dynamic checks.
Bug: https://crbug.com/gerrit/10418
Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383
Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Now that repo 2 requires Python 3, we can reland this.
This reverts commit 91d9587e45.
Change-Id: Id5b178ebb53bdba04bfa79cbb5c698ae5080c957
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258672
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Theodore Dubois <tbodt@google.com>
- E301 expected 1 blank line
- E302 expected 2 blank lines
- E303 too many blank lines
- E305 expected 2 blank lines after class or function definition
- E306 expected 1 blank line before a nested definition
Fixed automatically with autopep8:
git ls-files | grep py$ | xargs autopep8 --in-place \
--select E301,E302,E303,E305,E306
Manually fix issues in project.py caused by misuse of block comments.
Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: David Pursehouse <dpursehouse@collab.net>
This reverts commit 1e01a74445.
Not all platforms support select.poll() currently it seems.
At least macOS's Python 2 doesn't (while macOS Python 3 does).
Lets back this out for the existing release series and once we
start repo-2 which is Python 3-only, we can put this back in.
Change-Id: I205206b0fa4fe2d755f4fbc6ec683ad125f27cc2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253072
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
select() has a limit of FD_SETSIZE file descriptors. If you run repo
sync -j500 you'll pretty quickly hit this limit and get "file descriptor
out of range for select" errors. poll() has no such limit.
Change-Id: I21f350e472bda1db03dcbcc437645c23dbc7a901
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248852
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Theodore Dubois <tbodt@google.com>
Hitting Ctrl-C in the middle of this func will leave the .git in a
bad state that requires manual recovery. The code tries to catch
all exceptions and recover by deleting the incomplete .git dir, but
it omits KeyboardInterrupt which Exception misses.
We could add that to the recovery path, but we can make this more
robust with a different approach: set up everything in .git.tmp/
and only move it to .git/ once we've fully initialized it.
Change-Id: I0f5b97f2e19fc39cffc3e5e23993a2da7220f4e3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244733
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
There's no reason to support any other encoding in these files.
This only affects the files themselves and not streams they open.
Bug: https://crbug.com/gerrit/10418
Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
* Add more file i/o wrappers in platform_utils to allow using
long paths (length > MAX_PATH) on Windows.
* Paths using the long path syntax ("\\?\" prefix) should never
escape the platform_utils API surface area, so that this
specific syntax is not visible to the rest of the repo code base.
* Forward many calls from os.xxx to platform_utils.xxx in various place
to ensure long paths support, specifically when repo decides to delete
obsolete directories.
* There are more places that need to be converted to support long paths,
this commit is an initial effort to unblock a few common use cases.
* Also, fix remove function to handle directory symlinks
Change-Id: If82ccc408e516e96ff7260be25f8fd2fe3f9571a
os.remove raises an exception when deleting read-only files on
Windows. Replace all calls with calls to platform_utils.remove,
which deals with deals with that issue.
Change-Id: I4dc9e0c9a36b4238880520c69f5075eca40f3e66
os.rename fails on Windows if the destination exists, so replace
os.rename to platform_utils.rename which handles the platform
differences.
Change-Id: I15a86f10f65eedee5b003b80f88a0c28a3e1aa48
By default, shutil.rmtree raises an exception when deleting readonly
files on Windows.
Replace all shutil.rmtree with platform_utils.rmtree, which adds an
error handler to make files read-write when they can't be deleted.
Change-Id: I9cfea9a7b3703fb16a82cf69331540c2c179ed53
Replace all calls to os.symlink with platform_utils.symlink.
The Windows implementation calls into the CreateSymbolicLinkW Win32
API, as os.symlink is not supported.
Separate the Win32 API definitions into a separate module
platform_utils_win32 for clarity.
Change-Id: I0714c598664c2df93383734e609d948692c17ec5
Python on Windows does not support non blocking file operations.
To workaround this issue, we instead use Threads and a Queue to
simulate non-blocking calls. This is happens only when running
with the native Windows version of Python, meaning Linux and Cygwin
are not affected by this change.
Change-Id: I4ce23827b096c5138f67a85c721f58a12279bb6f
