android/git-repo
1
Fork 0
mirror of https://gerrit.googlesource.com/git-repo synced 2024-12-21 07:16:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

sync: Fix semaphore release bug that causes thread 'leaks'

Browse Source 
When repo syncs a manifest that utilizes multiple branches
in the same project, then the sync will use an extra
thread for each "duplicate".  For example, if
the manifest includes the project "foo" and "bar"
twice, then "repo sync -jN" will fetch with N+2 threads.

This is caused by _FetchHelper() releasing the thread semaphore
object each time it's called, even though _FetchProjectList()
may call this function multiple times within the scope of a
single thread.

Fix by moving the thread semaphore release to
_FetchProjectList(), which is only called once per thread
instance.

Change-Id: I1da78b145e09524d40457db5ca5c37d315432bd8
This commit is contained in:
Andrew Wheeler 2016-06-17 16:51:07 -05:00
parent eceeb1b1f5
commit 7f1ccfbb7b
1 changed files with 12 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
subcmds/sync.py
View File

@ -255,7 +255,7 @@ later is required to fix a server side protocol bug.
dest='repo_upgraded', action='store_true', dest='repo_upgraded', action='store_true',
help=SUPPRESS_HELP) help=SUPPRESS_HELP)
def _FetchProjectList(self, opt, projects, *args, **kwargs): def _FetchProjectList(self, opt, projects, sem, *args, **kwargs):
"""Main function of the fetch threads when jobs are > 1. """Main function of the fetch threads when jobs are > 1.
Delegates most of the work to _FetchHelper. Delegates most of the work to _FetchHelper.
@ -263,15 +263,20 @@ later is required to fix a server side protocol bug.
Args: Args:
opt: Program options returned from optparse. See _Options(). opt: Program options returned from optparse. See _Options().
projects: Projects to fetch. projects: Projects to fetch.
sem: We'll release() this semaphore when we exit so that another thread
can be started up.
*args, **kwargs: Remaining arguments to pass to _FetchHelper. See the *args, **kwargs: Remaining arguments to pass to _FetchHelper. See the
_FetchHelper docstring for details. _FetchHelper docstring for details.
""" """
for project in projects: try:
success = self._FetchHelper(opt, project, *args, **kwargs) for project in projects:
if not success and not opt.force_broken: success = self._FetchHelper(opt, project, *args, **kwargs)
break if not success and not opt.force_broken:
break
finally:
sem.release()
def _FetchHelper(self, opt, project, lock, fetched, pm, sem, err_event): def _FetchHelper(self, opt, project, lock, fetched, pm, err_event):
"""Fetch git objects for a single project. """Fetch git objects for a single project.
Args: Args:
@ -283,8 +288,6 @@ later is required to fix a server side protocol bug.
(with our lock held). (with our lock held).
pm: Instance of a Project object. We will call pm.update() (with our pm: Instance of a Project object. We will call pm.update() (with our
lock held). lock held).
sem: We'll release() this semaphore when we exit so that another thread
can be started up.
err_event: We'll set this event in the case of an error (after printing err_event: We'll set this event in the case of an error (after printing
out info about the error). out info about the error).
@ -340,7 +343,6 @@ later is required to fix a server side protocol bug.
finally: finally:
if did_lock: if did_lock:
lock.release() lock.release()
sem.release()
return success return success
@ -365,10 +367,10 @@ later is required to fix a server side protocol bug.
sem.acquire() sem.acquire()
kwargs = dict(opt=opt, kwargs = dict(opt=opt,
projects=project_list, projects=project_list,
sem=sem,
lock=lock, lock=lock,
fetched=fetched, fetched=fetched,
pm=pm, pm=pm,
sem=sem,
err_event=err_event) err_event=err_event)
if self.jobs > 1: if self.jobs > 1:
t = _threading.Thread(target = self._FetchProjectList, t = _threading.Thread(target = self._FetchProjectList,