fix possible failure-to-wake deadlock with robust mutexes

when the kernel is responsible for waking waiters on a robust mutex
whose owner died, it does not have a waiters count available and must
rely entirely on the waiter bit of the lock value.

normally, this bit is only set by newly arriving waiters, so it will
be clear if no new waiters arrived after the current owner obtained
the lock, even if there are other waiters present. leaving it clear is
desirable because it allows timed-lock operations to remove themselves
as waiters and avoid causing unnecessary futex wake syscalls. however,
for process-shared robust mutexes, we need to set the bit whenever
there are existing waiters so that the kernel will know to wake them.

for non-process-shared robust mutexes, the wake happens in userspace
and can look at the waiters count, so the bit does not need to be set
in the non-process-shared case.

1 files changed, 4 insertions, 1 deletions

diff --git a/src/thread/pthread_mutex_trylock.c b/src/thread/pthread_mutex_trylock.c
index 31587e1f..e8515179 100644
--- a/src/thread/pthread_mutex_trylock.c
+++ b/src/thread/pthread_mutex_trylock.c
@@ -22,7 +22,10 @@ int __pthread_mutex_trylock_owner(pthread_mutex_t *m)
 	}
 	if (own == 0x40000000) return ENOTRECOVERABLE;
 
-	self->robust_list.pending = &m->_m_next;
+	if (m->_m_type & 128) {
+		if (m->_m_waiters) tid |= 0x80000000;
+		self->robust_list.pending = &m->_m_next;
+	}
 
 	if ((own && (!(own & 0x40000000) || !(type & 4)))
 	    || a_cas(&m->_m_lock, old, tid) != old) {