pty/tty release issue

[Andrei Paskevich]

Hello. I encountered the following problem when using
rxvt-unicode 8.1 on Linux with setuid/setgid daemon.

1) Summary: when runned with the helper, the daemon process
does not close his pty file descriptor and the terminal pair
is never released.

2) Patch which cures the trouble:

===================== proof-of-concept patch =====================
--- rxvt-unicode-8.1/src/proxy.C	2006-12-07 04:18:34.000000000 +0100
+++ rxvt-unicode-8.1-ap/src/proxy.C	2007-01-16 11:14:49.000000000 +0100
@@ -127,6 +127,13 @@
     {
       NEED_TOKEN;

+      close_tty(); // redundant (init.C:1378)
+
+      if (this->pty >= 0)
+        close (this->pty);
+
+      this->pty = -1; // redundant, too
+
       command cmd;

       cmd.type = command::destroy;
========================= end of the diff ========================

3) Long and boresome explanation:

When I start a client, urxvtc, it gets a slave tty,
a device file in /dev/pts (I use linux devpts fs).
When I close the terminal window, this device file
stays in /dev/pts (though changes permissions to 0666),
and is never reused again by another terminal window,
until I kill the daemon. According to /proc/*/fd/,
this pts file does not belong anymore to any other
process. On the other hand, if I run the daemon
without setuid/setgid bit, that is without helper,
everything is fine: the pts device disappears from
/dev/pts immediately as I close the terminal window
and can be assigned again to new terminals.

I digged the source code a little, and found what
could be the bug, I guess. In the problem case:

1. We have two processes: the helper and the daemon.
2. Daemon prepares to start a client and requests
   a terminal pair from the helper (init.C:1315).
3. Helper obtains a terminal pair and sends the file
   descriptors to the daemon. Now, the both processes
   have both pty and tty open.
4. Daemon forks the client. Now, three processes have
   both pty and tty open. Then the daemon closes tty
   (init.C:1378) but not pty. Good.
5. When the client application quits, its file
   descriptors are closed automatically. Good.
6. When the client quits, the daemon destructs his
   ptytty_proxy object: which amounts to sending
   a "destroy" request to the helper (proxy.C:135).
7. Upon receiving a "destroy" request, the helper
   closes both pty and tty in his ptytty_unix
   struct. Good.
8. The end.

The problem is that at this moment the daemon still
has his pty fd open, and thus the terminal pair can
not be released. The problem went away when I added
close(pty) to the destructor method of ptytty_proxy.

Best regards,
Andrei