Friday, June 27, 2008

Python: subprocesses after all

Back in September 2007 I was sitting with dipesh until late in the night on a very interesting problem:
We wanted to implement a "tee"-like class in python which would work as the same command under unix: get input via one stream, double the input and give it out into a file and into a stream (like sys.stdout). This shouldn't be a big problem but when you try to give this class to subprocess.Popen as one of the streams you will find out that Popen doesn't use the write function but instead uses the fileno() function of the fileobject. But there would be a workaround for that:
We use subprocess.PIPE for the wanted input stream, then we check regularly for any stuff within the pipe - that this will not work on windows one could expect.
The problem is that all available read processes will block execution if not enough bytes can be read.

The solution is called pywin32 and is a python package for windows. Mark Hammond from starship has made them and they include a function called PeekNamedPipe which does exactly what we need: peek into the pipe without blocking it. I should leave the rest as a homework to the reader, but for those who don't want to invest any more research, I'll give a first solution from where you can start (look at the class Popen).


Sebastian Sauer said...

And thanks for sharing those research. Some more keywords I remember to have used to grep for that case;
python, popen2, popen3, popen4, os.waitpid, poll, child process, read, readline, write, flush, wait, thread, non-blocking, windows, win32, like at linux, posix, WTF?

yay, 10 months! :-)

Jay said...

I wanted to do something similar, I think.

I wanted to run programs and have their output redirected to the logging module. So I just called select() on stdout and stderr and when there was data to be read I'd read it and log it.

IIRC, is implemented for Windows as well. Does that not do what you want?

SaroEngels said...

hm, not sure about It is definitely not as easy as I would expect it from python. But maybe I have to dig still deeper into it.

Anonymous said...

I like to earn rappelz rupees, because it is very interesting. I think if you meet the rappelz gold, you will like it too. And when you play the game well you can get some cheap rappelz rupees as the rewards from the game. As long as you have a lot of rappelz money in the game you will be strong, and you can go to buy rupees with the other players.

Anonymous said...

Without hesitate, I bought eve isk , in the game I can find myself. I feel lonely, but I do not want to talk with anyone, so I buy isk . At present, think the happy day I spend in knight, I am eager to enter it, and buy eve online isk . Own eve online isk , it means that you own the life of happiness. So I will not leave cheap eve isk . It is the origin of the happiness.