Looking at the LAD page got me thinking that PyTone probally need to look at supporting JACK. JACK was designed by the LAD folk for low-latency no dropout audio and they have tons of experience gettting it to happen.
Does PyTone have any plans to try and set any of the realtime process flags for the decoder thread? Most of these would require you to suid root for pytone but it may go a long way to solving your drop out problems.
I'm thinking that the dropouts may be caused by the garbage collector in python. #1 non-deterministic problem in embedded and realtime systems is memory management. Most of the time it falls into the "Don't do that" catagory when running realtime stuff.
Considering the ammount of RAM you guys are reporting PyTone using the GC may take a good bit of time to figure out what it needs to do. I've not played with the GC tweaking stuff in python but I seem to remember that there is more than one algo you can select for GC. Perhaps a less intrusive GC + realtime priority on the decoder thread might help?
Hi Richard,
On 10.08.05, Richard Smith wrote:
Looking at the LAD page got me thinking that PyTone probally need to look at supporting JACK. JACK was designed by the LAD folk for low-latency no dropout audio and they have tons of experience gettting it to happen.
Are there any Python bindings for JACK?
Does PyTone have any plans to try and set any of the realtime process flags for the decoder thread? Most of these would require you to suid root for pytone but it may go a long way to solving your drop out problems.
I'm thinking that the dropouts may be caused by the garbage collector in python. #1 non-deterministic problem in embedded and realtime systems is memory management. Most of the time it falls into the "Don't do that" catagory when running realtime stuff.
Could be but it could also be some other thread not releasing Python's global interpreter lock (GIL) (for instance during I/O). To alleviate this problem a bit, I'm planning to rewrite the small output buffering class in C. Then we won't be dependent on having the GIL anymore. This buffering thread then could obtain realtime priority, although I currently do not know how to do this.
Considering the ammount of RAM you guys are reporting PyTone using the GC may take a good bit of time to figure out what it needs to do. I've not played with the GC tweaking stuff in python but I seem to remember that there is more than one algo you can select for GC.
Python uses a reference counting based GC together with a regular mark and sweep GC run to collect objects with circular references.
You can get a few statistics using the gc module.
Jörg
Are there any Python bindings for JACK?
I foud the following on the JackIt site. There are also py scripts that can be used to control jack. But I've not used any of this.
http://www.corpuselectronica.com/software/pyjack
Could be but it could also be some other thread not releasing Python's global interpreter lock (GIL) (for instance during I/O). To
Hmmm... If that's the case then that might explain why the eric debugger has such a hard time with pytone.
alleviate this problem a bit, I'm planning to rewrite the small output buffering class in C. Then we won't be dependent on having the GIL anymore. This buffering thread then could obtain realtime priority, although I currently do not know how to do this.
From what I just read there dosen't seem to be a way to schedule the
priority of an individual thread. pthreads made some mention of it but I couldn't find out if it was supported under linux.
Right now to you start the python process with sched FIFO or sched RR. I can't remember which one is better. All python threads will then inherit that priority. That may or may not be any better than before.
remember that there is more than one algo you can select for GC.
Python uses a reference counting based GC together with a regular mark and sweep GC run to collect objects with circular references.
You can get a few statistics using the gc module.
Ah. So I'm guessing you have allready looked at this then?
On 11.08.05, Richard Smith wrote:
Are there any Python bindings for JACK?
I foud the following on the JackIt site. There are also py scripts that can be used to control jack. But I've not used any of this.
Thanks. I'll have a look into this when I find some time.
Could be but it could also be some other thread not releasing Python's global interpreter lock (GIL) (for instance during I/O). To
Hmmm... If that's the case then that might explain why the eric debugger has such a hard time with pytone.
alleviate this problem a bit, I'm planning to rewrite the small output buffering class in C. Then we won't be dependent on having the GIL anymore. This buffering thread then could obtain realtime priority, although I currently do not know how to do this.
From what I just read there dosen't seem to be a way to schedule the
priority of an individual thread. pthreads made some mention of it but I couldn't find out if it was supported under linux.
Actually, that's what I also remembered. Of course one could make the buffering code a separate process but this would complicate things to a certain degree. Then it's probably better to rely on some external daemon like JACK.
Right now to you start the python process with sched FIFO or sched RR. I can't remember which one is better. All python threads will then inherit that priority. That may or may not be any better than before.
remember that there is more than one algo you can select for GC.
Python uses a reference counting based GC together with a regular mark and sweep GC run to collect objects with circular references.
You can get a few statistics using the gc module.
Ah. So I'm guessing you have allready looked at this then?
Yes, indeed :-) And I wasn't able to find any "leak". Everything looked perfectly fine at least during my (not so extensive) tests...
Jörg
Actually, that's what I also remembered. Of course one could make the buffering code a separate process but this would complicate things to a certain degree. Then it's probably better to rely on some external daemon like JACK.
The following thread references a pthread call that can set sched RR or sched FIFO.
http://lalists.stanford.edu/lad/2001/Nov/0508.html
However it says later on that in some cases raising the priority may make things worse since it will force a usleep to busy wait. It also talks about tweaking you HZ value. For example we may get better results by cranking up HZ to 1000 Hz to increate the time tick resolution on reschedules.
A 1 ms tick is what I use on all my embedded sytems.
Sounds reasonable and should be an easy test.
Ok now I'm really headed to get some sleep..
Actually, that's what I also remembered. Of course one could make the buffering code a separate process but this would complicate things to a certain degree. Then it's probably better to rely on some external daemon like JACK.
The following thread references a pthread call that can set sched RR or sched FIFO.
Here's the scoop...
http://www.developertutorials.com/tutorials/linux/processes-threads-050616/p...
I foud the following on the JackIt site. There are also py scripts that can be used to control jack. But I've not used any of this.
I just read though most of the README on the pyjack site and dosen't seem like JACK will be of any real help to PyTone. If you ever wanted to integrate with a buch of sound FX stuff or other mixer type things then yes but not for a single process cranking out a buffered stream.
Has anyone run a large setup under a kernel with Oprofile or any of the latency tools enabled?
Ugh.. I gotta go get some sleep..
Hi!
replying to myself.
On 11.08.05, Joerg Lehmann wrote:
Could be but it could also be some other thread not releasing Python's global interpreter lock (GIL) (for instance during I/O). To alleviate this problem a bit, I'm planning to rewrite the small output buffering class in C. Then we won't be dependent on having the GIL anymore. This buffering thread then could obtain realtime priority, although I currently do not know how to do this.
I've just checked in a C extension module, which does the buffering and thus does not have to rely on holding the GIL. Together with a large enough bufsize in the [players.main] section, this should improve the dropout situation a bit. Of course, we still need realtime priority (I'll have a look into this later)... and the Linux kernel also has to behave properly ;-)
Please test the lastest SVN version and tell me whether you see any improvement (I hope so, writing the extension module was quite some work ;-) )
Jörg
-
Joerg Lehmann -
Richard Smith