Open Hub
Open Hub

 Settings |  Report Duplicate

4
I Use This!
Moderate Activity

Commits : Listings

Analyzed 1 day ago. based on code collected 1 day ago.
Dec 18, 2024 — Dec 18, 2025
Commit Message Contributor Files Modified Lines Added Lines Removed Code Location Date
HSCondition: Fixed terrible bug which it inherited from its default python Condition implementation, related to the notify method not being treadsafe. Although I was aware of it, I missed the first check which tests for the size - the result could be incorrect if the whole method wasn't locked. Testing runs stable now, allowing to move on \!
Sebastian Thiel
More... over 15 years ago
HSCondition: now gets a lock even in the single-notify case, as it was required due to the non-atomiciy of the invovled operation. Removed one level of indirection for the lock, by refraining from calling my own 'wrapper' methods, which brought it back to the performance it had before the locking was introduced for the n==1 case
Sebastian Thiel
More... over 15 years ago
HSCondition: now deriving from deque, as the AsyncQeue does, to elimitate one more level of indirection. Clearly this not good from a design standpoint, as a Condition is no Deque, but it helps speeding things up which is what this is about. Could make it a hidden class to indicate how 'special' it is
Sebastian Thiel
More... over 15 years ago
HSCondition: using a deck to store waiters, for further speedup
Sebastian Thiel
More... over 15 years ago
thread: fixed initialization problem if an empty iterable was handed in queue: Queue now derives from deque directly, which safes one dict lookup as the queue does not need to be accessed through self anymore pool test improved to better verify threads are started correctly
Sebastian Thiel
More... over 15 years ago
queue: fixed critical bug in the notify method, as it was not at all thread-safe, causing locks to be released multiple times. Now it runs very fast, and very stable apparently. Now its about putting previous features back in, and studying their results, before more complex task graphs can be examined
Sebastian Thiel
More... over 15 years ago
Merge branch 'queue' into async
Sebastian Thiel
More... over 15 years ago
workerthread: adjusted to use a blocking queue, it will receive termination events only with its queue, with boosts performance into brigt green levels
Sebastian Thiel
More... over 15 years ago
Revised task deletion works well, adjusted test to be creating new tasks all the time instead of reusing its own one, it was somewhat hard to manage its state over time and could cause bugs. It works okay, but it occasionally hangs, it appears to be an empty queue, have to gradually put certain things back in, although in the current mode of operation, it should never have empty queues from the pool to the user
Sebastian Thiel
More... over 15 years ago
task: now deletes itself once its done - for the test this doesn't change a thing as the task deletes itself too late - its time for a paradigm change, the task should be deleted with its RPoolChannel or explicitly by the user. The test needs to adapt, and shouldn't assume anything unless the RPoolChannel is gone
Sebastian Thiel
More... over 15 years ago
Its getting better already - intermediate commit before further chaning the task class
Sebastian Thiel
More... over 15 years ago
queue: adjusted queue to be closable ( without own testing yet, except for the pool which runs it ) - its not yet stable, but should be solvable.
Sebastian Thiel
More... over 15 years ago
The new channeldesign actually works, but it also shows that its located at the wrong spot. The channel is nothing more than an adapter allowing to read multiple items from a thread-safe queue, the queue itself though must be 'closable' for writing, or needs something like a writable flag.
Sebastian Thiel
More... over 15 years ago
both versions of the async queue still have trouble in certain situations, at least with my totally overwritten version of the condition - the previous one was somewhat more stable it seems. Nonetheless, this is the fastest version so far
Sebastian Thiel
More... over 15 years ago
test implementation of async-queue with everything stripped from it that didn't seem necessary - its a failure, something is wrong - performance not much better than the original one, its depending on the condition performance actually, which I don't get faster
Sebastian Thiel
More... over 15 years ago
Merge branch 'stasks' into async
Sebastian Thiel
More... over 15 years ago
Task scheduled items lock now uses a dummy lock in serial mode, improving its performance considerably. Channels now use the AsyncQueue, boosting their throughput to about 5k items / s - this is something one can work with, considering the runtime of each item should be large enough to keep the threads busy. This could be a basis, further testing needed
Sebastian Thiel
More... over 15 years ago
Channel now uses the AsyncQueue, boosting performance by factor 4, its a start
Sebastian Thiel
More... over 15 years ago
introduced a new counter keeping track of the scheduled tasks - this prevent unnecessary tasks to be scheduled as we keep track of how many items will be produced for the task at hand. This introduces additional locking, but performns well in multithreaded mode. Performance of the master queue is still a huge issue, its currently the limiting factor, as bypassing the master queue in serial moode gives 15x performance, wich is what I would need
Sebastian Thiel
More... over 15 years ago
improved testing to test the actual async handling of the pool. there are still inconsistencies that need to be fixed, but it already improved, especially the 4-thread performance which now is as fast as the dual-threaded performance
Sebastian Thiel
More... over 15 years ago
task: Fixed incorrect handling of channel closure. Performance is alright for up to 2 threads, but 4 are killing the queue
Sebastian Thiel
More... over 15 years ago
Moved pool utilities into util module, fixed critical issue that caused havok - lets call this a safe-state
Sebastian Thiel
More... over 15 years ago
added high-speed locking facilities, allowing our Queue to be faster, at least in tests, and with multiple threads. There is still an sync bug in regard to closed channels to be fixed, as the Task.set_done handling is incorrecft
Sebastian Thiel
More... over 15 years ago
Added task order cache, and a lock to prevent us walking the graph while changing tasks Now processing more items to test performance, in dual-threaded mode as well, and its rather bad, have to figure out the reason for this, probably gil, but queues could help
Sebastian Thiel
More... over 15 years ago
changed scheduling and chunksize calculation in respect to the task.min_count, to fix theoretical option for a deadlock in serial mode, and unnecessary blocking in async mode
Sebastian Thiel
More... over 15 years ago
pool.consumed_tasks: is now a queue to be thread safe, in preparation for multiple connected pools Reduced waiting time in tests to make them complete faster
Sebastian Thiel
More... over 15 years ago
pool: First version which works as expected in async mode. Its just using a single task for now, but next up are dependent tasks
Sebastian Thiel
More... over 15 years ago
channel.read: enhanced to be sure we don't run into non-atomicity issues related to our channel closed flag, which is the only way not to block forever on read(0) channels which were closed by a thread 'in the meanwhile'
Sebastian Thiel
More... over 15 years ago
Plenty of fixes in the chunking routine, made possible by a serialized chunking test. Next up, actual async processing
Sebastian Thiel
More... over 15 years ago
First step of testing the pool - tasks have been separated into a new module including own tests, their design improved to prepare them for some specifics that would be needed for multiprocessing support
Sebastian Thiel
More... over 15 years ago