Open Hub
Open Hub

 Settings |  Report Duplicate

0
I Use This!
Very Low Activity

Commits : Listings

Analyzed about 3 hours ago. based on code collected 1 day ago.
Nov 05, 2025 — Nov 05, 2026
Commit Message Contributor Files Modified Lines Added Lines Removed Code Location Date
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Number to factorize has been changed to 147. The mergedtiles array has been set to size 256. The logs for this have been committed in testlogs/. The factors are printed in last few lines of the logs in shuffled globalcoordinates corresponding to 147 in sorted globalmergedtiles accumulator.
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
New threading function assign_compareexchange_multithreaded() has been implemented which is invoked in lieu of sequential for loop assignment , by creating a thread for each pair of (i,i+midpoint) compare-exchange assignment of accumulator globalmergedtiles.This thread function assigns the Spark Mapreduce result of Compare and Exchange Mulithreading is an alternative for global state mutability in the absence of Spark support and it does not require any third party in-memory cacheing products Coordinates are also shuffled first few elements of which correspond to number to factor N are the factors found finally. Multithreaded assignment maps to a multicore parallelism. Logs for this have been committed to testlogs with and without Coordinates Shuffling.
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
------------------------------------------------------------------------------------- Accumulator support for Bitonic Sort mapreduce globalmergedtiles ------------------------------------------------------------------------------------- - Spark context has been parametrized and is passed around in the recursion so that single spark context keeps track of the accumulator variables - globalmergedtiles_accum is the new accumulator variable which is created in driver and used for storing distributed mutable globalmergedtiles - AccumulatorParam has been overridden for Vectors as globals - Spark does not permit however, at present, mutable global state within worker tasks and only driver creates the accumulator variables which can only be incrementally extended within closure workers. - muting or accessing accumulator.value within worker raises Exception: Can't access value of accumulator in worker tasks - Logs for above have been committed to testlogs/ - Because of above limitation of Spark in global mutability, accumulator globalmergedtiles is updated only outside the Spark mapreduce closures. - Support for global mutables, if Spark has it, would make compare-and-exchange phase closer in behaviour to an NVIDIA Parallel bitonic sort. - Documentation on this necessary feature is scarce and some suggest use of datastores like Tachyon/Alluxio for distributed shared mutables which is non-trivial. This cloud parallel sort implementation achieves thus almost ~90% of NVIDIA CUDA parallel bitonic sort - CompareAndExchange is done in parallel but assigning CompareAndExchange results is done sequentially.
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
- The Map step now does both Compare and Exchange within local worker nodes for each closure and result is returned in tuples - The driver collect()s and just assigns the variables based on comparator boolean field in tuple which are already exchanged within local worker
Srinivasan Kannan
More... over 10 years ago
- Printed return value of bitonic_sort() in sorted. Earlier it was wrongly printing globalmergedtiles a global state which is not changed anymore - logs for bitonic sort of tesselated hyperbolic arc
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
- Indeed Bitonic Sort from previous commit itself works without any issues. - Nothing new in this except reduced sort array of size just 64 for circumventing frequent Py4Java Out of Memory errors and for quick runtime - Logs for this have been committed in testlogs/ - the debug line "merged sorted halves" is the final sorted tiles array - .coordinates and .mergedtiles files have been updated
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Interim commits 3 for Bitonic Sort Spark NC Factorization Implementation ------------------------------------------------------------------------- - AsFer Design Document updated for already implemented NVIDIA CUDA Parallel C reference code for bitonic sort on hypercube - Bitonic Sort C++ code updated for max size of sort array - reduced from 16384 to 256 - Bitonic Sort Python Spark implementation updated for returning a tuple containing index info also instead of plain boolean value because Spark collect() after a comparator map() does not preserve order and index information was lost. - if clauses in exchange phase have been appropriately changed - Still there are random mysterious jumbled tuples which will be looked into in future commits - This implementation becomes redundant because of already existing better NVIDIA CUDA Parallel C implementation and is of academic interest only.
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Interim commits 2 for ongoing investigation of race conditions in Spark Bitonic Sort: -------------------------------------------------------------------------------------- - bidict usage has been removed because it requires bidirectional uniqueness, there are better alternatives - The variable exchange code has been rewritten in a less complicated way and just mimicks what sequential version does in CompareAndExchange phase. - logs for Sequential version has been committed - number to factor has been set to an example low value - .coordinates and .mergedtiles files have been regenerated - Still the race condition remains despite being behaviourally similar to sequential version except the parallel comparator array returned b mapreduce - Parallel comparator preserves immutability in local worker nodes
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Interim Commits for ongoing investigation of a race condition in Spark MapReduce for NC Factorization with Bitonic Sort : ------------------------------------------------------------------------------------------------------------------------- - The sequential version of bitonic sort has been updated to do away with usage of boolean flag up by 2 compare functions for True and False - Spark NC implementation of Bitonic Sort for Factorization still has some strange behaviour in sorting. In progress commits for this do following in Compare-And-Exchange phase: - Comparator code has been rewritten to just do comparison of parallelized RDD set of tuples where each tuple is of the form: (i, i+midpoint) - This comparison returns a set of boolean flags collect()ed in compare() - The variable exchange is done sequentially at each local node. - This is because Spark documentation advises against changing global state in worker nodes (though there are exceptions in accumulator) - There were some unusual JVM out of memory crashes, logs for which have been committed - some trivial null checks in complement.py - Bitonic Sort presents one of the most non-trivial cases for parallelism - parallel variable compare-and-exchange to be specific and present commits are a result of few weeks of tweaks and trial-errors. - Still the race condition is observed in merge which would require further commits.
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
- Sundrry changes for Cython rebuild of Interview algorithm PySpark implementation - logs for testlogs/DiscreteHyperbolicFactorizationUpperbound_Bitonic_Spark.bidict.out.18April2016 with reduceByKey() to reduce() and shuffling changes
Srinivasan Kannan
More... over 10 years ago
- C++ implementation for Bitonic Sort step in Factorization has been changed to remove all paddings - example number to factor has been changed to a small number - new array for storing shuffled coordinates has been introduced - shell script for building and executing C++-Python-Spark code has been changed for PySpark submit commandline - new textfile for storing the coordinates array has been included - DiscreteHyperbolicFactorizationUpperbound_Bitonic.coordinates - Python Spark script has been modified for handling coordinates shuffling - Ironically reduceByKey() didn't work, comparator was not invoked and bidict mysteriously has the data already in sorted order (?!). This necessitated rewriting reduceByKey() as reduce() in mapreduce step of parallelized data so that comparators and shuffling are done. Some weird behaviour prevents reading entire mergedtiles data into bidict. This is still a murky kludge and might require further investigation.
Srinivasan Kannan
More... over 10 years ago
- C++ implementation for Bitonic Sort step in Factorization has been changed to remove all paddings - example number to factor has been changed to a small number - new array for storing shuffled coordinates has been introduced - shell script for building and executing C++-Python-Spark code has been changed for PySpark submit commandline - new textfile for storing the coordinates array has been included - DiscreteHyperbolicFactorizationUpperbound_Bitonic.coordinates - Python Spark script has been modified for handling coordinates shuffling - Ironically reduceByKey() didn't work, comparator was not invoked and bidict mysteriously has the data already in sorted order (?!). This necessitated rewriting reduceByKey() as reduce() in mapreduce step of parallelized data so that comparators and shuffling are done. Some weird behaviour prevents reading entire mergedtiles data into bidict. This is still a murky kludge and might require further investigation.
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Updated Requirements.txt
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 10 years ago