Open Hub
Open Hub

 Settings |  Report Duplicate

0
I Use This!
Very Low Activity

Commits : Listings

Analyzed about 4 hours ago. based on code collected 1 day ago.
Nov 05, 2025 — Nov 05, 2026
Commit Message Contributor Files Modified Lines Added Lines Removed Code Location Date
- In InterviewAlgorithm/InterviewAlgorithmWithIntrinisicMerit_SparkMapReducer.py increased local threads to 2 (for dual core cpu(s)) for all SparkContexts instantiated - Added screenshot and logs for 2 local threads SparkContext mapreduce
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
- updated spidered text - best_matching_synset() enabled in mapreduce backedges computation which accurately disambiguates the graph. Spark single node cluster is significantly slower than serial version of parents() computation with only python calls probably due to costly Python4Java back-and-forth stream socket reads. - logs, DOT file and screenshots for single node Spark cluster have been added to testlogs/
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
- Spark Recursive Gloss Overlap Intrinsic Merit code has been optimized and some minutiae bugs have been resolved. - Map function for parents computation in Spark cluster has been updated to act upon only the previous recursion level glosses by proprietary pickling of the keyword into a file storage and loading it, and not as a function argument - Problems with previous level gloss tokens were almost similar to MapReduce functions of the recursion - New pickling file for parents computation in Spark has been added - logs and screenshots have been added to testlogs - With this, Spark Intrinsic Merit computation is highly parallellized apt for large clouds - Also a parents_tokens() function that uses gloss tokens instead of synsets has been added - New pickling dump() and load() functions have been added for keyword storage - pickling is synchronized with python threading lock acquire() and release(). Shouldn't be necessary because of Concurrent Read Exclusive Write (CREW) of keyword, but for safer side to prevent Spark-internal races.
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Some optimizations and redundant code elimination in python-src/InterviewAlgorithm/InterviewAlgorithmWithIntrinisicMerit_Crawl_Visual_Spark.py
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Logs,DOT file and Screenshots for mapreduced parents() computation
Srinivasan Kannan
More... almost 11 years ago
- Updated python-src/InterviewAlgorithm/InterviewAlgorithmWithIntrinisicMerit_Crawl_Visual_Spark.py for choosing parallel spark mapreduce or serial parents backedges computation based on a boolean function. - This is because Recursive tokens mapreduce computation (Spark_MapReduce) is faster than serial version. But serial parents computation is faster than parallel parents computation ironically(Spark_MapReduce_Parents). This happens on single node Spark cluster. So, parents computation has been made configurable(serial or parallel) - Logs and Screenshots for various texts experimented have been added to testlogs/ - python-src/InterviewAlgorithm/InterviewAlgorithmWithIntrinisicMerit_SparkMapReducer.py mapreduce uses best_matching_synset() for disambiguation now which was commented earlier. - MapFunction_Parents() has a pickling problem in taking a tuple of synset objects as input arg due to which synsets have to be recomputed that causes the slowdown mentioned above compared to serial parents() version. MapFunction_Parents() has been rewritten to take previous level gloss tokens in lieu of synsets to circumvent pickling issues. - Slowdown could be resolved on a huge cluster. - Thus this is a mix of serial+parallel implementation.
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Some updates to parallelism in mapreduce of backedges computation. Ongoing.
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
- Updated AsFer Design Document - Added more parallelism to python-src/InterviewAlgorithm/InterviewAlgorithmWithIntrinisicMerit_Crawl_Visual_Spark.py. New map-reduce functions for computing parents (backedges in the recursion) have been added - mapFunction_Parents(), reduceFunction_Parents(), SparkMapReduce_Parents() in python-src/InterviewAlgorithm/InterviewAlgorithmWithIntrinisicMerit_SparkMapReducer.py - These functions take the previous level tokens as inputs instead if synsets
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
- Updated AsFer Design Document -------------------------------------------------------------------------------------------------------------- Apache Spark MapReduce Parallel Computation of Interview Algorithm Recursive Gloss Overlap Graph Construction: -------------------------------------------------------------------------------------------------------------- - 2 python files for Spark MapReduce of Recursive Gloss Overlap graph construction has been added to repository - These two implement Interview Algorithm Recursive Gloss Overlap graph construction and Map-Reduce functions that parallellize each recursion step computing the gloss tokens from previous recursion level. This is the most important part of the algorithm which consumes lot of time in serial version. Apache Spark has been recently gaining importance over its plain Hadoop counterpart and is many times faster than Hadoop in benchmarks. - In this implementation, Resilient Distributed Dataset is the set of tokens in each recursion level and is parallelly computable in a huge Spark cluster. - For example if Spark cluster has 10000 nodes, and each level of recursion produces X number of gloss tokens for graph vertices, time complexity in Spark is O(X/10000) where as serial version is O(X) with negligible network overhead. Spark has in-memory datasets which minimizes network latency because of disk access. - There were lot of implementation issues to make the parallellism. Map and Reduce functions use namedtuples to return data. With more than one field in namedtuple, there are internal pickling issues in Py4J - Python4Java which PySpark internally invokes to get streamed socket bytes from Java side of the object wrapped as an iterable. This prevents returning multiple values - for example Synsets - in Map-Reduce functions. - So only tokens at a level are map-reduced and returned while the prevlevelsynsets (required for adding edges across vertices) are "pickled" with a properietary asfer_pickle_load() and asfer_pickle_dump() functions that circumvents the python and java vagaries in pickling. - With proprietary pickling and Map-Reduce functions, Recursive Gloss Overlap graph has been constructed in parallel. Screenshots for this has bee added in testlogs. - Proprietary pickling is done to a text file which is also added to repository. This file is truncated at the end of each recursion. - Subtlety here is that maximum number of tokens at a recursion level t = number_of_tokens_at_level_(t-1) * maximum_size_of_gloss_per_word(s) which grows as series 1 + s + s^2 + s^3 + ... + s^t. If size of the Spark cluster is O(s^t_max), each recursion step is O(1) and total time for all levels is O(t_max) which is per text document. This bound neglects optimization from overlaps and isomorphic nodes removal and is worst case upperbound. This implies size of cluster is independent of document size and depends only on depth of recursion. For ranking n documents this bound becomes O(n*t_max). For constant t_max this bound is O(n) - linear in number of documents and thus scalable. - Maximum size of gloss per word (s) is also an upper-boundable constant. With constant t_max , size of cluster O(s^t_max) is constant too.
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Commits for NeuronRain WebServer-RESTfulAPI-GUI based on tornado: - NeuronRain entrypoint based on tornado that reads a template and implements GET and POST methods - templates/ contains renderable html templates - testlogs/ has neuronrain GUI logs
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 11 years ago
Junction Tree for RGO graph - logs and screenshots
Srinivasan Kannan
More... almost 11 years ago