Open Hub
Open Hub

 Settings |  Report Duplicate

0
I Use This!
Very Low Activity

Commits : Listings

Analyzed about 4 hours ago. based on code collected 1 day ago.
Nov 05, 2025 — Nov 05, 2026
Commit Message Contributor Files Modified Lines Added Lines Removed Code Location Date
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
------------------------------------------------------------------------------------------------------------------------------------ Spark Version Upgrade for Recursive Gloss Overlap Graph Intrinsic merit and Discrete Hyperbolic Factorization Cloud Implementations ------------------------------------------------------------------------------------------------------------------------------------ *) Re-executed Interview Algorithm and Factorization Spark Cloud implementations with Spark 2.1.0 and Cython optimization for Recursive Gloss Overlap Graph construction *) Earlier these were benchmarked with Spark 1.5.x *) Added debug statements in python-src/DiscreteHyperbolicFactorizationUpperbound_Bitonic_Spark.py and file locations updated. Logs committed to python-src/testlogs and python-src/InterviewAlgorithm/testlogs
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document - Majority Version of XOR Lemma (Section 318) has been rewritten with more analysis and error correction
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
------------------------------------------------------------------------------------------------------------------------------- Java Spark Streaming with NetCat WebServer update and some analysis on HiveServer2+Spark Integration - 18 February 2017 ------------------------------------------------------------------------------------------------------------------------------- *) Rewrote SparkGenericStreaming.java mapreduce functions with Spark 2.1.0 + Java 8 lambda functions *) Enabled Netcat Socket streaming boolean flag instead of URL socket with Jsoup *) With above changes (and no HiveServer2), socket streaming receiver works and Hive saveAsTable()/Parquet file saving works. *) HiveServer2 was enabled in Spark Conf by adding hive-site.xml in spark/conf directory. (Two example hive-site.xml(s) have been committed to repository.) *) Spark 2.1.0 was started with hive-site.xml to connect to HiveServer2 2.1.1. This resulted in 2 out of heapspace errors as documented in testlogs/ with exception "Error in instantiating HiveSessionState". *) Such OOM errors with HiveServer2 and Spark have been reported earlier in Apache JIRA. Instead of TTransport, FrameTransport was also tried which caused "Frame size exceeds maximum frame size" errors. SASL was also disabled in hive-site.xml. *) Reason for such OOM errors in HiveServer2 seems to be heavy memory footprint of HiveServer2 and Spark together and CPU usage reaches 100% for both cores. Might require a high-end server with large RAM size so that heapspace is set to atleast 8GB. HiveServer2 is known to be memory intensive application. Also Spark only supports Hive 1.2.1 and not Hive 2.1.1 which could be the reason. *) Increasing Java Heap space in spark config file (spark.executor.memory and spark.driver.memory) also did not have effect. *) Setting HADOOP_HEAPSIZE also did not have effect. OOM error occurs while instantiating HiveSessionState and bootstrapping Hive databases (get_all_databases). Isolated beeline CLI connection works though with HiveServer2. *) Hence presently Spark Streaming works with Spark provided metastore_db Hive Support and Parquet file support. *) testlogs/ have stream processing logs for NetCat (NC) webserver.
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
*) Changed number of variables and clauses to 14 and 14 with some additional debug statements. *) After 4000+ iterations ratio of random 3SAT instances satisfied is: ~70%
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
Updated Copyleft notices
Srinivasan Kannan
More... over 9 years ago
Deleted SparkGenericStreaming.py
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
------------------------------------------------------------------------------------------------------------------- Spark Streaming Java Update - Java Bean DataFrame(Dataset) creation, Hive Metastore support and persistence of DataFrame to Hive and as a Parquet file ------------------------------------------------------------------------------------------------------------------- *) New JavaSparkSingletonInstance class for singleton Spark Context has been added. *) New Java Bean Word.java has been added for creating DataFrame from JavaRDD iterable *) DataFrame has been persisted to filesystem as .parquet file in overwrite mode of DataFrameWriter *) DataFrame has also been persisted to Hive MetaStore db which Spark supports (Shark SQL) with saveAsTable() *) SparkGenericStreaming has been rewritten to do transformations on words JavaRDD iterable with VoidFunction() in 2-level nested iterations of foreachRDD() and forEach() *) map() transformation for each row in JavaRDD throws a null pointer exception and inner call() is never invoked. *) This has been remedied by replacing map() with a forEach() and storing the rows in each RDD in an ArrayList *) With Hive MetaStore + Shark integration, Streaming_AbstractGenerator.py abstraction now has access to Java Spark Streaming because it already supports Hive via thrift and NeuronRain now has ability to analyze any realtime or batch unstructured data. *) Primitive ETL to just scrape words in URL text sufficiently tracks trending topics in social media. *) Hive metastore db, Spark warehouse data have been committed and Logs for this has been added to testlogs/ *) sparkgenericstreaming.jar has been repackaged with recompilation
Srinivasan Kannan
More... over 9 years ago
logs for an example news streaming with rewritten SparkGenericStreaming class
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
--------------------------------------------------------------------------------------------------------------- Java Spark Streaming Generic Receiver update - 9 February 2017 --------------------------------------------------------------------------------------------------------------- *) Objectified the SparkGenericStreaming more by moving the SparkConf and JavaStreamingContext into class private static data. *) new method SparkGenericStreamingMain() instantiates the Spark Context and returns the JavaPairDStream<String,Integer> wordCounts object *) main() static method instantiates SparkGenericStreaming class and receives wordCounts JavaPairDStream from SparkGenericStreamingMain() method *) Spark JavaStreamingContext output operations are invoked by start() and print actions by lambda expressions. *) SparkConf and JavaStreamingContext variables have to be static because Not Serializable exceptions are thrown otherwise. *) sparkgenericstreaming.jar has been repackaged by recompilation.
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
Updated CNFSATSolver.py to create set of random 3CNFs and verify if the least squares assignment is satisfied. Presently it creates CNFs of 10 variables and 10 clauses in an infinite loop and prints a percentage of satisifed 3CNFs so far. Logs for this percentage after few hundred iterations has been committed to testlogs/. It shows an intriguing convergence at 80% (i.e 80% of 3CNFs are satisfied) after few hundred iterations of random CNFs
Srinivasan Kannan
More... over 9 years ago
Updated Requirements.txt
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... over 9 years ago
----------------------------------------------------------------------------------------------------------------------- Java Spark Streaming for Generic data - Jsoup ETL integration ----------------------------------------------------------------------------------------------------------------------- *) Jsoup HTML parser has been imported into SparkGenericStreaming.java and a very basic Extract-Transform-Load is performed on the URL passed in with a special boolean flag to use Jsoup *) Jsoup connects to URL and does a RESTful GET of the HTML, extracts text from HTML and stores the text in Spark RDD *) receive() is periodically invoked and word count is computed *) foreachRDD() is invoked on the DStream to iterate through all words in DStream. *) forEach() invokes a lambda function to print the word text *) Logs for this have been added to spark_streaming/testlogs/
Srinivasan Kannan
More... over 9 years ago
------------------------------------------------------------------------------------------------------------ Java Spark Streaming - Generic Stream Receiver Implementation ------------------------------------------------------------------------------------------------------------ Spark Streaming Java implementation to receive generic stream of unstructured text data from any URL and Sockets has been added to NeuronRain AsFer java/bigdata_analytics/. It is based on Spart 2.1.0 + Hadoop 2.7 single node cluster. Following are the setup prerequisites:
Srinivasan Kannan
More... over 9 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 10 years ago
---------------------------------------------------------------------------------------------------------- CNFSAT solver update - polynomial time approximation ----------------------------------------------------------------------------------------------------------
Srinivasan Kannan
More... almost 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 10 years ago
Updated AsFer Design Document
Srinivasan Kannan
More... almost 10 years ago