Open Hub
Open Hub

 Settings |  Report Duplicate

0
I Use This!
Inactive

Commits : Listings

Analyzed about 4 hours ago. based on code collected about 4 hours ago.
Jan 22, 2025 — Jan 22, 2026
Commit Message Contributor Files Modified Lines Added Lines Removed Code Location Date
Updated USBmd_notes.txt
Srinivasan Kannan
More... about 7 years ago
----------------------------------------------------------------------------------------------------------- (FEATURE) USBmd FTrace Kernel Function CallGraph Generation for Analysis - 22 November 2018 ----------------------------------------------------------------------------------------------------------- 1.New bash shell script usb_md_ftrace.sh has been committed to repository which writes out an ftrace.log file containing kernel function call graph sequences for an executable code. It is invoked as: $usb_md_ftrace.sh <executable-to-trace> usb_md_ftrace.sh summarizes previously mentioned ftrace options enabling commands into single file with an option for commandline argument of an executable to trace. 2.usb_wwan_modified/python-src/Spark_USBWWANLogMapReduceParser.py has been changed to include a new function ftrace_callgraph_dot() which parses an ftrace log generated by usb_md_ftrace.sh for command: $usb_md_ftrace.sh traceroute <ip-address> 3.ftrace_callgraph_dot() parses each line of ftrace.log and adds them as edges in a NetworkX Directed Graph. DOT file for this call graph is written to Spark_USBWWANLogMapReduceParser.ftrace_callgraph.dot 4.As a novelty, PageRank and Degree Centrality measures of the call graph NetworkX DiGraph are printed which show the prominently active regions of the kernel for traceroute . PageRank/Degree Centrality of kernel function callgraph is quite useful by treating every function caller as a voter to function callee. Theoretically, this centrality in kernel throws light on suspicious, malevolent invocations particularly involving memory and locking. In this traceroute ftrace example, lock and kmalloc functions have high centrality, and USB URB related functions are way down the ranking. More the ranking, deeper the function is in callstack trace in kernel. 5. Lot of functions have ISRA optimization of GCC. ISRA is known to cause signed int bugs (0 was erroneously promoted to 1 in loops) and ISRA has been disabled in ARM kernel: https://patchwork.kernel.org/patch/7113091/ by -fno-ipa-sra GCC flag. This kind of instability could be the reason for 32-bit VIRGO heisenbugs in string functions in older kernels. 6.Previous FTrace kernel call graph analysis is not only limited to USBmd WLAN analytics but can be applied to any executable requiring kernel profiling. Usual profilers measure time spent in the function whereas this graph theoretic analysis is superior and finds kernel bottlenecks and malicious patterns by analyzing call graphs within kernel. 7. Malicious code (e.g virus, worms, root-kits, bots, keystroke loggers) are usually associated with high cpu and memory footprint causing abnormal traffic. Analyzing infected kernel callgraph patterns might help in identifying the root cause. 8. FTrace kernel function call graph complements already implemented Program Analyzers: SATURN CFG driver in VIRGO kernels (accessible only in kernelspace) and Valgrind/KCachegrind/Callgrind userspace call graph analyzer in AsFer. By this kernel activity is partially visible and can be analyzed graph theoretically from userspace. 9. Outbreak of epidemics have been analyzed as Game Theoretic problem (https://blogs.cornell.edu/info2040/2016/09/16/game-theory-in-the-context-of-epidemics/) - on how people behave in epidemics and their conclusion - "faster information limits disease spread". Cybercrimes are epidemics counterpart in cloud of computers only difference being damage inflicted on intellectual property than humans and adversaries are hackers/malicious code in place of viri. This makes Cybercrimes a multi-player adversarial game involving Hackers/Malicious code Versus Aggrieved. Translating the previous conclusion to cybercrimes: Faster information about malicious code limits the damage.
Srinivasan Kannan
More... about 7 years ago
Updated USBmd_notes.txt
Srinivasan Kannan
More... about 7 years ago
------------------------------------------------------------------------------------------------------------- (FEATURE) USBWWAN analytics - USBmon and FTrace logs analysis - 15 November 2018 ------------------------------------------------------------------------------------------------------------- 1. Logs Analysis for 2 standard kernel tracing facilities have been included - USBmon and FTrace. USBmon is the kernel debugfs tracing facility and FTrace is the Kernel functions tracing utility accessible from user space. (Kernel Address Sanitizer - KASAN - is only enabled in kernelspace via KBuild config and kernel build transparent to userspace) 2. USBmon traces are enabled by debugfs in /sys/kernel/debug/usb/usbmon and can be loaded in wireshark in libpcap format or usbmon pseudodevices can be viewed in tcpdump: 467 ls /sys/kernel/debug/ 468 modprobe usbmon 472 dumpcap -D 474 ls /dev/usbmon0 475 ls -lrt /dev/usbmon* 487 tcpdump -i usbmon1 488 tcpdump -i usbmon2 489 tcpdump -i usbmon0 490 tcpdump -i usbmon3 491 tcpdump -i usbmon4 520 cat /sys/kernel/debug/usb/usbmon/1t 2>&1 > usbmon.mon 3. FTrace for function graph analysis are enabled by (Kernel.org FTrace Documentation: https://www.kernel.org/doc/Documentation/trace/ftrace.txt): 536 ls /sys/kernel/debug/tracing/current_tracer 537 echo nop > /sys/kernel/debug/tracing/current_tracer 538 echo 0 > /sys/kernel/debug/tracing/tracing_on 539 echo $$ > /sys/kernel/debug/tracing/set_ftrace_pid 541 echo function > /sys/kernel/debug/tracing/current_tracer 545 echo 1 > /sys/kernel/debug/tracing/tracing_on 557 ls -lrt /sys/kernel/debug/tracing/trace 561 cat /sys/kernel/debug/tracing/set_graph_function 562 cat /sys/kernel/debug/tracing/trace_options 563 echo funcgraph-duration > /sys/kernel/debug/tracing/trace_options 566 cat /sys/kernel/debug/tracing/set_graph_function 567 cat /sys/kernel/debug/tracing/trace_options 568 cat /sys/kernel/debug/tracing/trace_options 569 echo funcgraph-cpu 2>&1 > /sys/kernel/debug/tracing/trace_options 620 cat /sys/kernel/debug/tracing/set_ftrace_pid 624 echo 7379 > /sys/kernel/debug/tracing/set_ftrace_pid 625 cat /sys/kernel/debug/tracing/trace 2>&1 > ftrace.log.15November2018 639 export JAVA_HOME=/media/Ubuntu2/jdk1.8.0_171/ 640 export PATH=/usr/bin:$PATH 671 /media/Ubuntu2/spark-2.3.1-bin-hadoop2.7/bin/spark-submit Spark_USBWWANLogMapReduceParser.py 2>&1 > testlogs/Spark_USBWWANLogMapReduceParser.FTraceAndUSBMon.log.15November2018 4. FTrace traces for specific userspace threads/processes are enabled by previous example commandlines and available through /sys/kernel/debug/tracing/trace (circular buffer). Function graph traces show kernel function invocations as call graph edges (denoted by fn2 <- fn1) 5. Spark_USBWWANLogMapReduceParser.py has been changed to invoke log analyzer for USBmon and FTrace logs for patterns Bi(BULK IN) and usb from USBmon and FTrace logs respectively: - usbmon.15November2018.mon - ftrace.ping.log.15November2018 (ftraces for ping of an IP address) 6. Logs for Spark Analyzer have been committed to Spark_USBWWANLogMapReduceParser.FTraceAndUSBMon.log.15November2018 which analyze the USBmon logs and WLAN traffic for IP address ping. 7. include/linux/serial.h has been committed (similar to other versions of USBmd - 32 and 64 bits - in SourceForge,GitHub and GitLab)
Srinivasan Kannan
More... about 7 years ago
Updated USBmd_notes.txt
Srinivasan Kannan
More... over 7 years ago
-------------------------------------------------------------------------------------------------------------- USBWWAN Kernel Log Spark Analyzer Update - Refactoring to a new python function -------------------------------------------------------------------------------------------------------------- 1. Spark Log Analyzer Spark_USBWWANLogMapReduceParser.py has been changed to modularize the pattern extraction by defining a new function accepting kern.log file, pattern and filter and also creates Spark DataFrame SQL table and queries it. 2. This is similar to NeuronRain AsFer log_mapreducer()
Srinivasan Kannan
More... over 7 years ago
Updated USBmd_notes.txt
Srinivasan Kannan
More... over 8 years ago
Updated USBmd_notes.txt
Srinivasan Kannan
More... over 8 years ago
(*) Added a substring match filter to RDD map/reduce transformations chain (*) Presently hardcoded as "+0x" which extracts all kernel functions invoked from Kernel Address Sanitizer kern.log and their frequencies
Srinivasan Kannan
More... over 8 years ago
Updated USBmd_notes.txt
Srinivasan Kannan
More... over 8 years ago
(*) Upgraded Spark version to 2.1.0 on Hadoop 2.7 (*) Changed to SparkContext text file instead of reading the input kernel log in python I/O (*) Added flatMap to front of MapReduce chain of transformations for tokenizer (*) Changed the input kernel log to 64bit 4.10.3 Kernel Address Sanitizer enabled kern.log which prints lot of debugging information on memory accesses especially for USBWWAN and USB Storage drivers. (*) This is an alternative to traditional promiscuous USB Analyzers like WireShark to get kernel stack traces for USB and WLAN operations. (*) Particularly useful in malware related untoward memory access and traffic analysis (*) Unifies Kernel Address Sanitizer, USB storage/WLAN driver and Spark Cloud for analytics (*) Logs for this have been committed to testlogs/ and python-src/testlogs
Srinivasan Kannan
More... over 8 years ago
Updated USBmd_notes.txt
K.Srinivasan
More... over 8 years ago
Updated USBmd_notes.txt
K.Srinivasan
More... over 8 years ago
Updated USBmd_notes.txt
Srinivasan Kannan
More... over 9 years ago
1. Miscellaneous code changes to suppress kbuild warnings - cast etc., 2. PRINT_BUFFER block changed to print the bytes in single line for each buffer
Srinivasan Kannan
More... over 9 years ago
1.New build script for drivers/usb top level folder has been added. 2.Copyleft notices updated 3.print_buffer() in usb.h has been #ifdef-ed based on a build time flag to suppress the buffer bytes dump preferentially so that kern.log is not flooded. 4.Flag PRINT_BUFFER has to be defined with #define somewhere within KBuild makefiles or externally. 5..ko files rebuilt
Srinivasan Kannan
More... over 9 years ago
Updated USBmd notes
Srinivasan Kannan
More... about 10 years ago
Updated USBmd notes
Srinivasan Kannan
More... about 10 years ago
- New Parsed kern.log with buffer bytes and Spark MapReduce logs
Srinivasan Kannan
More... about 10 years ago
Updated USBmd notes
Srinivasan Kannan
More... about 10 years ago
Added dict() and sort() for query results and printed cardinality of the stream data set which is the size of the dict. An example log has been added which prints the cardinality as ~250. In contrast, LogLog and HyperLogLog counter estimations approximate the cardinality to 140 and 110 respectively.
Srinivasan Kannan
More... about 10 years ago
- added Spark Mapreduce and DataFrame log for USBWWAN byte stream - added a parsed kern.log with only bytes from USBWWAN stream
Srinivasan Kannan
More... about 10 years ago
Updated USBmd notes
Srinivasan Kannan
More... about 10 years ago
Updated USBmd notes
Srinivasan Kannan
More... about 10 years ago
- Apache Spark script for analyzing the USBWWAN byte stream logs has been updated with byte counts map-reduce functions from print_buffer() logs and temp DataFrame Table creation with SparkSQL. - logs for the script have been added in usb_wwan_modified/python-src/testlogs/Spark_USBWWANLogMapReduceParser.out.3December2015 - kern.log parser shellscript has been updated
Srinivasan Kannan
More... about 10 years ago
Updated USBmd notes
Srinivasan Kannan
More... about 10 years ago
Updated USBmd notes
Srinivasan Kannan
More... about 10 years ago
Added an Apache Spark MapReduce python script to compute byte frequency in parsed print_buffer() kern.log
Srinivasan Kannan
More... about 10 years ago
- changed print_buffer() printk() to print a delimiter in each byte for AsFer Machine Learning code processing - add a parser script for kern.log to print print_buffer() lines - parsed kern.log with print_buffer() lines has been added
Srinivasan Kannan
More... about 10 years ago
Updated USBmd notes
Srinivasan Kannan
More... about 10 years ago