Open Hub
Open Hub

 Settings |  Report Duplicate

0
I Use This!
Very Low Activity

Commits : Listings

Analyzed about 3 hours ago. based on code collected 1 day ago.
Nov 05, 2025 — Nov 05, 2026
Commit Message Contributor Files Modified Lines Added Lines Removed Code Location Date
1450. (THEORY and FEATURE) Astronomical and Archaeological analytics - Space Cone Search and Archaeoastronomical dating by Proper Motion - 6 September 2023 - related to 2,18,714,731,1206,1255,1236 and all sections on Merit of Large Scale Visuals, Archaeology, Text Restoration, Astronomical Datasets, Archaeology, Topological Script Recognition
Srinivasan Kannan
More... about 3 years ago
-------------------------------------------------------------------------------------------------------------------------------------------- 1450. (THEORY and FEATURE) Astronomical and Archaeological analytics - Space Cone Search and Archaeoastronomical dating by Proper Motion - 6 September 2023 - related to 2,18,714,731,1206,1255,1236 and all sections on Merit of Large Scale Visuals, Archaeology, Text Restoration, Astronomical Datasets, Archaeology, Topological Script Recognition -------------------------------------------------------------------------------------------------------------------------------------------- 1450.1 python-src/EphemerisSearch.py has been updated to import Gaia and angular_sepation() from astroquery and astropy. 1450.2 New function query_space(name,ra,dec,radius=0.0833333) has been implemented to search the outer space for celestial bodies in a cone of radius centered around a right ascension-declination point by SQL query to Gaia. 1450.3 New function locate_proper_motion_conjunction(queryresults1,queryresults2,duration=10000000,separation=0.01) compares the cone search query results from query_space() for 2 celestial bodies and computes the pairwise angular separation between the bodies within 2 cones over a range of years within a separation upper limit by multiplying the Proper Motion velocity values (pmra and pmdec values - right ascension and declination velocities in milliarcsecnds per annum). 1450.4 query_space() has been invoked for cones centered around 2 celestial bodies - Constellations of Hasta (Gamma Corvi) and Uttaraphalguni (Beta Leonis) and the angular separation for 8000 years is printed for every pair of celestial bodies (source_id) with in an upperlimit of 0.01: q1=query_space("Hasta",12.15,17.32) q2=query_space("Uttaraphalguni",11.49,14.34) locate_proper_motion_conjunction(q1,q2,duration=8000) 1450.5 Matching years for given separation limits in testlogs/EphemerisSearch.log.ArchaeoastronomySearch.6September2023. Angular separation between above two constellations are especially important to decipher the astronomical reference of 1255.26 in Ramayana on Hasta-Uttaraphalguni stellar occulation. 1450.6 Most available datings of Ramayana are in 7000 years BCE range and following are some of the cone search results matching this range when stars in Hasta and Uttaraphalguni constellations were quite close (more accurate date range could be retrieved by making separation in locate_proper_motion_conjunction(queryresults1,queryresults2,duration=10000000,separation=0.01) close to 0): Angular separation between two bodies 2781935220532827136 and 2777952583258551936 in 7278 years: 0.007010503463215622 Angular separation between two bodies 2781935220532827136 and 2777952514539071488 in 7346 years: 0.009569323848829388 Angular separation between two bodies 2781935220532827136 and 2777964819620884608 in 7351 years: 0.0096135191577218
Srinivasan Kannan
More... about 3 years ago
Section 1449 updated
Srinivasan Kannan
More... about 3 years ago
1449. (JIRA,THEORY and FEATURE) GIS Urban Sprawl Analytics - Updates to OECD-GHSL-JRC Functional Urban Area Lineation Regression Model for Global Human Settlement Layer - Chennai Metropolitan Area Example - 31 Augut 2023 - related to 823,1163,1164,1174,1175,1177,1179,1194,1211,1226,1234,1290,1291,1292,1295,1296,1297,1301,1302,1304,1312,1313,1320,1323,1326,1328,1342,1343,1349,1350,1351,1362,1363,1377,1386,1387,1395,1397,1398,1412,1414,1423,1448 and all sections on GIS Urban Sprawl Analytics, Merit of Large Scale Visuals, Gridded Population Estimators (SEDAC GPW4 and GHSL convex hull estimators), DBSCAN and Neural Network Urban Extent Learners, Computational Geometry, Intrinsic Merit Rankings, Verlhulste-R Ricker Population Estimation, Community detection, Social Network Analytics, Space filling by Curves, Unequal Circle and Polynomial packing, Computational Geometry and Polygonal packing by Voronoi Tessellation, Four color theorem, Economic Merit, Computational Economics - GDP estimation from Urban sprawl demographics, electricity consumption, Poverty alleviation, Preferential Attachment, Automatic Delineation of Urban Sprawls, Cellular Automata-SLEUTH-Polya Urn-Dynamic Facegraph Urban Growth Models,Expander Graphs,Timeseries analysis,Graph Edit Distance and Earth mover distance, Face recognition,Graph Isomorphism (Exact, Approximate and Subgraph),Graph matching,Graph similarity,Image Search,Quadrilateral Mesh Deformation and Isomorphism,Maximum Common Subgraph Problem and Approximate Topological Matching
Srinivasan Kannan
More... about 3 years ago
-------------------------------------------------------------------------------------------------------------------------------------------- 1449. (JIRA,THEORY and FEATURE) GIS Urban Sprawl Analytics - Updates to OECD-GHSL-JRC Functional Urban Area Lineation Regression Model for Global Human Settlement Layer - Chennai Metropolitan Area Example - 31 Augut 2023 - related to 823,1163,1164,1174,1175,1177,1179,1194,1211,1226,1234,1290,1291,1292,1295,1296,1297,1301,1302,1304,1312,1313,1320,1323,1326,1328,1342,1343,1349,1350,1351,1362,1363,1377,1386,1387,1395,1397,1398,1412,1414,1423,1448 and all sections on GIS Urban Sprawl Analytics, Merit of Large Scale Visuals, Gridded Population Estimators (SEDAC GPW4 and GHSL convex hull estimators), DBSCAN and Neural Network Urban Extent Learners, Computational Geometry, Intrinsic Merit Rankings, Verlhulste-R Ricker Population Estimation, Community detection, Social Network Analytics, Space filling by Curves, Unequal Circle and Polynomial packing, Computational Geometry and Polygonal packing by Voronoi Tessellation, Four color theorem, Economic Merit, Computational Economics - GDP estimation from Urban sprawl demographics, electricity consumption, Poverty alleviation, Preferential Attachment, Automatic Delineation of Urban Sprawls, Cellular Automata-SLEUTH-Polya Urn-Dynamic Facegraph Urban Growth Models,Expander Graphs,Timeseries analysis,Graph Edit Distance and Earth mover distance, Face recognition,Graph Isomorphism (Exact, Approximate and Subgraph),Graph matching,Graph similarity,Image Search,Quadrilateral Mesh Deformation and Isomorphism,Maximum Common Subgraph Problem and Approximate Topological Matching -------------------------------------------------------------------------------------------------------------------------------------------- 1449.1 Updated python-src/image_pattern_mining/ImageNet/GISUrbanSprawlAnalytics.py to import geodesic and great_circle functions from geopy (Haversine geodesic distances between two longitude-latitude spherical coordinates on earth) 1449.2 GHSL R2023A Degree of Urbanization SMOD pixel classification (Section 1312.7) has been encoded as a dictionary of class id to population per cell as per definition in Section 2.6.2 of GHSL R2023A report - https://ghsl.jrc.ec.europa.eu/documents/GHSL_Data_Package_2023.pdf: class2population={30:1500,23:1500,22:300,21:300,13:300,12:50,11:50,10:0} which is lookedup in GHSL regresssion as FUAij = b0 + b1*distij + b2*distij*distij + g1*class2population[sizeij] + p1*populationj + p2*populationj*populationj + w*gdp + epsilonij (sizeij is proxied by GHSL pixel class id which have population range of 0 to 1500 per sqkm) 1449.3 Definition of functional_urban_area_delineation_GHSL_regression(b0,b1,b2,distij,g1,sizeij,p1,p2,populationj,w,gdp,epsilonij,binarythreshold=1000000,lon=-1,lat=-1) has been extended to include a user-supplied binary threshold for FUAij and longitude-latitude of the GHSL cell as arguments 1449.4 Inclusion of FUAij in Urban Area is returned as a binary value of 1 or 0 depending on whether it exceeds or falls short of binary threshold if FUAij > binarythreshold: print("GHSL lon-lat ",(lon,lat)," could belong to functional urban area for given criteria") return 1 else: print("GHSL lon-lat ",(lon,lat)," may not belong to functional urban area for given criteria") return 0 1449.5 Definition for function urban_sprawl_from_raster(longx,latx,longy,laty,raster,dt,granularity=0.01,delineationparams=None,bboxcentroid=True) has been extended to include search granularity of longitude-latitude bounding box, delineation parameters for GHSL functional urban area regression, centroid of the bounding box which takes precedence over urban center long-lat in delineation parameters. If bboxcentroid is true midpoints of X-Y axes are computed for rectangle centroid. Hardcoded granularity has been replaced by user defined granularity argument. 1449.6 If bboxcentroid is True, geodesic() between bounding box centroid and GHSL cell long-lat is set as distance from urban center else geodesic() is computed from user supplied urban center within delineationparams. 1449.7 functional_urban_area_delineation_GHSL_regression() is invoked inside urban_sprawl_from_raster() - thereby integrating rasters with urban delineation - with following parameters for a manually found binary threshold of 1292769530000 (which is the midpoint of maximum possible FUAij value). functional_urban_area_delineation_GHSL_regression(delineationparams[0],delineationparams[1],delineationparams[2],distancefromurbancenter,delineationparams[3],longlatstat[(lon,lat)][0],delineationparams[4],delineationparams[5],delineationparams[6],delineationparams[7],delineationparams[8],delineationparams[9],binarythreshold=1292769530000,lon=lon,lat=lat) 1449.8 Function urban_sprawl_from_raster() is invoked as follows on GHSL R2023A SMOD raster GHS_SMOD_E2030_GLOBE_R2023A_54009_1000_V1_0.tif (from https://jeodpp.jrc.ec.europa.eu/ftp/jrc-opendata/GHSL/GHS_SMOD_GLOBE_R2023A/GHS_SMOD_E2030_GLOBE_R2023A_54009_1000/V1-0/GHS_SMOD_E2030_GLOBE_R2023A_54009_1000_V1_0.zip) for an approximate 110 km * 110 km bounding box of Chennai Metropolitan Area - http://bboxfinder.com/#12.439259,79.271851,13.568572,80.351257 - with all weights set to 0.01, for Verhulste-model projected 2023 population of 1,60,79,611 and GDP of $200 billion. Weights of the regression can be learnt by reverse-engineering the regression from cells of present metro area (5904 sqkm) which have all FUAij values as 1: r3data=urban_sprawl_from_raster(79.271851,12.439259,80.351257,13.568572,"testlogs/RemoteSensingGIS/GHS_SMOD_E2030_GLOBE_R2023A_54009_1000_V1_0.tif",dt="Degree of Urbanization R2023A",delineationparams=[0.01,0.01,0.01,0.01,0.01,0.01,16079611,0.01,200,0.01,(77.4479607,12.985801)]) 1449.9 Delineation boolean values found by above regression for 12204 GHSL raster cells (~12204 square kilometers) are captured in testlogs/GISUrbanSprawlAnalytics.log.31August2023 - Example: urban_sprawl_from_raster(): distance between ( 79.811554 , 13.0039155 ),( 79.271851 , 12.439259 ) = 61.33835299311702 longitude-latitude of the GHSL Raster cell: (79.271851, 12.439259) b0 (bias): 0.01 b1 (distance weight1): 0.01 b2 (distance weight2): 0.01 distij (distance of the GHSL cell i from urban center j) in kilometers: 61.33835299311702 g1 (size weight): 0.01 sizeij (size of the cell j) in population: 300 p1 (population weight1): 0.01 p2 (population weight2): 0.01 populationj (population of the urban area): 16079611 w (GDP weight): 0.01 gdp (GDP of the urban area or country) in billion dollars: 200 error term(epsilonij): 0.01 FUAij (can be converted to a boolean value for whether periurban cell i belongs to urban area j): 2585539059952.5776 GHSL lon-lat (79.271851, 12.439259) could belong to functional urban area for given criteria 1449.10 This update resolves JIRA issue https://krishnairesearch.atlassian.net/browse/NEUR-35
Srinivasan Kannan
More... about 3 years ago
Section 1438 updated
Srinivasan Kannan
More... about 3 years ago
Section 1285 updated
Srinivasan Kannan
More... about 3 years ago
Section 1448 updated
Srinivasan Kannan
More... about 3 years ago
Section 1285 updated
Srinivasan Kannan
More... about 3 years ago
Section 1285 updated
Srinivasan Kannan
More... about 3 years ago
Section 1285 updated
Srinivasan Kannan
More... about 3 years ago
Section 1285 updated
Srinivasan Kannan
More... about 3 years ago
+1448. (THEORY and FEATURE) GIS Urban Sprawl Analytics - Implementation of OECD-GHSL-JRC Functional Urban Area Lineation Regression Model for Global Human Settlement Layer - Chennai Metropolitan Area Example - 28 Augut 2023 - related to 823,1163,1164,1174,1175,1177,1179,1194,1211,1226,1234,1290,1291,1292,1295,1296,1297,1301,1302,1304,1312,1313,1320,1323,1326,1328,1342,1343,1349,1350,1351,1362,1363,1377,1386,1387,1395,1397,1398,1412,1414,1423 and all sections on GIS Urban Sprawl Analytics, Merit of Large Scale Visuals, Gridded Population Estimators (SEDAC GPW4 and GHSL convex hull estimators), DBSCAN and Neural Network Urban Extent Learners, Computational Geometry, Intrinsic Merit Rankings, Verlhulste-R Ricker Population Estimation, Community detection, Social Network Analytics, Space filling by Curves, Unequal Circle and Polynomial packing, Computational Geometry and Polygonal packing by Voronoi Tessellation, Four color theorem, Economic Merit, Computational Economics - GDP estimation from Urban sprawl demographics, electricity consumption, Poverty alleviation, Preferential Attachment, Automatic Delineation of Urban Sprawls, Cellular Automata-SLEUTH-Polya Urn-Dynamic Facegraph Urban Growth Models,Expander Graphs,Timeseries analysis,Graph Edit Distance and Earth mover distance, Face recognition,Graph Isomorphism (Exact, Approximate and Subgraph),Graph matching,Graph similarity,Image Search,Quadrilateral Mesh Deformation and Isomorphism,Maximum Common Subgraph Problem and Approximate Topological Matching ^M
Srinivasan Kannan
More... about 3 years ago
-------------------------------------------------------------------------------------------------------------------------------------------- 1448. (THEORY and FEATURE) GIS Urban Sprawl Analytics - Implementation of OECD-GHSL-JRC Functional Urban Area Lineation Regression Model for Global Human Settlement Layer - Chennai Metropolitan Area Example - 28 Augut 2023 - related to 823,1163,1164,1174,1175,1177,1179,1194,1211,1226,1234,1290,1291,1292,1295,1296,1297,1301,1302,1304,1312,1313,1320,1323,1326,1328,1342,1343,1349,1350,1351,1362,1363,1377,1386,1387,1395,1397,1398,1412,1414,1423 and all sections on GIS Urban Sprawl Analytics, Merit of Large Scale Visuals, Gridded Population Estimators (SEDAC GPW4 and GHSL convex hull estimators), DBSCAN and Neural Network Urban Extent Learners, Computational Geometry, Intrinsic Merit Rankings, Verlhulste-R Ricker Population Estimation, Community detection, Social Network Analytics, Space filling by Curves, Unequal Circle and Polynomial packing, Computational Geometry and Polygonal packing by Voronoi Tessellation, Four color theorem, Economic Merit, Computational Economics - GDP estimation from Urban sprawl demographics, electricity consumption, Poverty alleviation, Preferential Attachment, Automatic Delineation of Urban Sprawls, Cellular Automata-SLEUTH-Polya Urn-Dynamic Facegraph Urban Growth Models,Expander Graphs,Timeseries analysis,Graph Edit Distance and Earth mover distance, Face recognition,Graph Isomorphism (Exact, Approximate and Subgraph),Graph matching,Graph similarity,Image Search,Quadrilateral Mesh Deformation and Isomorphism,Maximum Common Subgraph Problem and Approximate Topological Matching -------------------------------------------------------------------------------------------------------------------------------------------- 1448.1 python-src/image_pattern_mining/ImageNet/GISUrbanSprawlAnalytics.py has been updated to define a new function that implements quadratic regression model (2) for functional urban area delineation (FUA) described in paper: Metropolitan areas in the world - Delineation and population trends - [OECD-GHSL-JRC] - https://www.sciencedirect.com/science/article/pii/S0094119020300139 - "..... πΉπ‘ˆπ΄π‘–π‘—πΆ = 𝛽0 + βˆ‘2 π‘˜=1 π›½π‘˜π‘‘π‘–π‘ π‘‘π‘˜ 𝑖𝑗𝐢 + 𝛾1𝑠𝑖𝑧𝑒𝑖𝑗𝐢 + βˆ‘2 π‘˜=1 πœ‘π‘˜π‘π‘œπ‘π‘˜ 𝑗𝐢 + ω𝐺𝐷𝑃𝐢 + βˆˆπ‘–π‘—πΆ (2)....." - referenced in Section 1423.8 1448.2 Function functional_urban_area_delineation_GHSL_regression(b0,b1,b2,distij,g1,sizeij,p1,p2,populationj,w,gdp,epsilonij) interprets the parameters of the above regression as below: FUAij = b0 + b1*distij + b2*distij*distij + g1*sizeij + p1*populationj + p2*populationj*populationj + w*gdp + epsilonij print("b0 (bias):",b0) print("b1 (distance weight1):",b1) print("b2 (distance weight2):",b2) print("distij (distance of the GHSL cell i from urban center j) in kilometers:",distij) print("g1 (size weight):",g1) print("sizeij (size of the cell j) in square kilometers:",sizeij) print("p1 (population weight1):",p1) print("p2 (population weight2):",p2) print("populationj (population of the urban area):",populationj) print("w (GDP weight):",w) print("gdp (GDP of the urban area or country) in billion dollars:",gdp) print("error term(epsilonij):",epsilonij) print("FUAij (can be converted to a boolean value for whether periurban cell i belongs to urban area j):",FUAij) 1448.3 Delineation function is invoked for parameters functional_urban_area_delineation_GHSL_regression(0.01,0.01,0.01,100,0.01,1,0.01,0.01,16079611,0.01,130,0.01) which correspond to GHSL R2023A study in 1423 of Chennai Metropolitan Area - for (*) a GHSL class 21 periurban cell of 1 square kilometer (growth rate of 25.377%) at a distance of 100 kilometers from urban center (*) metropolitan area population of 1,60,79,611 in 2023 (computed by Verhulste population growth model for annual growth rate of 2.39% based on 2022 Chennai metropolitan area population of 1.59 crores - Sections 1343 and 1350.14) and (*) Chennai Metropolitan Area GDP of $130 billion (GDP estimate sources: https://en.wikipedia.org/wiki/Chennai_metropolitan_area - $130 billion and https://en.wikipedia.org/wiki/Economy_of_Chennai - $200 billion). Presently the sizeij (cell size) values are presumptive and can be substituted from a raster GeoTIFF file. 1448.4 Earlier formula (2) is probably a first ever official regression for urban area delineation independent of machine learning-cellular automata models (1226.13),Polya Urn urban growth models and NightTimeLights radiance and it replaces urban area size by its population as proxy. 1448.5 Computation of FUA value for above data is at python-src/image_pattern_mining/ImageNet/testlogs/GISUrbanSprawlAnalytics.log.28August2023 and can be normalized to a boolean value by adjustment of regression weights (by predefined minimum and maximum values for FUAij): FUAij (can be converted to a boolean value for whether periurban cell i belongs to urban area j): 2585539060011.65 1448.6 Population is chosen as a predictor of urban area size as described in the model (2) of the paper - "...... we compare three different proxies for urban centre size: area, population and nightlight....." while model (1) (section 1423.8) relied on number of cars which are proportional to nightlights. Distance in above model is actually proportional to the travel time from cell i to urban center j by public or private transport (or ease of mobility) - though it is still debatable on which travel time (fastest or slowest mode) has to be chosen. 1448.7 Travel time to a periurban cell and Isochrone ego graphs in general could be estimated from Google Maps Live Traffic ViteJS implementation in NeuronRain Grafit (Drive Time Radius Isochrones are periodic snapshots of live traffic) - Section 1387 - https://github.com/shrinivaasanka/Grafit/tree/master/course_material/Programming/GUI/code/livetraffic . Emphasis on size of urban center as one of the FUA criteria is a rephrasal of preferential attachment (Big cities tend to get Bigger) - quoted excerpts from the paper - "...... Travel time from each cell to the urban centre is the most powerful predictor of the probability of belonging to a FUA, followed by the size of the urban centre ....."
Srinivasan Kannan
More... about 3 years ago
Section 1423 updated
Srinivasan Kannan
More... about 3 years ago
Section 1447 updated
Srinivasan Kannan
More... about 3 years ago
Section 1447 updated
Srinivasan Kannan
More... about 3 years ago
1447. (THEORY and FEATURE) GIS Weather Analytics and Urban Sprawl Analytics - Chennai Metropolitan Area - Periurban Road Network Elevation Model and Flood Isochrone - 22 August 2023 - related to 823,1163,1164,1174,1175,1177,1179,1194,1211,1226,1234,1290,1291,1292,1295,1296,1297,1301,1302,1304,1312,1313,1320,1323,1326,1328,1342,1343,1349,1350,1351,1362,1363,1377,1386,1395,1397,1398,1412,1414,1423,1424,1425,1430,1431 and all sections on GIS Urban Sprawl Analytics, Merit of Large Scale Visuals, Gridded Population Estimators (SEDAC GPW4 and GHSL convex hull estimators), DBSCAN and Neural Network Urban Extent Learners, Computational Geometry, Intrinsic Merit Rankings, Verlhulste-R Ricker Population Estimation, Community detection, Social Network Analytics, Space filling by Curves, Unequal Circle and Polynomial packing, Computational Geometry and Polygonal packing by Voronoi Tessellation, Four color theorem, Economic Merit, Computational Economics - GDP estimation from Urban sprawl demographics, electricity consumption, Poverty alleviation, Preferential Attachment, Automatic Delineation of Urban Sprawls, Cellular Automata-SLEUTH-Polya Urn-Dynamic Facegraph Urban Growth Models,Expander Graphs,Timeseries analysis,Graph Edit Distance and Earth mover distance, Face recognition,Graph Isomorphism (Exact, Approximate and Subgraph),Graph matching,Graph similarity,Image Search,Quadrilateral Mesh Deformation and Isomorphism,Maximum Common Subgraph Problem and Approximate Topological Matching, Weather Forecast, Climate Analytics, Flood Mitigation
Srinivasan Kannan
More... about 3 years ago
------------------------------------------------------------------------------------------------------------------------------------------- 1447. (THEORY and FEATURE) GIS Weather Analytics and Urban Sprawl Analytics - Chennai Metropolitan Area - Periurban Road Network Elevation Model and Flood Isochrone - 22 August 2023 - related to 823,1163,1164,1174,1175,1177,1179,1194,1211,1226,1234,1290,1291,1292,1295,1296,1297,1301,1302,1304,1312,1313,1320,1323,1326,1328,1342,1343,1349,1350,1351,1362,1363,1377,1386,1395,1397,1398,1412,1414,1423,1424,1425,1430,1431 and all sections on GIS Urban Sprawl Analytics, Merit of Large Scale Visuals, Gridded Population Estimators (SEDAC GPW4 and GHSL convex hull estimators), DBSCAN and Neural Network Urban Extent Learners, Computational Geometry, Intrinsic Merit Rankings, Verlhulste-R Ricker Population Estimation, Community detection, Social Network Analytics, Space filling by Curves, Unequal Circle and Polynomial packing, Computational Geometry and Polygonal packing by Voronoi Tessellation, Four color theorem, Economic Merit, Computational Economics - GDP estimation from Urban sprawl demographics, electricity consumption, Poverty alleviation, Preferential Attachment, Automatic Delineation of Urban Sprawls, Cellular Automata-SLEUTH-Polya Urn-Dynamic Facegraph Urban Growth Models,Expander Graphs,Timeseries analysis,Graph Edit Distance and Earth mover distance, Face recognition,Graph Isomorphism (Exact, Approximate and Subgraph),Graph matching,Graph similarity,Image Search,Quadrilateral Mesh Deformation and Isomorphism,Maximum Common Subgraph Problem and Approximate Topological Matching, Weather Forecast, Climate Analytics, Flood Mitigation ------------------------------------------------------------------------------------------------------------------------------------------- 1447.1 python-src/image_pattern_mining/ImageNet/GISUrbanSprawlAnalytics.py has been updated to redefine def urban_sprawl_road_network_OSM(cityname=None,latx=0,laty=0,longx=0,longy=0,address=None,radius=1000,defaultcapacity=1,travel_speed=30,trip_times=[10,15,20,25,30],maxelevationpoints=10) to include maxelevationpoints as argument (maximum number of longitude-latitude points in the elevation computed) 1447.2 For each of the longitude-latitude vertex in gdf_nodes JSON data of a road network graph, elevation of the vertex is lookedup in https://open-elevation.com/ (an opensource alternative to premium Google Elevation API) through REST API endpoint curl requests and an elevations array is populated within urban_sprawl_road_network_OSM() limited by maxelevationpoints. 1447.3 Elevations data of road and street network determines the flow time radius or radial distance and volume covered by floodwater in unit time in all directions (3D isochrones) - a prerequisite for augmenting Watershed algorithm with depth relief to simulate flooding. Gradient between high and low elevation vertices of road network decides the flow direction and velocity of fluid (Section 1395). 1447.4 Elevations for following Chennai metropolitan periurban areas have been looked-up through REST calls to https://open-elevation.com/ : urban_sprawl_road_network_OSM(address="Uthiramerur, Tamil Nadu, India",radius=6000) urban_sprawl_road_network_OSM(address="Madurantakam, Tamil Nadu, India",radius=6000) urban_sprawl_road_network_OSM(address="Cheyyur, Tamil Nadu, India",radius=6000) urban_sprawl_road_network_OSM(address="Sholinghur, Tamil Nadu, India",radius=6000) urban_sprawl_road_network_OSM(address="Ranipet, Tamil Nadu, India",radius=6000) urban_sprawl_road_network_OSM(address="Cheyyar, Tamil Nadu, India",radius=6000) urban_sprawl_road_network_OSM(address="Sricity, AP, India",radius=6000) 1447.5 Elevations (in meters above sea level) for road network vertices in above periurban areas are captured in testlogs/GISUrbanSprawlAnalytics.log.22August2023 - quoted excerpts below: Elevations of terrain: ['{"results":[{"latitude":12.692168,"longitude":79.833495,"elevation":68.0}]}'... Elevations of terrain: ['{"results":[{"latitude":12.510228,"longitude":79.889486,"elevation":30.0}]}'... Elevations of terrain: ['{"results":[{"latitude":12.423847,"longitude":79.830085,"elevation":46.0}]}'... Elevations of terrain: ['{"results":[{"latitude":13.164986,"longitude":79.43726,"elevation":158.0}]}'... Elevations of terrain: ['{"results":[{"latitude":12.904864,"longitude":79.324126,"elevation":171.0}]}'... Elevations of terrain: ['{"results":[{"latitude":12.645849,"longitude":79.535548,"elevation":93.0}]}'... Elevations of terrain: ['{"results":[{"latitude":13.512628,"longitude":79.955501,"elevation":47.0}]}'...
Srinivasan Kannan
More... about 3 years ago
1444. (THEORY and FEATURE) Software Analytics and Kernel Analytics - Python script to parse asfer_syscall_trace.log for Filesystem prefetch VIRGO kernel analytics config - related to 1191,619,770,906,911,771,1192,1443 and all sections on Program Analysis, AsFer-USBmd-VIRGO-KingCobra FTrace kernel callgraph analysis - Valgrind-Callgrind-KCacheGrind-Flamegraph userspace callgraph analysis, SATURN control flow analysis of VIRGO linux kernel, Software Analytics, Pandemic model of Botnet-Defense adversarial Game theoretic security, Cybercrime analytics, Anti-Malware,Anti-Ransomware,Anti-Theft and Anti-Plagiarism tools, Filesystem Prefetch, KML, LAKE - 14 August 2023
Srinivasan Kannan
More... about 3 years ago
-------------------------------------------------------------------------------------------------------------------------------------------- 1444. (THEORY and FEATURE) Software Analytics and Kernel Analytics - Python script to parse asfer_syscall_trace.log for Filesystem prefetch VIRGO kernel analytics config - related to 1191,619,770,906,911,771,1192,1443 and all sections on Program Analysis, AsFer-USBmd-VIRGO-KingCobra FTrace kernel callgraph analysis - Valgrind-Callgrind-KCacheGrind-Flamegraph userspace callgraph analysis, SATURN control flow analysis of VIRGO linux kernel, Software Analytics, Pandemic model of Botnet-Defense adversarial Game theoretic security, Cybercrime analytics, Anti-Malware,Anti-Ransomware,Anti-Theft and Anti-Plagiarism tools, Filesystem Prefetch, KML, LAKE - 14 August 2023 -------------------------------------------------------------------------------------------------------------------------------------------- 1444.1 New Python script asfer_syscall_trace.py has been added to repository for parsing strace log for read() systemcalls captured from a process in asfer_syscall_trace.log (process_id 2206 - pulseaudio - in this example), get a set of frequently read filedescriptors and append a keyvalue pair "frequently_read_filedescriptors=<set of filedescriptors>" to /etc/virgo_kernel_analytics.conf read and published within VIRGO linux kernel by virgo_kernel_analytics.ko driver module. 1444.2 Parse logs for asfer_syscall_trace.py can be found in testlogs/asfer_syscall_trace.py.log.14August2023 1444.3 Sample /etc/virgo_kernel_analytics.conf config file (list of comma separated key-value pairs) for virgo_kernel_analytics.ko driver written earlier has also been added to repository. 1444.4 Suspicious syscall sequences could as well point to malware infection (Section 1192.10) apart from filesystem prefetch. 1444.5 Present implementation of asfer_syscall_trace.sh and asfer_syscall_trace.py is limited to read() system calls and could be extended to any other system calls including write(), Network Socket I/O and VIRGO system call internal loadbalancers (Example: function char* get_host_from_cloud_Loadtrack_mempool() - https://github.com/shrinivaasanka/virgo64-linux-github-code/blob/master/linux-kernel-extensions/virgo_malloc/virgo_malloc.c - which is supposed to return least loaded host based on load stats might benefit by reading /etc/virgo_kernel_analytics.conf key-value pairs for load stats set from /proc/loadavg , top or uptime commands data - but not implemented at present). Script https://github.com/shrinivaasanka/virgo64-linux-github-code/blob/master/virgo-python-src/VirgoPRGLoadBalancer_probab_calc.py computes expected load per host for pseudorandom loadbalancer (randomly route requests to any node).
Srinivasan Kannan
More... about 3 years ago
Updated Requirements.txt
Srinivasan Kannan
More... about 3 years ago
1443. (THEORY and FEATURE) Software Analytics and Kernel Analytics - an STrace-LTrace shell script to machine-learn Filesystem prefetch patterns for readahead() system call - related to 1191,619,770,906,911,771,1192 and all sections on Program Analysis, AsFer-USBmd-VIRGO-KingCobra FTrace kernel callgraph analysis - Valgrind-Callgrind-KCacheGrind-Flamegraph userspace callgraph analysis, SATURN control flow analysis of VIRGO linux kernel, Software Analytics, Pandemic model of Botnet-Defense adversarial Game theoretic security, Cybercrime analytics, Anti-Malware,Anti-Ransomware,Anti-Theft and Anti-Plagiarism tools, KML, LAKE - 11 August 2023
Srinivasan Kannan
More... about 3 years ago
-------------------------------------------------------------------------------------------------------------------------------------------- 1443. (THEORY and FEATURE) Software Analytics and Kernel Analytics - an STrace-LTrace shell script to machine-learn Filesystem prefetch patterns for readahead() system call - related to 1191,619,770,906,911,771,1192 and all sections on Program Analysis, AsFer-USBmd-VIRGO-KingCobra FTrace kernel callgraph analysis - Valgrind-Callgrind-KCacheGrind-Flamegraph userspace callgraph analysis, SATURN control flow analysis of VIRGO linux kernel, Software Analytics, Pandemic model of Botnet-Defense adversarial Game theoretic security, Cybercrime analytics, Anti-Malware,Anti-Ransomware,Anti-Theft and Anti-Plagiarism tools, KML, LAKE - 11 August 2023 -------------------------------------------------------------------------------------------------------------------------------------------- 1443.1 Shell script "python-src/software_analytics/asfer_syscall_trace.sh <pid>" has been added to repository which invokes strace and ltrace system call trace utilities attaching to a process id specified in <pid> filtered by system call "read()". 1443.2 Output of strace and ltrace written to python-src/software_analytics/asfer_syscall_trace.log contains a chronological list of read() system calls on multiple file descriptors which could be parsed to get frequencies of reads for each file descriptor. 1443.3 Filesystem prefetch system call readahead() (https://man7.org/linux/man-pages/man2/readahead.2.html) prefetches a frequently read file descriptor and stores in pagecache. Filesystem prefetch is one of the most solved usecases in machine learning assisted linux kernel implementations - KML and LAKE. 1443.4 Filesystem Prefetch hints could be passed on to VIRGO linux kernel through NeuronRain Kernel Analytics which interfaces userspace machine learning and kernelspace cloud RPC through /etc/virgo_kernel_analytics.conf key-value pairs (exported as kernel wide variables by kernel_analytics VIRGO linux kernel module). 1443.5 For example, following is the file system read() pattern repeated by pulseaudio processid 2206 in asfer_syscall_trace.log for commandline "./asfer_syscall_trace.sh 2206" - file descriptors 22,3,48,20 are repetitively accessed which could be a kernel analytics key-value pair "frequently_read_filedescriptors=[22,3,48,20]" within /etc/virgo_kernel_analytics.conf read and published as a global variable within VIRGO linux kernel by driver virgo_kernel_analytics.ko. Based on this variable, VIRGO linux kernel could dynamically adjust to userspace filesystem read() activity: [pid 2206] read(22, "\1\0\0\0\0\0\0\0", 8) = 8 [pid 2206] read(3, "W", 10) = 1 [pid 2206] read(3, 0x7fff4f2919ce, 10) = -1 EAGAIN (Resource temporarily unavailable) [pid 2206] read(22, "\1\0\0\0\0\0\0\0", 8) = 8 [pid 2206] read(3, "W", 10) = 1 [pid 2206] read(3, 0x7fff4f2919ce, 10) = -1 EAGAIN (Resource temporarily unavailable) [pid 2206] read(48, "\1\0\0\0\0\0\0\0", 8) = 8 [pid 2207] read(20, "\1\0\0\0\0\0\0\0", 8) = 8 [pid 2206] read(3, 0x7fff4f2919ce, 10) = -1 EAGAIN (Resource temporarily unavailable)
Srinivasan Kannan
More... about 3 years ago
Updated BestPractices.txt
Srinivasan Kannan
More... about 3 years ago
Updated Requirements.txt
Srinivasan Kannan
More... about 3 years ago
+1442. (THEORY and FEATURE) People Analytics - Social Network Analysis - GitHub Code Search REST API statistics and Source lines of code for COCOMO Effort Estimation - related to 14,365,626,711,727,753,876,1127,1128,1129,1130,1204,1263,1269,1427 and all sections on People Analytics, Intrinsic Merit Versus Fame Equilibrium, BKS Conjecture, Stability and Noise Sensitivity of Interview LTF/PTF,Peres Theorem,Unique Identification,Talent Analytics,HR Analytics and Automated Recruitment,Question-Answering,Majority Voting,Condorcet Jury Theorem Variants,Margulis-Ruzzo Thresholds,Novelty Detection,Recommender Systems,AI Question-Answering, Partial Ordered Ranking, Fame-Merit equilibrium, Intrinsic Merit as Query Complexity,Software Analytics-Mythical Manmonth-CathedralAndBazaar - 9 August 2023^M
Srinivasan Kannan
More... about 3 years ago
------------------------------------------------------------------------------------------------------------------------------------------- 1442. (THEORY and FEATURE) People Analytics - Social Network Analysis - GitHub Code Search REST API statistics and Source lines of code for COCOMO Effort Estimation - related to 14,365,626,711,727,753,876,1127,1128,1129,1130,1204,1263,1269,1427 and all sections on People Analytics, Intrinsic Merit Versus Fame Equilibrium, BKS Conjecture, Stability and Noise Sensitivity of Interview LTF/PTF,Peres Theorem,Unique Identification,Talent Analytics,HR Analytics and Automated Recruitment,Question-Answering,Majority Voting,Condorcet Jury Theorem Variants,Margulis-Ruzzo Thresholds,Novelty Detection,Recommender Systems,AI Question-Answering, Partial Ordered Ranking, Fame-Merit equilibrium, Intrinsic Merit as Query Complexity,Software Analytics-Mythical Manmonth-CathedralAndBazaar - 9 August 2023 ------------------------------------------------------------------------------------------------------------------------------------------- 1442.1 Retrieving domain specific data for a unique person, especially opensource effort in the context of Information Technology domain, is quite necessary for quantifying absolute merit metrics for software effort estimation,talent analytics and automated recruitment which has been suggested as an alternative to conventional manual Q&A and interviews (Section 1204) often marred by theoretical limitations of sensitivity errors (Hardness of openended and multiple choice Question-Answering has been analyzed earlier and in Section 1394 in multiple theoretical formats - LTF,PTF,Switching Circuits,TQBFSAT,Halfplane intersections,Polynomial interpolation over reals,multiple choice CNFSAT,Query complexity model). 1442.2 As it is infeasible to quantify the effort of an engineer exactly in IT domain because of proprietary closedsource products, estimation is limited to technical data in public domain only - opensource profiles, code contributions being some of them. Usual search engine results do not display indepth data about internals of a FOSS repository and code commits by a user. Code Search REST API in GitHub recently unveiled fill in this void through programmatic search queries to GitHub repositories and user code commits data. 1442.3 python-src/SocialNetworkAnalysis_PeopleAnalytics.py has been updated to implement two new functions codesearch_statistics() and parse_sloc() for searching the GitHub repositories for statistics (code,repositories,commits,users) and source lines of code(SLOC) by CLOC utility matching a unique person identifier. These two metrics have been included in domain specific dictionary implemented in 1127. 1442.4 Function codesearch_statistics(self,opensourceid,personalaccesstoken) makes four REST API calls by cURL to GitHub codesearch to get the search commits,repositories,users and code data matching a username (opensourceid). Stdout of the cURL are captured by os.popen().read() into a JSON string. Temporary credential PersonalAccessToken is passed on as additional argument: codesearchjson=os.popen(" curl -L -H \"Accept: application/vnd.github+json\" -H \"Authorization: Bearer " + personalaccesstoken + "\" -H \"X-GitHub-Api-Version: 2022-11-28\" \"https://api.github.com/search/users?q=" + opensourceid + "\" ").read() codesearchjson+=os.popen(" curl -L -H \"Accept: application/vnd.github+json\" -H \"Authorization: Bearer " + personalaccesstoken + "\" -H \"X-GitHub-Api-Version: 2022-11-28\" \"https://api.github.com/search/code?q=" + opensourceid + "\" ").read() codesearchjson+=os.popen(" curl -L -H \"Accept: application/vnd.github+json\" -H \"Authorization: Bearer " + personalaccesstoken +"\" -H \"X-GitHub-Api-Version: 2022-11-28\" \"https://api.github.com/search/repositories?q=" + opensourceid + "\" ").read() codesearchjson+=os.popen(" curl -L -H \"Accept: application/vnd.github+json\" -H \"Authorization: Bearer " + personalaccesstoken + "\" -H \"X-GitHub-Api-Version: 2022-11-28\" \"https://api.github.com/search/commits?q=" + opensourceid + "\" ").read() 1442.5 There is no direct GitHub CodeSearch REST API call available to compute SLOC of a repository though size of repository data can be retrieved. For SLOC metric, CLOC utility has to be pre-installed and invoked on a local repository clone. CLOC produces a detailed per language SLOC report of a repository. Function parse_sloc() reads a text file SocialNetworkAnalysis_PeopleAnalytics.OpenSource_SLOC containing the stdout of CLOC command (https://github.com/AlDanial/cloc) and returns the total Source Lines of Code in a repository. Kilo lines of Code is the most important variable in COCOMO (https://en.wikipedia.org//wiki/COCOMO) effort estimation (in person-hours) which comes in 3 flavours - Basic, Intermediate and Detailed. Intermediate COCOMO is relevant for estimating effort involved in most medium size software products, given by formula - E = ai(KLoC)^bi(EAF). Example COCOMO estimation of NeuronRain AstroInfer repository - https://openhub.net/p/asfer-github-code/estimated_cost - COCOMO Coefficients used by OpenHub COCOMO are a=2.4 and b=1.05. 1442.6 EAF or Effort Adjustment Factor is a product of following attributes of a software product, each weighted between 0.9 to 1.4: ------------------------------- Product attributes: Required software reliability extent Size of application database Complexity of the product
Srinivasan Kannan
More... about 3 years ago
+1441. (JIRA,THEORY and FEATURE) Music Information Retrieval - AI Music Synthesis by Random walks on Music Weighted Automata Directed Graph - Concert synthesis by multiple instrument orchestra overlay on Piano music - through Music21 MIDI,MuseScore,FluidSynth,Pydub support - refineme nts to synthesized notes based on WFA state transition probability distribution - some more bug fixes - 8 August 2023 - related to 3,110,134,2 98,1223,1224,1257,1258,1261,1305,1306,1307,1308,1309,1310,1311,1315,1353,1432,1433,1434,1438,1439,1440 and all sections on Music search-Music discovery, Intrinsic Merit of Music, MIR(Music Information Retrieval), Music Similarity and Clustering, AI Music Synthesis from summation of d amped sinusoids, Additive Synthesis, fMRI connectomes - OpenNeuro - Psychology of music, Weighted automata, Chaos, Deep Learning, Dynamic Time Warping-Music Timeseries analysis, Music Evoked Autobiographical Memories, Waveshaping, ThoughtNet, Majority Voting (Enjoyable music) Vs Intr insic (Quality music) - Condorcet Jury Theorem variants,Computational Learning Theory, Boolean Function Composition, Longest Common Subsequenc e, Suffix Trees - Longest Repeated Substring, MultiFractal Detrended Fluctuation Analysis of Non-stationary timeseries, Ramsey Theory-Van Der Waerden Theorem for coloring of sequences, Pumping Lemma, Normalized Compression Distance-Kolmogorov complexity, Merit of Large Scale Visuals, Recurrent Neural Networks,Neo-Riemannian Theory of Music
Srinivasan Kannan
More... about 3 years ago
---------------------------------------------------------------------------------------------------------------------------------------------- 1441. (JIRA,THEORY and FEATURE) Music Information Retrieval - AI Music Synthesis by Random walks on Music Weighted Automata Directed Graph - Concert synthesis by multiple instrument orchestra overlay on Piano music - through Music21 MIDI,MuseScore,FluidSynth,Pydub support - refinements to synthesized notes based on WFA state transition probability distribution - some more bug fixes - 8 August 2023 - related to 3,110,134,298,1223,1224,1257,1258,1261,1305,1306,1307,1308,1309,1310,1311,1315,1353,1432,1433,1434,1438,1439,1440 and all sections on Music search-Music discovery, Intrinsic Merit of Music, MIR(Music Information Retrieval), Music Similarity and Clustering, AI Music Synthesis from summation of damped sinusoids, Additive Synthesis, fMRI connectomes - OpenNeuro - Psychology of music, Weighted automata, Chaos, Deep Learning, Dynamic Time Warping-Music Timeseries analysis, Music Evoked Autobiographical Memories, Waveshaping, ThoughtNet, Majority Voting (Enjoyable music) Vs Intrinsic (Quality music) - Condorcet Jury Theorem variants,Computational Learning Theory, Boolean Function Composition, Longest Common Subsequence, Suffix Trees - Longest Repeated Substring, MultiFractal Detrended Fluctuation Analysis of Non-stationary timeseries, Ramsey Theory-Van Der Waerden Theorem for coloring of sequences, Pumping Lemma, Normalized Compression Distance-Kolmogorov complexity, Merit of Large Scale Visuals, Recurrent Neural Networks,Neo-Riemannian Theory of Music ---------------------------------------------------------------------------------------------------------------------------------------------- 1441.1 python-src/music_pattern_mining/AudioToBitMatrix.py has been updated to change the definition of percussion_synthesis(virtual_piano_notes=None,percussions=["Saxophone","Guitar","Trumpet","Violin","SteelDrum","Flute"],tempo=2000) to include more instruments. 1441.2 Definition of notes_to_audio(automaton=False, function=None, deterministic=True, samplerate=44100, fractal=True, periodicity=500, weightedautomatadotfile=None,state2notedict=None,genre="WesternClassical",wfaweight_threshold=0.0,instruments=["Violin","Flute"]) has been changed to include instruments as arguments. 1441.3 Vertices for WFA random walk are obtained from edges(). Error conditions for no successors has been handled, in which case random walk resumes from a randomly chosen vertex. Error conditions when sum of transition weights are not equal to 1 are handled and sum of the array elements is normalized to 1 to get probability distribution. 1441.4 Percussion synthesis thread is instantiated with additional kwargs for instruments. 1441.5 An example execution of notes_to_audio(automaton=True,weightedautomatadotfile="testlogs/Bach_Flute_Sonata_EFlat.mp4_MusicWeightedAutomaton.dot",state2notedict={'0':'C','1':'D','2':'E','3':'F','4':'G','5':'A','6':'B','7':'Aβ™―','8':'Cβ™―','9':'Dβ™―','10':'Fβ™―','11':'Gβ™―'},genre="WesternClassical",instruments=["Guitar"]) permutes the state2notedict and for each permutation of the map, different music is synthesized. 1441.6 python-src/music_pattern_mining/MusicWeightedAutomaton.py has been updated to change music_weighted_automaton(name=None,sample=None,splarray=None,rows=10,columns=10,thresh=0.01) for non-zero threshold. GraphViz rendering has been disabled due to internal graphviz runtime memory exception. 1441.7 python-src/music_pattern_mining/testlogs/Bach_Flute_Sonata_EFlat.mp4_MusicWeightedAutomaton.dot has been regenerated by MusicWeightedAutomaton.py 1441.8 Following are the affected logs,DOT-GV and music waveforms for this update: modified: music21_percussion_for_piano.midi modified: music21_percussion_for_piano.wav modified: music21_piano_with_percussions.mp3 modified: testlogs/Bach_Flute_Sonata_EFlat.mp4_MusicWeightedAutomaton.dot modified: testlogs/Bach_Flute_Sonata_EFlat.mp4_MusicWeightedAutomaton.gv modified: virtual_piano_music.WesternClassical.wav testlogs/AudioToBitMatrix.log.8August2023 1441.9 This update resolves JIRA issue : https://krishnairesearch.atlassian.net/browse/NEUR-34
Srinivasan Kannan
More... about 3 years ago