P

python_crawler

Settings | Report Duplicate

1

I Use This!

Inactive

Commits : Listings

Analyzed about 17 hours ago. based on code collected 1 day ago.

Commit Message	Contributor	Files Modified	Lines Added	Lines Removed	Code Location	Date
Aug 25, 2025 — Aug 25, 2026 Showing page 1 of 1 Search / Filter on:
Added 'requestInterval' parameter to globalData, which adds a delay between each URL request.	David Arbuckle	More...				about 15 years ago
Now parsing control characters out of urls prior to saving to database, and converting to lower case.	David Arbuckle	More...				about 15 years ago
improved efficiency of updatecanonical() by an order of magnitude, replaced One Big Query with a bunch of little indexed selects.	David Arbuckle	More...				about 15 years ago
debug output was throwing errors when an error occured while loading the page. handled.	David Arbuckle	More...				about 15 years ago
added debug toggle and url-specific print statements to spice up the console output.	David Arbuckle	More...				about 15 years ago
Added indexes to tables. Runs about 10x faster (for now)	David Arbuckle	More...				about 15 years ago
updated description, issues/functoinality	David Arbuckle	More...				about 15 years ago
moved queue.join() call to the appropriate location.	David Arbuckle	More...				about 15 years ago
Added sleeps to initial URL retrieval	David Arbuckle	More...				about 15 years ago
Added threading module for URL requests.	David Arbuckle	More...				about 15 years ago
Updated README with better descriptionn of project.	David Arbuckle	More...				about 15 years ago
tests continuing...	David Arbuckle	More...				about 15 years ago
testing	David Arbuckle	More...				about 15 years ago
test	David Arbuckle	More...				about 15 years ago
first commit	David Arbuckle	More...				about 15 years ago