N

node-simplecrawler

Settings | Report Duplicate

0

I Use This!

Inactive

Commits : Listings

Analyzed about 14 hours ago. based on code collected 1 day ago.

Commit Message	Contributor	Files Modified	Lines Added	Lines Removed	Code Location	Date
Aug 28, 2025 — Aug 28, 2026 Showing page 12 of 23 Search / Filter on:
Merge pull request #215 from fredrikekelund/cookieErrorAsEvent	Fredrik Ekelund	More...				over 10 years ago
Catch errors from adding cookies and emit event	Fredrik Ekelund	More...				over 10 years ago
Fetch robots.txt in Crawler.crawl and for each host	Fredrik Ekelund	More...				over 10 years ago
Set queueItem.fetched and decrement _openRequests on fetchdataerror	Fredrik Ekelund	More...				over 10 years ago
Removed some TODO comments from testcrawl.js	Fredrik Ekelund	More...				over 10 years ago
Emit a fetchdataerror when Content-Length is too big	Fredrik Ekelund	More...				over 10 years ago
Updated and sped up tests	Fredrik Ekelund	More...				over 10 years ago
Return actual Errors from getRobotsTxt and emit "robotstxterror" event	Fredrik Ekelund	More...				over 10 years ago
Make getRequestOptions accept a single string as well	Fredrik Ekelund	More...				over 10 years ago
Emit "fetchdisallowed" event when robots.txt disallows	Fredrik Ekelund	More...				over 10 years ago
Make _robotsTxtFetchCondition a top level method instead	Fredrik Ekelund	More...				over 10 years ago
Bumped version to 0.6.1	Fredrik Ekelund	More...				over 10 years ago
Bumped version to 0.6.1	Fredrik Ekelund	More...				over 10 years ago
Merge pull request #209 from fredrikekelund/newResponseCloseBehavior	Fredrik Ekelund	More...				over 10 years ago
Clean up receiveData	Fredrik Ekelund	More...				over 10 years ago
Clean up receiveData	Fredrik Ekelund	More...				over 10 years ago
De-bumped mocha timeout	Fredrik Ekelund	More...				over 10 years ago
Removed manual robots.txt parsing example	Fredrik Ekelund	More...				over 10 years ago
Cover edge case in robotsTxtFetchCondition	Fredrik Ekelund	More...				over 10 years ago
Added details to README about respectRobotsTxt setting	Fredrik Ekelund	More...				over 10 years ago
Added tests for obeying of robots.txt rules	Fredrik Ekelund	More...				over 10 years ago
Initial support for robots.txt parsing	Fredrik Ekelund	More...				over 10 years ago
Bumped version to 0.6.0	Fredrik Ekelund	More...				over 10 years ago
Always close HTTP response streams on "end"	Fredrik Ekelund	More...				over 10 years ago
Always close HTTP response streams on "end"	Fredrik Ekelund	More...				over 10 years ago
Merge pull request #203 from bfwg/master	Fredrik Ekelund	More...				over 10 years ago
PR for #197	Fan Jin	More...				over 10 years ago
Added test to ensure fragment identifiers are stripped	Fredrik Ekelund	More...				over 10 years ago
Require whitespace before 'href' or 'src' in discoverRegex	Fredrik Ekelund	More...				over 10 years ago
Always pass the decoded response when decodeResponses is true	Fredrik Ekelund	More...				over 10 years ago

←
1
2
…
8
9
10
11
12
13
14
15
16
…
22
23
→