Projects tagged ‘web-crawler’

LinkChecker

Analyzed 5 months ago

Check websites and HTML documents for broken links. * recursive and multithreaded checking * output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats * HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support * restriction of link ... [More]

45.2K lines of code

10 current contributors

6 months since last commit

3 users on Open Hub

Activity Not Available

0 Reviews

I Use This

Mostly written in Python

Licenses: gpl

Spidr

Analyzed 3 days ago

Spidr is a versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

4.39K lines of code

1 current contributors

10 months since last commit

1 users on Open Hub

Very Low Activity

0 Reviews

I Use This

Mostly written in Ruby

Licenses: gpl, mit

Tags crawler rdoc robot rspec ruby rubygem spider spidr web web-crawler webcrawler web-spider 1 more...

Anemone

Analyzed about 1 hour ago

Anemone is a Ruby library that makes it quick and painless to write programs that spider a website. It provides a simple DSL for performing actions on every page of a site, skipping certain URLs, and calculating the shortest path to a given page on a site. The multi-threaded design makes Anemone ... [More]

2.15K lines of code

0 current contributors

over 12 years since last commit

1 users on Open Hub

Inactive

0 Reviews

I Use This