Select a tag to browse associated projects and drill deeper into the tag cloud.
Heritrix: Internet Archive Web Crawler
H
Analyzed 1 day ago
The archive-crawler project is building a flexible, extensible, robust, and scalable web crawler capable of fetching, archiving, and analyzing the full diversity and breadth of internet-accesible content.
Licenses:
lgpl
crawler4j
C
Analyzed about 20 hours ago
Crawler4j is an open source Java Crawler which provides a simple interface for crawling the web. Using it, you can setup a multi-threaded web crawler in 5 minutes!
Sample UsageFirst, you need to create a crawler class that extends WebCrawler. This class decides which URLs should be crawled and
Licenses:
apache_2
LinkChecker
Analyzed about 2 years ago
Check websites and HTML documents for broken links.
* recursive and multithreaded checking
* output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats
* HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support
* restriction of link
Smart Cache Loader
Analyzed 1 day ago
Smart Cache Loader is most configurable web batch downloader in world! If you have a very specific needs to grab some portions of web site -- this is right tool for you! This program can be also used as web crawler if you need to crawl defined parts of www site(s).
Spidr
Analyzed 1 day ago
Spidr is a versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Licenses:
gpl, mit
Anemone
Analyzed about 18 hours ago
Anemone is a Ruby library that makes it quick and painless to write programs that spider a website. It provides a simple DSL for performing actions on every page of a site, skipping certain URLs, and calculating the shortest path to a given page on a site.
The multi-threaded design makes Anemone
mediacloud_backend
M
Analyzed about 23 hours ago
MediaCloud backend repository
Licenses:
No declared licenses
Tachyon_project
T
Analyzed 1 day ago
Tachyon is a fast web application security reconnaissance tool.
It is specifically meant to crawl a web application and look for left over or non-indexed files with the addition of reporting pages or scripts leaking internal data.
Licenses:
GNU-GPLv2
heritrix-crawl-filter
H
Analyzed 1 day ago
A candidate chain processor for applying regular expression filters on URIs coming from defined seeds
Licenses:
No declared licenses
Zeitcrawler
Z
Analyzed about 16 hours ago
A specialized crawler for the German newspaper 'Die Zeit'.
Starting from the front page or from a given list of links, the crawler retrieves newspaper articles and gathers new links to explore as it goes, stripping the text of each article out of the HTML formatting and saving it into a raw text