Select a tag to browse associated projects and drill deeper into the tag cloud.
htDig2
No analysis available
The ht://Dig system is a complete WWW indexing and searching system for a domain or intranet. This system is not meant to replace the need for internet-wide search systems like Lycos, Infoseek, Google, and AltaVista. Instead, it is meant to cover the search needs for a single company, campus, or
Mostly written in language not available
Licenses:
gpl
Grub
No analysis available
Grub Next Generation is distributed web crawling system (clients/servers) which helps to build and maintain free (as in freedom) index of the Web. At this moment we have very simple search engine (as a proof of concept) too.
Mostly written in language not available
Licenses:
BSD-3-Clause, gpl3_or_l...
Apache ManifoldCF
Claimed by Apache Software Foundation Analyzed about 21 hours ago
ManifoldCF is an effort to provide an open source framework for connecting source content repositories like Microsoft Sharepoint and EMC Documentum, to target repositories or indexes, such as Apache Solr. ManifoldCF also defines a security model for target repositories that permits them to enforce source-repository security policies.
Licenses:
apache_2
crawler4j
C
Analyzed 1 day ago
Crawler4j is an open source Java Crawler which provides a simple interface for crawling the web. Using it, you can setup a multi-threaded web crawler in 5 minutes!
Sample UsageFirst, you need to create a crawler class that extends WebCrawler. This class decides which URLs should be crawled and
Licenses:
apache_2
OpenSearchServer
No analysis available
OpenSearchServer is a powerful, enterprise-class, search engine program. With the web user interface, the crawlers (web, file, database, ...) and its REST API you will be able to integrate quickly and easily advanced full-text search capabilities in your application. OpenSearchServer runs on Windows
Mostly written in language not available
Licenses:
gpl3
LinkChecker
Analyzed 11 months ago
Check websites and HTML documents for broken links.
* recursive and multithreaded checking
* output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats
* HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support
* restriction of link
Ronin
No analysis available
Ronin is a Ruby platform for exploit development and security research. Ronin allows for the rapid development and distribution of code, exploits or payloads over many common Source-Code-Management (SCM) systems.
Mostly written in language not available
Licenses:
gpl3, lgpl3
Monkey-Spider
M
Analyzed about 23 hours ago
The Monkey-Spider is a crawler based low-interaction Honeyclient Project. It is not only restricted to this use but it is developed as such. The Monkey-Spider crawles Web sites to expose their threats to Web clients.
Licenses:
gpl3_or_l...
Smart and Simple Web Crawler
S
No analysis available
Simple framework to implement crawling technolgy in own programs and libraries.
Mostly written in language not available
Licenses:
No declared licenses