Open Hub
Black Duck Open Hub

Tags : Browse Projects

Select a tag to browse associated projects and drill deeper into the tag cloud.

html-snapshots

Compare

  Analyzed about 19 hours ago

A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input

6.31K lines of code

1 current contributors

1 day since last commit

1 users on Open Hub

High Activity
0.0
 
0 Reviews
I Use This
Mostly written in JavaScript
Licenses: mit
Tags ajax crawler html javascript robots_txt seo singlepageapplication sitemap sitemapxml snapshots

robots.txt-go

Compare

  Analyzed 1 day ago

robots.txt exclusion protocol implementation for Go language (golang). Contains logically separate parser and checker. Parser may be used to check robots.txt file for correctness. Checker may be used for may-i-visit-this-url kind of queries against parsed robots.txt data.

1.11K lines of code

2 current contributors

over 1 year since last commit

0 users on Open Hub

Very Low Activity
0.0
 
0 Reviews
I Use This
Mostly written in Go
Licenses: mit
Tags api crawling go golang parser robots_txt

rbt_prs

Compare

  Analyzed about 15 hours ago

rbt_prs is a robots.txt parser written in PHP and intended for checking whether or not your bot can access a given URL.

287 lines of code

0 current contributors

over 10 years since last commit

0 users on Open Hub

Inactive
0.0
 
0 Reviews
I Use This
Mostly written in PHP
Licenses: Public_Do...
Tags robots_txt