Open Hub
Open Hub

Tags : Browse Projects

Select a tag to browse associated projects and drill deeper into the tag cloud.

html-snapshots

Compare

  Analyzed about 4 hours ago

A selector-based html snapshot tool using PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input

6.34K lines of code

1 current contributors

1 day since last commit

1 users on Open Hub

Moderate Activity
0.0
 
0 Reviews
I Use This
Mostly written in JavaScript
Licenses: mit
Tags ajax crawler html javascript robots_txt seo singlepageapplication sitemap sitemapxml snapshots

robots.txt-go

Compare

  Analyzed about 14 hours ago

robots.txt exclusion protocol implementation for Go language (golang). Contains logically separate parser and checker. Parser may be used to check robots.txt file for correctness. Checker may be used for may-i-visit-this-url kind of queries against parsed robots.txt data.

1.11K lines of code

2 current contributors

almost 3 years since last commit

0 users on Open Hub

Inactive
0.0
 
0 Reviews
I Use This
Mostly written in Go
Licenses: mit
Tags api crawling go golang parser robots_txt

rbt_prs

Compare

  Analyzed about 13 hours ago

rbt_prs is a robots.txt parser written in PHP and intended for checking whether or not your bot can access a given URL.

287 lines of code

0 current contributors

almost 12 years since last commit

0 users on Open Hub

Inactive
0.0
 
0 Reviews
I Use This
Mostly written in PHP
Licenses: Public_Do...
Tags robots_txt