Seen
A lightweight crawling/spider framework for everyone
Screenshots
Hunter's comment
"Seen is a lightweight web crawling framework for everyone."
This is used to crawl your webpages very fast.
No need of Robot.txt
Pros
Supports JavaScript
Crawls webpages Fast
Simple to use
Open Source
Note:Python 3.5+ is Needed to install
Syntax for Installation
pip install seen
Link
https://github.com/HuberTRoy/Seen
Contributors
This is posted on Steemhunt - A place where you can dig products and earn STEEM.
View on Steemhunt.com