English (United States)
See more
Web pages are crawled by being loaded into browser using multiple tabs parallelly
Latest Version
Version
0.0.3
Update
March 8, 2019
Size
121KiB
Developer
Category
Users
46 
Rating
1 ★
Manifest Version
2
Web Store Status
Extension ID
Languages Support
Permissions
Tabs
ActiveTab
WebNavigation
See more
ActiveTab
WebNavigation
Wild Spider Extension
WATCH OUT: more tabs you use, more computer resources (CPU, memory) will be used, and each page costs a bit disk to save the content (in IndexedDb, accessible from extensions -> Inspect views: background page).
The \"spider\" works in this way:
1) The current url is used as the starting point, and it's loaded again in a new tab.
2) After this page is loaded, fetch all the links on the page.
3) Get all the links on the page, including relative urls.
4) Open the extracted link parallelly in all the tabs used (by default 3, set in eventPage).
5) repeat 2-4
All source code at: https://github.com/nobodxbodon/ChromeCrawlerWildSpider
Read more
The \"spider\" works in this way:
1) The current url is used as the starting point, and it's loaded again in a new tab.
2) After this page is loaded, fetch all the links on the page.
3) Get all the links on the page, including relative urls.
4) Open the extracted link parallelly in all the tabs used (by default 3, set in eventPage).
5) repeat 2-4
All source code at: https://github.com/nobodxbodon/ChromeCrawlerWildSpider
Source Manifest.json
{"update_url": "https://clients2.google.com/service/update2/crx", "name": "Wild Spider", "short_name": "demo web crawler that's still in experimenting", "description": "web pages are crawled by being loaded into browser using multiple tabs parallelly", "version": "0.0.3", "browser_action": { "default_icon": "icon.png" }, "permissions": [ "tabs", "activeTab", "webNavigation" ], "background": { "scripts": ["Dexie.js", "eventPage.js"], "persistent": false }, "content_scripts": [ { "matches": ["*://*/*"], "js": ["htmlparser2.js", "content.js"] } ], "manifest_version": 2}
Read more
Similar Extensions
Fun Crawler 46 3 ★ 14.91KiB
JIRA Crawler 46 1 ★ 95.03KiB
Page Crawler 46 5 ★ 26.33KiB
Creeper 46 4 ★ 340KiB
Semantic Spider 46 3 ★ 596KiB
Link Collector 46 2 ★ 24.44KiB
Recursive 46 12 ★ 1.04MiB
Input hidden Monitor 46 0 ★ 36.05KiB
LinkAnalysis 46 18 ★ 57.68KiB
beABot 46 3 ★ 51.2KiB
Scraper Crawler V3 46 28 ★ 174KiB
Crawler 46 12 ★ 235KiB
Spider Free - The simplest web scraper 46 27 ★ 17.4KiB
DirectFN web scraping tools 46 3 ★ 81.5KiB
Link Crawler 46 0 ★ 27.53KiB
Spider Search 46 1 ★ 328KiB
get-set, Fetch! web scraper 46 2 ★ 760KiB
Grabbly 46 12 ★ 6.92MiB
Grab 'n Tag 46 4 ★ 41.52KiB
Webmapper 46 13 ★ 200KiB
Bishop Vulnerability Scanner 46 12 ★ 198KiB
Cyber Web Tools 46 6 ★ 340KiB
Lazy Linker 46 7 ★ 54.37KiB
SearchMage 46 3 ★ 153KiB
User reviews
Matt C (ID: 0000000b621d3a6c) Voted 1 ★ - Updated 03/19/2018
Latest Updates
Top Rated
Hola VPN - Your Website Unblocker
Adblock for Youtube™
AdBlock — block ads across the web
Video Downloader Professional
Adblock Plus - free ad blocker
Stands AdBlocker
Evernote Web Clipper
Argent X - Starknet Wallet
Touch VPN - Secure and unlimited VPN proxy
Pop up blocker for Chrome™ - Poper Blocker