Urlgrab - A Golang Utility To Spider Through A Website Searching For Additional Links

A golang utility to spider through a website searching for additional links with support for JavaScript rendering.

Install

go get -u github.com/iamstoxe/urlgrab

Features

• Customizable Parallelism
• Ability to Render JavaScript ^{(including Single Page Applications such as Angular and React)}

Usage

Usage of urlgrab:
  -cache-dir string
        Specify a directory to utilize caching. Works between sessions as well.
  -debug
        Extremely verbose debugging output. Useful mainly for development.
  -delay int
        Milliseconds to randomly apply as a delay between requests. (default 2000)
  -depth int
        The maximum limit on the recursion depth of visited URLs.  (default 2)
  -headless
        If true the browser will be displayed while crawling.
        Note: Requires render-js flag
        Note: Usage to show browser: --headless=false (default true)
  -ignore-query
        Strip the query portion of the URL before determining if we've visited it yet.
  -ignore-ssl
        Scrape pages with invalid SSL certificates
  -js-timeout int
        The amount of seconds before a request to render    javascript should timeout. (default 10)
  -json string
        The filename where we should store the output JSON file.
  -max-body int
        The limit of the retrieved response body in kilobytes.
        0 means unlimited.
        Supply this value in kilobytes. (i.e. 10 * 1024kb = 10MB) (default 10240)
  -no-head
        Do not send HEAD requests prior to GET for pre-validation.
  -output-all string
        The directory where we should store the output files.
  -proxy string
        The SOCKS5 proxy to utilize (format: socks5://127.0.0.1:8080 OR http://127.0.0.1:8080).
        Supply multiple proxies by separating them with a comma.
  -random-agent
        Utilize a random user agent string.
  -render-js
        Determines if we utilize a headless chrome instance to render javascript.
  -root-domain    string
        The root domain we should match links against.
        If not specified it will default to the host of --url.
        Example: --root-domain google.com
  -threads int
        The number of threads to utilize. (default 5)
  -timeout int
        The amount of seconds before a request should timeout. (default 10)
  -url string
        The URL where we should start crawling.
  -urls string
        A file path that contains a list of urls to supply as starting urls.
        Requires --root-domain flag.
  -user-agent string
        A user agent such as (Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0).
  -verbose
        Verbose output

Author
Devin Stokes

• Twitter: @DevinStokes
• Github: @IAmStoxe

Download Urlgrab

via KitPloit

This article is the property of Tenochtitlan Offensive Security. Verlo Completo --> https://tenochtitlan-sec.blogspot.com

More articles

1. Pentest Tools For Windows
2. Hacking Tools For Kali Linux
3. Nsa Hack Tools
4. How To Hack
5. Best Hacking Tools 2020
6. Hack Tools For Mac
7. Pentest Tools Find Subdomains
8. Hacker Tools Online
9. Hacker Tools Github
10. Black Hat Hacker Tools
11. Tools Used For Hacking
12. Hacking Tools For Pc
13. Hacker Tools Github
14. Hack Tool Apk
15. Hacking Tools Mac
16. Pentest Tools Tcp Port Scanner
17. Black Hat Hacker Tools
18. How To Make Hacking Tools
19. Hacker Tools List
20. Bluetooth Hacking Tools Kali
21. World No 1 Hacker Software
22. Pentest Tools Tcp Port Scanner
23. Hacking Tools Name
24. Hacking Tools For Kali Linux
25. Pentest Tools Github
26. Pentest Tools Framework
27. Hacker Tools Software
28. Hacker Tools Free
29. How To Hack
30. Termux Hacking Tools 2019
31. Best Hacking Tools 2020
32. Hack Tools For Games
33. Hack Tools Pc
34. Beginner Hacker Tools
35. Pentest Tools Review
36. Pentest Tools Download
37. How To Make Hacking Tools
38. Blackhat Hacker Tools
39. Hacking Tools Windows
40. Hacking Tools For Games
41. Hacking Tools Free Download
42. Hacker Tools Software
43. Hacker Tools For Pc
44. Nsa Hack Tools
45. Tools For Hacker
46. New Hacker Tools
47. Free Pentest Tools For Windows
48. Pentest Tools Free
49. Hack Tools For Pc
50. Hack Tools
51. Android Hack Tools Github
52. Hacker Tools For Windows
53. Hacking Tools For Games
54. Hacking Tools Github
55. Hack Tools For Windows
56. Termux Hacking Tools 2019
57. Pentest Tools
58. Beginner Hacker Tools
59. New Hacker Tools
60. Hacker Tools 2020
61. Hacker Tools Apk Download
62. What Is Hacking Tools
63. Hack Tools
64. Pentest Box Tools Download
65. Install Pentest Tools Ubuntu
66. Hacking Tools Download
67. Physical Pentest Tools
68. Hacker Tools 2019
69. Beginner Hacker Tools
70. Pentest Tools Windows
71. Hacking Tools For Kali Linux
72. Hack Tools Pc
73. Pentest Tools For Android
74. Hackrf Tools
75. Pentest Tools Subdomain
76. Best Hacking Tools 2019
77. Pentest Tools Tcp Port Scanner
78. Nsa Hack Tools
79. Hack Tools Pc
80. Hacking Tools Hardware
81. Android Hack Tools Github
82. Pentest Tools
83. Hacker
84. Pentest Automation Tools
85. Pentest Tools Apk
86. Pentest Tools Port Scanner
87. Pentest Tools Website Vulnerability
88. Hacker Hardware Tools
89. Ethical Hacker Tools
90. Hak5 Tools
91. Black Hat Hacker Tools
92. Pentest Tools Apk
93. Pentest Tools Windows
94. Hacking Apps
95. Hacker Tools For Windows
96. Hacker Tools Apk Download
97. Hack Tools
98. Pentest Tools Framework
99. Hacker Tools Mac
100. Hacker Tools 2020
101. Hacker Tool Kit
102. Pentest Tools Linux
103. Pentest Reporting Tools
104. Easy Hack Tools
105. Nsa Hacker Tools
106. Hack Tools For Ubuntu
107. Hack Tools Mac
108. Hacking App
109. Pentest Tools Android
110. Tools For Hacker
111. Hacker Tools Free Download
112. Hack Tools
113. Top Pentest Tools
114. Black Hat Hacker Tools
115. Pentest Tools Windows
116. Hacking Tools Hardware