From 5513ab003744ddfd82b001b0a54d5bfc5080ba90 Mon Sep 17 00:00:00 2001 From: AntoTheSol Date: Wed, 22 Oct 2025 23:39:03 -0400 Subject: [PATCH 1/3] Added url export to webscrape.py, created export function export_links() and modified scrape() function. --- scraped_data/github_com_20251022_231702.txt | 124 ++++++++++++++++++++ utils/web_scrape.py | 101 ++++++++++++++++ 2 files changed, 225 insertions(+) create mode 100644 scraped_data/github_com_20251022_231702.txt diff --git a/scraped_data/github_com_20251022_231702.txt b/scraped_data/github_com_20251022_231702.txt new file mode 100644 index 0000000..d6c2725 --- /dev/null +++ b/scraped_data/github_com_20251022_231702.txt @@ -0,0 +1,124 @@ +Scraped links from: https://github.com/AntoTheSol/H4X-Tools +Date: 2025-10-22 23:17:02 +Total links: 119 +-------------------------------------------------------------------------------- + +https://GitHub.com/vil/H4X-Tools/commit/ +https://GitHub.com/vil/H4X-Tools/network/ +https://GitHub.com/vil/H4X-Tools/stargazers/ +https://docs.github.com +https://docs.github.com/ +https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax +https://docs.github.com/site-policy/github-terms/github-terms-of-service +https://docs.github.com/site-policy/privacy-policies/github-privacy-statement +https://git-scm.com/downloads +https://github.blog +https://github.com +https://github.com/ +https://github.com/AntoTheSol +https://github.com/AntoTheSol/H4X-Tools +https://github.com/AntoTheSol/H4X-Tools#GPL-3.0-1-ov-file +https://github.com/AntoTheSol/H4X-Tools#contributing-ov-file +https://github.com/AntoTheSol/H4X-Tools#current-tools +https://github.com/AntoTheSol/H4X-Tools#h4x-tools +https://github.com/AntoTheSol/H4X-Tools#license +https://github.com/AntoTheSol/H4X-Tools#linux +https://github.com/AntoTheSol/H4X-Tools#looking-for-contributors +https://github.com/AntoTheSol/H4X-Tools#readme-ov-file +https://github.com/AntoTheSol/H4X-Tools#setup +https://github.com/AntoTheSol/H4X-Tools#start-of-content +https://github.com/AntoTheSol/H4X-Tools#windows +https://github.com/AntoTheSol/H4X-Tools/actions +https://github.com/AntoTheSol/H4X-Tools/activity +https://github.com/AntoTheSol/H4X-Tools/blob/master/.gitignore +https://github.com/AntoTheSol/H4X-Tools/blob/master/CONTRIBUTING.md +https://github.com/AntoTheSol/H4X-Tools/blob/master/LICENSE +https://github.com/AntoTheSol/H4X-Tools/blob/master/README.md +https://github.com/AntoTheSol/H4X-Tools/blob/master/h4xtools.py +https://github.com/AntoTheSol/H4X-Tools/blob/master/requirements.txt +https://github.com/AntoTheSol/H4X-Tools/blob/master/setup.bat +https://github.com/AntoTheSol/H4X-Tools/blob/master/setup.sh +https://github.com/AntoTheSol/H4X-Tools/blob/master/update.bat +https://github.com/AntoTheSol/H4X-Tools/blob/master/update.sh +https://github.com/AntoTheSol/H4X-Tools/branches +https://github.com/AntoTheSol/H4X-Tools/commits/master/ +https://github.com/AntoTheSol/H4X-Tools/forks +https://github.com/AntoTheSol/H4X-Tools/projects +https://github.com/AntoTheSol/H4X-Tools/pulls +https://github.com/AntoTheSol/H4X-Tools/pulse +https://github.com/AntoTheSol/H4X-Tools/releases +https://github.com/AntoTheSol/H4X-Tools/security +https://github.com/AntoTheSol/H4X-Tools/stargazers +https://github.com/AntoTheSol/H4X-Tools/tags +https://github.com/AntoTheSol/H4X-Tools/tree/master/.github +https://github.com/AntoTheSol/H4X-Tools/tree/master/helper +https://github.com/AntoTheSol/H4X-Tools/tree/master/img +https://github.com/AntoTheSol/H4X-Tools/tree/master/resources +https://github.com/AntoTheSol/H4X-Tools/tree/master/utils +https://github.com/AntoTheSol/H4X-Tools/watchers +https://github.com/collections +https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FAntoTheSol%2FH4X-Tools&report=AntoTheSol+%28user%29 +https://github.com/customer-stories +https://github.com/enterprise +https://github.com/enterprise/startups +https://github.com/features +https://github.com/features/actions +https://github.com/features/code-review +https://github.com/features/code-search +https://github.com/features/codespaces +https://github.com/features/copilot +https://github.com/features/copilot/copilot-business +https://github.com/features/discussions +https://github.com/features/issues +https://github.com/features/models +https://github.com/features/spark +https://github.com/login?return_to=%2FAntoTheSol%2FH4X-Tools +https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FAntoTheSol%2FH4X-Tools +https://github.com/marketplace +https://github.com/mcp +https://github.com/megadose/holehe +https://github.com/partners +https://github.com/premium-support +https://github.com/pricing +https://github.com/readme +https://github.com/resources/articles +https://github.com/resources/articles?topic=ai +https://github.com/resources/articles?topic=devops +https://github.com/resources/articles?topic=security +https://github.com/resources/articles?topic=software-development +https://github.com/resources/events +https://github.com/resources/whitepapers +https://github.com/security +https://github.com/security/advanced-security +https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=AntoTheSol%2FH4X-Tools +https://github.com/solutions +https://github.com/solutions/executive-insights +https://github.com/solutions/industry +https://github.com/solutions/industry/financial-services +https://github.com/solutions/industry/government +https://github.com/solutions/industry/healthcare +https://github.com/solutions/industry/manufacturing +https://github.com/solutions/industry/nonprofits +https://github.com/solutions/use-case +https://github.com/solutions/use-case/app-modernization +https://github.com/solutions/use-case/ci-cd +https://github.com/solutions/use-case/devops +https://github.com/solutions/use-case/devsecops +https://github.com/sponsors +https://github.com/team +https://github.com/topics +https://github.com/trending +https://github.com/users/AntoTheSol/packages?repo_name=H4X-Tools +https://github.com/vil/H4X-Tools +https://github.com/vil/H4X-Tools/blob/master/img/gui-v0.3.2.png +https://github.com/vil/H4X-Tools/issues +https://github.com/vil/H4X-Tools/wiki +https://github.com/why-github +https://github.community/ +https://pypi.org/project/Faker/ +https://resources.github.com/learn/pathways +https://skills.github.com +https://support.github.com?tags=dotcom-footer +https://www.githubstatus.com/ +https://www.gnu.org/licenses/gpl-3.0.txt +https://www.python.org/downloads/ diff --git a/utils/web_scrape.py b/utils/web_scrape.py index 6c409e0..c952ecb 100644 --- a/utils/web_scrape.py +++ b/utils/web_scrape.py @@ -17,8 +17,12 @@ import aiohttp import asyncio +import json +import csv from typing import Any from urllib.parse import urljoin, urlparse +from datetime import datetime +from pathlib import Path from bs4 import BeautifulSoup from colorama import Style @@ -29,6 +33,78 @@ scraped_links = set() +def export_links(links: set, base_url: str, format_type: str = "txt") -> None: + """ + Exports scraped links to a file in the specified format. + + :param links: Set of scraped links + :param base_url: Base URL that was scraped + :param format_type: Export format ('txt', 'csv', or 'json') + """ + if not links: + printer.warning("No links to export!") + return + + # Create output directory if it doesn't exist + output_dir = Path("scraped_data") + output_dir.mkdir(exist_ok=True) + + # Generate filename with timestamp + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + domain = urlparse(base_url).netloc.replace(".", "_") + filename = f"{domain}_{timestamp}" + + try: + if format_type.lower() == "txt": + filepath = output_dir / f"{filename}.txt" + with open(filepath, "w", encoding="utf-8") as f: + f.write(f"Scraped links from: {base_url}\n") + f.write(f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n") + f.write(f"Total links: {len(links)}\n") + f.write("-" * 80 + "\n\n") + for link in sorted(links): + f.write(f"{link}\n") + printer.success(f"Links exported to {Style.BRIGHT}{filepath}{Style.RESET_ALL}") + + elif format_type.lower() == "csv": + filepath = output_dir / f"{filename}.csv" + with open(filepath, "w", newline="", encoding="utf-8") as f: + writer = csv.writer(f) + writer.writerow(["URL", "Domain", "Path"]) + for link in sorted(links): + parsed = urlparse(link) + writer.writerow([link, parsed.netloc, parsed.path]) + printer.success(f"Links exported to {Style.BRIGHT}{filepath}{Style.RESET_ALL}") + + elif format_type.lower() == "json": + filepath = output_dir / f"{filename}.json" + link_data = { + "metadata": { + "source_url": base_url, + "scraped_date": datetime.now().isoformat(), + "total_links": len(links) + }, + "links": [ + { + "url": link, + "domain": urlparse(link).netloc, + "path": urlparse(link).path, + "scheme": urlparse(link).scheme + } + for link in sorted(links) + ] + } + with open(filepath, "w", encoding="utf-8") as f: + json.dump(link_data, f, indent=2, ensure_ascii=False) + printer.success(f"Links exported to {Style.BRIGHT}{filepath}{Style.RESET_ALL}") + + else: + printer.error(f"Invalid format: {format_type}. Use 'txt', 'csv', or 'json'.") + + except Exception as e: + printer.error(f"Error exporting links: {e}") + + @timer.timer(require_input=True) def scrape(url: str) -> None: """ @@ -52,6 +128,31 @@ def scrape(url: str) -> None: printer.info(f"Trying to scrape links from {Style.BRIGHT}{url}{Style.RESET_ALL}...") asyncio.run(scrape_links(url, recursive=False)) printer.success(f"Scraping completed..!") + + # Ask user if they want to export the results + if scraped_links: + export_response = printer.inp("\nDo you want to export the scraped links? (y/N) : ") + if export_response.lower() == 'y' or export_response.lower() == "yes": + printer.info("Available export formats:") + printer.info(" 1. TXT (plain text)") + printer.info(" 2. CSV (comma-separated values)") + printer.info(" 3. JSON (structured data)") + + format_choice = printer.inp("Choose format (1/2/3) [default: 1] : ").strip() + + format_map = { + "1": "txt", + "2": "csv", + "3": "json", + "": "txt" # default + } + + export_format = format_map.get(format_choice, "txt") + export_links(scraped_links, url, export_format) + + # Clear scraped links for next run + scraped_links.clear() + except Exception as e: printer.error(f"Error : {e}") except KeyboardInterrupt: From 0d97b01f34799ccdecea2d9ce51a3c2efb4ed3f7 Mon Sep 17 00:00:00 2001 From: AntoTheSol Date: Wed, 22 Oct 2025 23:57:04 -0400 Subject: [PATCH 2/3] Remove scraped_data from version control --- scraped_data/github_com_20251022_231702.txt | 124 -------------------- 1 file changed, 124 deletions(-) delete mode 100644 scraped_data/github_com_20251022_231702.txt diff --git a/scraped_data/github_com_20251022_231702.txt b/scraped_data/github_com_20251022_231702.txt deleted file mode 100644 index d6c2725..0000000 --- a/scraped_data/github_com_20251022_231702.txt +++ /dev/null @@ -1,124 +0,0 @@ -Scraped links from: https://github.com/AntoTheSol/H4X-Tools -Date: 2025-10-22 23:17:02 -Total links: 119 --------------------------------------------------------------------------------- - -https://GitHub.com/vil/H4X-Tools/commit/ -https://GitHub.com/vil/H4X-Tools/network/ -https://GitHub.com/vil/H4X-Tools/stargazers/ -https://docs.github.com -https://docs.github.com/ -https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax -https://docs.github.com/site-policy/github-terms/github-terms-of-service -https://docs.github.com/site-policy/privacy-policies/github-privacy-statement -https://git-scm.com/downloads -https://github.blog -https://github.com -https://github.com/ -https://github.com/AntoTheSol -https://github.com/AntoTheSol/H4X-Tools -https://github.com/AntoTheSol/H4X-Tools#GPL-3.0-1-ov-file -https://github.com/AntoTheSol/H4X-Tools#contributing-ov-file -https://github.com/AntoTheSol/H4X-Tools#current-tools -https://github.com/AntoTheSol/H4X-Tools#h4x-tools -https://github.com/AntoTheSol/H4X-Tools#license -https://github.com/AntoTheSol/H4X-Tools#linux -https://github.com/AntoTheSol/H4X-Tools#looking-for-contributors -https://github.com/AntoTheSol/H4X-Tools#readme-ov-file -https://github.com/AntoTheSol/H4X-Tools#setup -https://github.com/AntoTheSol/H4X-Tools#start-of-content -https://github.com/AntoTheSol/H4X-Tools#windows -https://github.com/AntoTheSol/H4X-Tools/actions -https://github.com/AntoTheSol/H4X-Tools/activity -https://github.com/AntoTheSol/H4X-Tools/blob/master/.gitignore -https://github.com/AntoTheSol/H4X-Tools/blob/master/CONTRIBUTING.md -https://github.com/AntoTheSol/H4X-Tools/blob/master/LICENSE -https://github.com/AntoTheSol/H4X-Tools/blob/master/README.md -https://github.com/AntoTheSol/H4X-Tools/blob/master/h4xtools.py -https://github.com/AntoTheSol/H4X-Tools/blob/master/requirements.txt -https://github.com/AntoTheSol/H4X-Tools/blob/master/setup.bat -https://github.com/AntoTheSol/H4X-Tools/blob/master/setup.sh -https://github.com/AntoTheSol/H4X-Tools/blob/master/update.bat -https://github.com/AntoTheSol/H4X-Tools/blob/master/update.sh -https://github.com/AntoTheSol/H4X-Tools/branches -https://github.com/AntoTheSol/H4X-Tools/commits/master/ -https://github.com/AntoTheSol/H4X-Tools/forks -https://github.com/AntoTheSol/H4X-Tools/projects -https://github.com/AntoTheSol/H4X-Tools/pulls -https://github.com/AntoTheSol/H4X-Tools/pulse -https://github.com/AntoTheSol/H4X-Tools/releases -https://github.com/AntoTheSol/H4X-Tools/security -https://github.com/AntoTheSol/H4X-Tools/stargazers -https://github.com/AntoTheSol/H4X-Tools/tags -https://github.com/AntoTheSol/H4X-Tools/tree/master/.github -https://github.com/AntoTheSol/H4X-Tools/tree/master/helper -https://github.com/AntoTheSol/H4X-Tools/tree/master/img -https://github.com/AntoTheSol/H4X-Tools/tree/master/resources -https://github.com/AntoTheSol/H4X-Tools/tree/master/utils -https://github.com/AntoTheSol/H4X-Tools/watchers -https://github.com/collections -https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FAntoTheSol%2FH4X-Tools&report=AntoTheSol+%28user%29 -https://github.com/customer-stories -https://github.com/enterprise -https://github.com/enterprise/startups -https://github.com/features -https://github.com/features/actions -https://github.com/features/code-review -https://github.com/features/code-search -https://github.com/features/codespaces -https://github.com/features/copilot -https://github.com/features/copilot/copilot-business -https://github.com/features/discussions -https://github.com/features/issues -https://github.com/features/models -https://github.com/features/spark -https://github.com/login?return_to=%2FAntoTheSol%2FH4X-Tools -https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FAntoTheSol%2FH4X-Tools -https://github.com/marketplace -https://github.com/mcp -https://github.com/megadose/holehe -https://github.com/partners -https://github.com/premium-support -https://github.com/pricing -https://github.com/readme -https://github.com/resources/articles -https://github.com/resources/articles?topic=ai -https://github.com/resources/articles?topic=devops -https://github.com/resources/articles?topic=security -https://github.com/resources/articles?topic=software-development -https://github.com/resources/events -https://github.com/resources/whitepapers -https://github.com/security -https://github.com/security/advanced-security -https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=AntoTheSol%2FH4X-Tools -https://github.com/solutions -https://github.com/solutions/executive-insights -https://github.com/solutions/industry -https://github.com/solutions/industry/financial-services -https://github.com/solutions/industry/government -https://github.com/solutions/industry/healthcare -https://github.com/solutions/industry/manufacturing -https://github.com/solutions/industry/nonprofits -https://github.com/solutions/use-case -https://github.com/solutions/use-case/app-modernization -https://github.com/solutions/use-case/ci-cd -https://github.com/solutions/use-case/devops -https://github.com/solutions/use-case/devsecops -https://github.com/sponsors -https://github.com/team -https://github.com/topics -https://github.com/trending -https://github.com/users/AntoTheSol/packages?repo_name=H4X-Tools -https://github.com/vil/H4X-Tools -https://github.com/vil/H4X-Tools/blob/master/img/gui-v0.3.2.png -https://github.com/vil/H4X-Tools/issues -https://github.com/vil/H4X-Tools/wiki -https://github.com/why-github -https://github.community/ -https://pypi.org/project/Faker/ -https://resources.github.com/learn/pathways -https://skills.github.com -https://support.github.com?tags=dotcom-footer -https://www.githubstatus.com/ -https://www.gnu.org/licenses/gpl-3.0.txt -https://www.python.org/downloads/ From 1a184501861cabfba3d2ad850da0b4d0f81662aa Mon Sep 17 00:00:00 2001 From: AntoTheSol Date: Wed, 22 Oct 2025 23:58:03 -0400 Subject: [PATCH 3/3] Added scraped_data folder in .gitignore file --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 819ac9a..f2e797d 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,7 @@ tokenlogger .idea build dist +scraped_data venv .venv .env