Парсер сайтов из веб архива

Archive-Master v0.0.1 is a program for creating local copies of websites from the Internet archive

Archive-Master v0.0.1 is a specialized software solution designed to automatically restore websites based on saved copies from a web archive. The utility allows you to download historical versions of pages for a selected time period and create a full-fledged local copy of the resource. This approach makes the program useful for analyzing the structure of websites, data backup, SEO auditing, and studying previously published content.

The main feature of Archive-Master is the complete automation of the data acquisition process. The user only needs to specify the domain name and the time range of the search, after which the program independently detects available archived snapshots of the pages, downloads them and restores the site structure on the computer. As a result, an offline version of the resource is generated, containing HTML pages, images, style sheets, and JavaScript files.

Before starting work, an authorization system is provided that restricts access to the program. After successful login, the user enters the website address without specifying the protocol, as well as the start and end dates of the search in the YYYY-MMDD format. Based on this data, a search is performed for suitable archived copies.

Next, an OFFLINE_SITE working directory is automatically created, inside which an assets directory is formed to store all related files. After preparing the structure, a sequential scan of the site begins: the home page becomes the starting point, and the internal links found are queued for subsequent processing.

During the processing of each page, the program performs a set of operations:

accesses the web archive service to search for the most appropriate archived version of the page for the specified period;
receives the HTML code of the found snapshot;
cleans the document of unnecessary elements, including individual scripts and iframes.;
automatically downloads related images, CSS files, and JavaScript, saving them in a local folder.;
modifies internal links so that all pages and resources open correctly without a network connection.

Each processed document is saved as a separate HTML file, and the directory structure follows the original site as closely as possible. To increase efficiency, the program maintains a list of already processed pages, eliminating repeated downloads of identical materials and preventing endless link crawling.

To increase reliability, a mechanism for repeated download attempts has been implemented. If temporary errors or network failures occur when accessing the server, the program automatically executes a repeat request. Additional pauses between requests help reduce the likelihood of restrictions from the remote service and ensure more stable operation.

Upon completion of processing, the user receives a fully prepared offline copy of the site, which can be opened in any modern browser without Internet access. Archive-Master will become a useful tool for SEO specialists, web developers, site administrators, researchers and anyone who works with archived versions of Internet resources.

The program greatly simplifies the process of restoring historical data, allowing you to quickly convert archived images into a full-fledged local copy of the site, suitable for further analysis, storage and use.

Instructions are in the program folder!!!