🕷 Automatically detect changes made to the official Telegram sites, clients and servers. https://t.me/tgcrawl
Find a file
2021-04-24 17:34:55 +02:00
.github/workflows sub dynamic api hash from html 2021-04-24 15:01:53 +02:00
.gitignore enable workflow 2021-04-24 14:28:06 +02:00
LICENSE add readme and license file 2021-04-24 17:34:55 +02:00
make_files_tree.py sub passport ssid and nonce 2021-04-24 16:42:51 +02:00
make_tracked_links_list.py disable crawling of bugs.telegram.org 2021-04-24 14:43:01 +02:00
README.md add readme and license file 2021-04-24 17:34:55 +02:00
requirements.txt move tracked links list to main branch; 2021-04-24 14:19:01 +02:00
tracked_links.txt Update tracked links 2021-04-24 15:04:10 +00:00

Telegram Web Crawler

This project is developed to automatically detect changes made to the official Telegram sites. This is necessary for anticipating future updates and other things (new vacancies, API updates).

Fetch new content of tracked links to files Site updates tracker

Generate or update list of tracked links Site links tracker

How it should work in dreams

  1. Link crawling runs once an hour. Starts crawling from the home page of the site. Detects relative and absolute sub links and recursively repeats the operation. Writes a list of unique links for future content comparison. Additionally, there is the ability to add links by hand to help the script find more hidden (links to which no one refers) links.

  2. Content crawling is launched as often as possible and uses the existing list of links collected in step 1. Going through the base it gets contains and builds a system of subfolders and files. Removes all dynamic content from files.

  3. Works without own servers. Used GitHub Actions. All file changes are tracked by the GIT and are beautifully displayed on the GitHub. Github Actions should be built correctly only if there are changes on the Telegram website. Otherwise, the workflow should fail. If build was successful, we can send notifications to Telegram channel and so on.

The real world

  • Scheduled actions cannot be run more than once every 5 minutes.
  • GitHub Actions freeze for 5 minutes when updating the list of links. Locally, this takes less than 10 seconds for several hundred requests. This is not really a big problem. The list of links is rarely updated.
  • When updating links, ~one link may be lost, and the next list generation it is returned. This will lead to the successful execution of workflow when there were no server changes. Most likely this is a bug in my script that can be fixed. As a last resort, compare the old version of the linkbase with the new one.

TODO list

  • bug fixes;
  • alert system.

License

Licensed under the MIT License.