OpenSource/telegram-crawler
OpenSource/telegram-crawler
1
0
Fork 0
mirror of https://github.com/MarshalX/telegram-crawler.git synced 2024-10-23 17:47:21 +02:00
Code Issues Projects Releases Packages Wiki Activity

add CIDR

Browse source
This commit is contained in:
Il'ya (Marshal) 2021-05-24 13:52:40 +02:00
parent ad9fdc5496
commit f9cddfdb47
1 changed files with 4 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
make_tracked_links_list.py
View file

@ -24,6 +24,8 @@ HIDDEN_URLS = {
'desktop.telegram.org/changelog',
'instantview.telegram.org/rules',
'core.telegram.org/resources/cidr.txt',
}
BASE_URL_REGEX = r'telegram.org'
@ -208,6 +210,8 @@ async def crawl(url: str, session: aiohttp.ClientSession):
LINKS_TO_TRACK.add(url)
elif 'text/css' in content_type:
LINKS_TO_TRACK.add(url)
elif 'text/plain' in content_type:
LINKS_TO_TRACK.add(url)
elif 'application/json' in content_type:
LINKS_TO_TRACK.add(url)
else: