OpenSource/telegram-crawler
OpenSource/telegram-crawler
1
0
Fork 0
mirror of https://github.com/MarshalX/telegram-crawler.git synced 2024-10-23 17:47:21 +02:00
Code Issues Projects Releases Packages Wiki Activity

fix bug with tracking of plain text urls

Browse source
This commit is contained in:
Il'ya (Marshal) 2021-10-01 19:47:35 +02:00
parent 864ed719e7
commit 6262ec0372
1 changed files with 2 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
make_tracked_links_list.py
View file

@ -235,9 +235,9 @@ async def crawl(url: str, session: aiohttp.ClientSession):
await asyncio.gather(*[crawl(url, session) for url in sub_links])
elif 'application/javascript' in content_type:
LINKS_TO_TRACK.add(url)
elif 'text/css' in content_type:
elif 'css' in content_type:
LINKS_TO_TRACK.add(url)
elif 'text/plain' in content_type:
elif 'plain' in content_type:
LINKS_TO_TRACK.add(url)
elif 'application/json' in content_type:
LINKS_TO_TRACK.add(url)