OpenSource/telegram-crawler
OpenSource/telegram-crawler
1
0
Fork 0
mirror of https://github.com/MarshalX/telegram-crawler.git synced 2025-03-14 04:51:42 +01:00
Code Issues Projects Releases Packages Wiki Activity

lmao happy 8.8

Browse source
This commit is contained in:
Il'ya (Marshal) 2022-06-19 20:53:10 +02:00
parent 33fbdcad25
commit df8c3e8421
1 changed files with 3 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
make_tracked_links_list.py
View file

@ -335,7 +335,9 @@ async def crawl(url: str, session: aiohttp.ClientSession):
await _crawl(url, session)
except (ServerSideError, ServerDisconnectedError, TimeoutError, ClientConnectorError):
logger.warning(f'Client or timeout error. Retrying {url}')
VISITED_LINKS.remove(url)
if url in VISITED_LINKS:
VISITED_LINKS.remove(url)
else:
break