OpenSource/telegram-crawler
OpenSource/telegram-crawler
1
0
Fork 0
mirror of https://github.com/MarshalX/telegram-crawler.git synced 2025-03-15 13:22:43 +01:00
Code Issues Projects Releases Packages Wiki Activity

return tracking of corefork

Browse source
This commit is contained in:
Il'ya (Marshal) 2021-09-09 20:43:53 +02:00
parent 4a72253936
commit 58e3417baa
1 changed files with 12 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
make_tracked_links_list.py
View file

@ -14,7 +14,7 @@ PROTOCOL = 'https://'
BASE_URL = 'telegram.org'
# its necessary to help crawler to find more links
HIDDEN_URLS = {
# 'corefork.telegram.org',
'corefork.telegram.org',
'telegram.org/privacy/gmailbot',
'telegram.org/tos',
@ -77,6 +77,17 @@ CRAWL_RULES = {
'validatedRequestedInfo',
},
},
'corefork.telegram.org': {
'deny': {
'file/',
'bots/payments',
'tdlib/docs/classtd',
'validatedRequestedInfo',
},
},
'telegram.org': {
'deny': {
'file/',