OpenSource/telegram-crawler
OpenSource/telegram-crawler
1
0
Fork 0
mirror of https://github.com/MarshalX/telegram-crawler.git synced 2025-03-14 04:51:42 +01:00
Code Issues Projects Releases Packages Wiki Activity

exclude dynamic pages from blogfork

Browse source
This commit is contained in:
Il'ya (Marshal) 2022-05-14 01:06:40 +02:00
parent c6cbcab230
commit 52baa36520
3 changed files with 9 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
make_tracked_links_list.py
View file

@ -111,6 +111,15 @@ CRAWL_RULES = {
'validatedRequestedInfo',
},
},
'blogfork.telegram.org': {
'deny': {
'bots/payments',
'tdlib/docs/classtd',
'validatedRequestedInfo',
},
},
'telegram.org': {
'deny': {
r'apps$',

3
tracked_links.txt
View file

@ -64,8 +64,6 @@ blogfork.telegram.org/bots/api-changelog
blogfork.telegram.org/bots/faq
blogfork.telegram.org/bots/games
blogfork.telegram.org/bots/inline
blogfork.telegram.org/bots/payments
blogfork.telegram.org/bots/payments/currencies.json
blogfork.telegram.org/bots/samples
blogfork.telegram.org/bots/samples/hellobot
blogfork.telegram.org/bots/self-signed
@ -727,7 +725,6 @@ blogfork.telegram.org/constructor/payments.paymentReceipt
blogfork.telegram.org/constructor/payments.paymentResult
blogfork.telegram.org/constructor/payments.paymentVerificationNeeded
blogfork.telegram.org/constructor/payments.savedInfo
blogfork.telegram.org/constructor/payments.validatedRequestedInfo
blogfork.telegram.org/constructor/peerBlocked
blogfork.telegram.org/constructor/peerChannel
blogfork.telegram.org/constructor/peerChat

7
tracked_res_links.txt
View file

@ -1,22 +1,15 @@
blogfork.telegram.org/file/464001058/109a6/XLMi7E5_IvY.218203/90b01ba97b6bc3ab8e
blogfork.telegram.org/file/464001085/2/E4hNXSNQimQ.2503/bf6ffcab3cb3afd43d
blogfork.telegram.org/file/464001162/3/C2wdLOXaMA0.97425/32091afe52ef0e59f2
blogfork.telegram.org/file/464001257/12087/QNQUbIi864k.909800.mp4/8ea7adad7db407388e
blogfork.telegram.org/file/464001274/2/e1tzJ2HkJOk.156878/ca3fda3984e764eee4
blogfork.telegram.org/file/464001285/4/UBRF1ViiEOU.106159/0bbcdcb7f5af77a782
blogfork.telegram.org/file/464001302/1/H6KwJFdtIRk.4794843.mp4/e8b988bdd940e7a952
blogfork.telegram.org/file/464001467/1/aAY6bzDmuCI.37875/076ad2c2586eca3c06
blogfork.telegram.org/file/464001511/3/yrhjSo2FPDw.14020/b040084e8ad9951a4b
blogfork.telegram.org/file/464001548/2/Ug-7BpYHOGI.51801/f0095722cd99a34a46
blogfork.telegram.org/file/464001648/1/1-P1Pb7TusA.105628/b9cb2cf4c7427302c0
blogfork.telegram.org/file/464001679/11aa9/KQx_BlPVXRo.4922145.mp4/c65433c8ac11a347a8
blogfork.telegram.org/file/464001701/6/-BJZJqe9VeA.70226/75a192d58d76dc96c1
blogfork.telegram.org/file/464001702/1194c/mBQhTbIWqw8.451557/08727d5d763e24d750
blogfork.telegram.org/file/464001705/2/Hg-klQnZ_n4.4889245.mp4/d324511ffdab7b4668
blogfork.telegram.org/file/464001715/1/YPZWHRfqpXI.3577036.mp4/bb24535cb655a57cd7
blogfork.telegram.org/file/464001836/2/rwUQeqMOsnI.31142/b0cc9d885ccb78ab39
blogfork.telegram.org/file/464001838/10fa2/WrJmkuIMan0.1217917.mp4/e25a5f31bc4e6493f7
blogfork.telegram.org/file/464001896/2/9gTF8_aGQkE.45826/d5b11a999a40a1a6d9
blogfork.telegram.org/file/811140010/2/omTlwQW_Wpk.133258/5db0b29d43f5558413
blogfork.telegram.org/file/811140015/1734/8VZFkwWXalM.97872/6127fa62d8a0bf2b3c
blogfork.telegram.org/file/811140016/2/2b_B7nq9OQA/161b06d38843930fe5