OpenSource/telegram-crawler
OpenSource/telegram-crawler
1
0
Fork 0
mirror of https://github.com/MarshalX/telegram-crawler.git synced 2024-12-02 21:17:49 +01:00
Code Issues Projects Releases Packages Wiki Activity
588 commits 3 branches 0 tags 797 MiB
Commit graph

62 commits

Author SHA1 Message Date
Il'ya (Marshal)
4f4a1568b9 exclude emoji images 2022-05-27 17:13:04 +02:00
Il'ya (Marshal)
52baa36520 exclude dynamic pages from blogfork 2022-05-14 01:06:40 +02:00
Il'ya (Marshal)
dcca4f7e1a add blogfork 2022-05-14 00:35:40 +02:00
Il'ya (Marshal)
9cab070822 reorganise structure of data folder; separate the "making files tree" workflow on 2 part (should be separated by 4 later) 2022-05-13 11:57:11 +02:00
Il'ya (Marshal)
f468f5a966 rollback layer 2022-05-11 12:42:08 +02:00
Il'ya (Marshal)
c6e13dc54d stel dev layer, headers, loop instead of rec; workflow timeout; aiohttp limit increase. 2022-05-11 11:10:08 +02:00
Il'ya (Marshal)
9790dcb8e0 delete urls with random content 2022-04-24 09:43:32 +02:00
Il'ya (Marshal)
2b36099ecc add more tracking of files 2022-04-24 09:02:48 +02:00
Il'ya (Marshal)
14fb9bc6fc a little speedup 2022-04-18 00:11:29 +02:00
Il'ya (Marshal)
65628a2c23 add tracking of tdesktop updates 2022-04-17 23:31:44 +02:00
Il'ya (Marshal)
cb5e8c1069 add crawling of scripts from other scripts. close #2 2022-04-10 19:33:33 +02:00
Il'ya (Marshal)
961320022c add tracking of all media 2022-04-10 17:46:36 +02:00
Il'ya (Marshal)
03c9c08361 add webapp cofe demo 2022-04-09 14:21:12 +02:00
Il'ya (Marshal)
508d6ea2cd add new url of webapp demo 2022-04-07 08:16:44 +02:00
Il'ya (Marshal)
26eb32e2ae add css files of translations platform 2022-03-30 15:09:50 +02:00
Il'ya (Marshal)
29077dd280 add WebApp links and sparkle of macOS beta 2022-03-30 09:19:53 +02:00
Il'ya (Marshal)
6932a49c2a add tracking of all en translations strings 2022-03-12 01:20:05 +01:00
Il'ya (Marshal)
7aee3658e3 resend requests with 5XX status code 2022-02-24 21:58:17 +01:00
Il'ya (Marshal)
d74e0551d9 addd sticker related links 2022-01-31 19:22:23 +01:00
Il'ya (Marshal)
3fda615c53 add video stickers guidelines 2022-01-29 12:02:16 +01:00
Il'ya (Marshal)
344e550bd9 add 304 as success response; 
add handling of xml;
add sparkle file of macOS client;
fix request retrying with 500 status code;
fix encoding.
 2021-12-30 14:22:01 +01:00
Il'ya (Marshal)
c3a66d3397 return convertToGigagroup on corefork 2021-11-29 19:21:16 +01:00
Il'ya (Marshal)
2f84611c90 add td_api of tdlib 2021-11-11 10:18:01 +01:00
Il'ya (Marshal)
57ace577fa add mtproto tl 2021-11-04 09:24:24 +01:00
Il'ya (Marshal)
45c7211f67 fix tracking of ad panel 2021-10-26 17:55:38 +02:00
Il'ya (Marshal)
e5e09b7156 new link for ads 2021-10-26 17:40:07 +02:00
Il'ya (Marshal)
6262ec0372 fix bug with tracking of plain text urls 2021-10-01 19:47:42 +02:00
Il'ya (Marshal)
d17a2978f7 add getProxyConfig 2021-10-01 19:18:27 +02:00
Il'ya (Marshal)
9e3ac83140 exclude strange url 2021-09-26 21:02:35 +02:00
Il'ya (Marshal)
58e3417baa return tracking of corefork 2021-09-09 20:43:53 +02:00
Il'ya
addc1f66e2
waiting for ads 2021-09-01 14:55:53 +02:00
Alisa Sireneva
22d07f21f7
Added TDLib schema link 2021-08-31 15:18:25 +03:00
Il'ya (Marshal)
1961b658a8 add comparing output to github ci logs 2021-06-27 23:06:29 +02:00
Il'ya (Marshal)
12672aba45 resend requests with server error response 2021-06-21 18:39:20 +02:00
Il'ya (Marshal)
0b86273326 add logging of skips 2021-06-21 16:09:58 +02:00
Il'ya (Marshal)
4b84e48098 add contest.com 2021-06-01 09:03:44 +02:00
Il'ya (Marshal)
1b8af50843 add additional urls section; 
add tracking of mtproto schema.
 2021-06-01 08:57:18 +02:00
Il'ya (Marshal)
83fbc4b4a3 add apple privacy page 2021-05-28 20:43:54 +02:00
Il'ya (Marshal)
f9cddfdb47 add CIDR 2021-05-24 13:52:40 +02:00
Il'ya (Marshal)
fe44e713b7 exclude dynamic list of users 2021-05-05 12:59:01 +02:00
Il'ya (Marshal)
002c2c61d9 temp exclude strange dynamic page 2021-05-01 16:28:12 +02:00
Il'ya (Marshal)
ede1ee0888 add mtproto api to crawler 2021-05-01 15:39:24 +02:00
Il'ya (Marshal)
e55ff9fdbc exclude frequently updated pages 2021-05-01 15:10:20 +02:00
Il'ya (Marshal)
0aa464ac96 Revert "test alert system" 
This reverts commit fad0585d
 2021-04-25 18:59:03 +02:00
Il'ya (Marshal)
fad0585daf test alert system 2021-04-25 16:37:52 +02:00
Il'ya (Marshal)
ae43e9e442 cleanup readme file; disable tests of link crawler. 2021-04-25 14:13:52 +02:00
Il'ya (Marshal)
2dffddeaae fix resending of requests; 
fix timeout settings;
fix handling of invalid urls;
edit schedule of generation list to test.
 2021-04-25 08:18:06 +02:00
Il'ya (Marshal)
9a4a6e55ac temporary remove /apps page with easter egg from Telegram (random order web clients); update links list as often as possible for tests 2021-04-24 23:06:56 +02:00
Il'ya (Marshal)
dcd5adc08a fix, improve and cleanup links crawler; add diff or files 2021-04-24 22:40:39 +02:00
Il'ya (Marshal)
e927373229 disable crawling of bugs.telegram.org 2021-04-24 14:43:01 +02:00