OpenSource/telegram-crawler
OpenSource/telegram-crawler
1
0
Fork 0
mirror of https://github.com/MarshalX/telegram-crawler.git synced 2024-12-02 13:07:32 +01:00
Code Issues Projects Releases Packages Wiki Activity
143 commits 3 branches 0 tags 795 MiB
Commit graph

29 commits

Author SHA1 Message Date
Il'ya (Marshal)
12672aba45 resend requests with server error response 2021-06-21 18:39:20 +02:00
Il'ya (Marshal)
0b86273326 add logging of skips 2021-06-21 16:09:58 +02:00
Il'ya (Marshal)
4b84e48098 add contest.com 2021-06-01 09:03:44 +02:00
Il'ya (Marshal)
1b8af50843 add additional urls section; 
add tracking of mtproto schema.
 2021-06-01 08:57:18 +02:00
Il'ya (Marshal)
83fbc4b4a3 add apple privacy page 2021-05-28 20:43:54 +02:00
Il'ya (Marshal)
f9cddfdb47 add CIDR 2021-05-24 13:52:40 +02:00
Il'ya (Marshal)
fe44e713b7 exclude dynamic list of users 2021-05-05 12:59:01 +02:00
Il'ya (Marshal)
002c2c61d9 temp exclude strange dynamic page 2021-05-01 16:28:12 +02:00
Il'ya (Marshal)
ede1ee0888 add mtproto api to crawler 2021-05-01 15:39:24 +02:00
Il'ya (Marshal)
e55ff9fdbc exclude frequently updated pages 2021-05-01 15:10:20 +02:00
Il'ya (Marshal)
0aa464ac96 Revert "test alert system" 
This reverts commit fad0585d
 2021-04-25 18:59:03 +02:00
Il'ya (Marshal)
fad0585daf test alert system 2021-04-25 16:37:52 +02:00
Il'ya (Marshal)
ae43e9e442 cleanup readme file; disable tests of link crawler. 2021-04-25 14:13:52 +02:00
Il'ya (Marshal)
2dffddeaae fix resending of requests; 
fix timeout settings;
fix handling of invalid urls;
edit schedule of generation list to test.
 2021-04-25 08:18:06 +02:00
Il'ya (Marshal)
9a4a6e55ac temporary remove /apps page with easter egg from Telegram (random order web clients); update links list as often as possible for tests 2021-04-24 23:06:56 +02:00
Il'ya (Marshal)
dcd5adc08a fix, improve and cleanup links crawler; add diff or files 2021-04-24 22:40:39 +02:00
Il'ya (Marshal)
e927373229 disable crawling of bugs.telegram.org 2021-04-24 14:43:01 +02:00
Il'ya (Marshal)
6ceb897e33 move tracked links list to main branch; 
add generation of content tree;
add workflow for tree generation;
add new tg hidden link.
 2021-04-24 14:19:01 +02:00
Il'ya (Marshal)
5f29beda73 handle timeout and retry 2021-04-24 11:38:33 +02:00
Il'ya (Marshal)
cd5653605f rewrite logic of rules system (not its uses regex); 
fix trailing slash;
many attempt to send request;
 2021-04-24 11:29:19 +02:00
Il'ya (Marshal)
b2afbd72fc rework exclude system; add exclude by count of slashes; add translations.telegram.org 2021-04-24 00:06:50 +02:00
Il'ya (Marshal)
e385f45da6 print lead time 2021-04-23 22:49:21 +02:00
Il'ya (Marshal)
246a0087e1 fix performance and memory issues 2021-04-23 22:38:54 +02:00
Il'ya (Marshal)
d449290dfb add ability to set hidden urls 2021-04-23 22:31:23 +02:00
Il'ya (Marshal)
fba0bd1d64 return api section of tg core 2021-04-23 22:05:00 +02:00
Il'ya (Marshal)
c118a32a2a test correct diff 2021-04-23 21:52:14 +02:00
Il'ya (Marshal)
4a422db809 the end. 2021-04-23 21:47:15 +02:00
Il'ya (Marshal)
b053015289 fix set operation 2021-04-23 20:44:17 +02:00
Il'ya (Marshal)
80934fecaa Initial commit 2021-04-23 20:34:42 +02:00