77 commits

Author	SHA1	Message	Date
Il'ya (Marshal)	4b84e48098	add contest.com	2021-06-01 09:03:44 +02:00
Il'ya (Marshal)	1b8af50843	add additional urls section; ... add tracking of mtproto schema.	2021-06-01 08:57:18 +02:00
Il'ya (Marshal)	83fbc4b4a3	add apple privacy page	2021-05-28 20:43:54 +02:00
Il'ya (Marshal)	f9cddfdb47	add CIDR	2021-05-24 13:52:40 +02:00
Il'ya (Marshal)	fe44e713b7	exclude dynamic list of users	2021-05-05 12:59:01 +02:00
Il'ya (Marshal)	002c2c61d9	temp exclude strange dynamic page	2021-05-01 16:28:12 +02:00
Il'ya (Marshal)	ede1ee0888	add mtproto api to crawler	2021-05-01 15:39:24 +02:00
Il'ya (Marshal)	e55ff9fdbc	exclude frequently updated pages	2021-05-01 15:10:20 +02:00
Il'ya (Marshal)	0aa464ac96	Revert "test alert system" ... This reverts commit fad0585d	2021-04-25 18:59:03 +02:00
Il'ya (Marshal)	fad0585daf	test alert system	2021-04-25 16:37:52 +02:00
Il'ya (Marshal)	ae43e9e442	cleanup readme file; disable tests of link crawler.	2021-04-25 14:13:52 +02:00
Il'ya (Marshal)	2dffddeaae	fix resending of requests; ... fix timeout settings; fix handling of invalid urls; edit schedule of generation list to test.	2021-04-25 08:18:06 +02:00
Il'ya (Marshal)	9a4a6e55ac	temporary remove /apps page with easter egg from Telegram (random order web clients); update links list as often as possible for tests	2021-04-24 23:06:56 +02:00
Il'ya (Marshal)	dcd5adc08a	fix, improve and cleanup links crawler; add diff or files	2021-04-24 22:40:39 +02:00
Il'ya (Marshal)	e927373229	disable crawling of bugs.telegram.org	2021-04-24 14:43:01 +02:00
Il'ya (Marshal)	6ceb897e33	move tracked links list to main branch; ... add generation of content tree; add workflow for tree generation; add new tg hidden link.	2021-04-24 14:19:01 +02:00
Il'ya (Marshal)	5f29beda73	handle timeout and retry	2021-04-24 11:38:33 +02:00
Il'ya (Marshal)	cd5653605f	rewrite logic of rules system (not its uses regex); ... fix trailing slash; many attempt to send request;	2021-04-24 11:29:19 +02:00
Il'ya (Marshal)	b2afbd72fc	rework exclude system; add exclude by count of slashes; add translations.telegram.org	2021-04-24 00:06:50 +02:00
Il'ya (Marshal)	e385f45da6	print lead time	2021-04-23 22:49:21 +02:00
Il'ya (Marshal)	246a0087e1	fix performance and memory issues	2021-04-23 22:38:54 +02:00
Il'ya (Marshal)	d449290dfb	add ability to set hidden urls	2021-04-23 22:31:23 +02:00
Il'ya (Marshal)	fba0bd1d64	return api section of tg core	2021-04-23 22:05:00 +02:00
Il'ya (Marshal)	c118a32a2a	test correct diff	2021-04-23 21:52:14 +02:00
Il'ya (Marshal)	4a422db809	the end.	2021-04-23 21:47:15 +02:00
Il'ya (Marshal)	b053015289	fix set operation	2021-04-23 20:44:17 +02:00
Il'ya (Marshal)	80934fecaa	Initial commit	2021-04-23 20:34:42 +02:00