Commit graph

87 commits

Author SHA1 Message Date
Eugen Rochko
e263db276f
Change indexing jobs to use database replica (#26692) 2023-08-27 22:38:01 +02:00
jsgoldstein
30c191aaa0
Add new public status index (#26344)
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-08-24 16:40:04 +02:00
Eugen Rochko
a0fad5c8bb
Change indexing frequency from 5 minutes to 1 minute, add locks to schedulers (#26304) 2023-08-03 11:04:05 +02:00
Claire
b4e739ff0f
Change interaction modal in web UI (#26075)
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
2023-07-27 16:11:17 +02:00
Claire
144a406d33
Clean up unused application records (#24871) 2023-07-21 13:13:16 +02:00
Emelia Smith
3a91603b15
Prevent UserCleanupScheduler from overwhelming streaming (#25519) 2023-06-20 18:04:35 +02:00
Claire
c3c7d59072
Fix infinite loop in AccountsStatusesCleanupScheduler (#24840) 2023-05-04 10:13:00 +02:00
Claire
8b636a29c6
Improve automatic post cleanup worker performances (#24785) 2023-05-03 10:31:40 +02:00
Claire
32a030dd74
Rewrite import feature (#21054) 2023-05-02 12:08:48 +02:00
Claire
5dc3173ef8
Fix AccountsStatusesCleanupScheduler not spreading deletes across accounts correctly (#24607) 2023-04-23 22:25:40 +02:00
Claire
501d6197c4
Change automatic post deletion thresholds and load detection (#24614) 2023-04-21 18:14:19 +02:00
Vyr Cossont
d5ad8b6422
Fix Redis client and type errors introduced in #24285 (#24342) 2023-04-01 08:28:35 +02:00
Vyr Cossont
7646ad8a2b
IndexingScheduler: fetch and import in batches (#24285)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-03-31 14:38:47 +02:00
Jean byroot Boussier
922837dc96
Upgrade to latest redis-rb 4.x and fix deprecations (#23616)
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
2023-03-04 16:38:28 +01:00
Claire
20b80c62ff
Change auto-deletion throttling constants to better scale with server size (#23320) 2023-02-23 16:52:40 +01:00
Nick Schonning
717683d1c3
Autofix Rubocop remaining Layout rules (#23679) 2023-02-20 06:58:28 +01:00
Nick Schonning
21bf326356
Autofix Rubocop Rails/Pluck (#23730) 2023-02-20 02:28:40 +01:00
Nick Schonning
2177daeae9
Autofix Rubocop Style/RedundantBegin (#23703) 2023-02-19 07:09:40 +09:00
Claire
9edefc779f
Fix UserCleanupScheduler crash when an unconfirmed account has a moderation note (#23318)
* Fix `UserCleanupScheduler` crash when an unconfirmed account has a moderation note

* Add tests
2023-02-07 01:14:44 +01:00
luzpaz
596923da4a
Fix typos in source documentation (#21046)
Fixed 2 source comment/documentation typos
2022-12-15 15:57:26 +01:00
Claire
5333447be0
Change account deletion requests to spread out over time (#20222) 2022-11-09 14:08:19 +01:00
Eugen Rochko
d0c9ac3919
Fix indexing scheduler trying to index when Elasticsearch is disabled (#19805)
Fix #19646
2022-11-05 22:31:52 +01:00
Claire
03d9618595
Fix UserCleanupScheduler crash when an unconfirmed account has a moderation note (#19629)
Fixes #19109
2022-11-01 12:59:23 +01:00
Eugen Rochko
bf0ab3e0fa
Fix vacuum scheduler missing lock, locks never expiring (#19458)
Remove vacuuming of orphaned preview cards
2022-10-26 12:10:48 +02:00
Eugen Rochko
5c9abdeff1
Add retention policy for cached content and media (#19232) 2022-09-27 03:08:19 +02:00
Eugen Rochko
0b3e4fd5de
Remove digest e-mails (#17985)
* Remove digest e-mails

* Remove digest-related code
2022-08-25 23:38:22 +02:00
Jeremy Kescher
25e076505e
ip_cleanup_scheduler: Make IP and session retention configurable (#18757) 2022-07-07 03:14:28 +02:00
Eugen Rochko
679b7158e3
Change search indexing to use batches to minimize resource usage (#18451) 2022-05-18 23:29:14 +02:00
Eugen Rochko
f6d35ed57d
Remove IP matching from e-mail domain blocks (#18190)
Clear out e-mail domain blocks created from automatically resolved DNS records
2022-04-29 23:27:03 +02:00
Eugen Rochko
3917353645
Fix single Redis connection being used across all threads (#18135)
* Fix single Redis connection being used across all Sidekiq threads

* Fix tests
2022-04-28 17:47:34 +02:00
Jeong Arm
3349cbeb3e
Fix clean_discarded_statuses (#17822) 2022-03-19 09:24:08 +01:00
Eugen Rochko
50ea54b3ed
Change authorized applications page (#17656)
* Change authorized applications page

* Hide revoke button for superapps and suspended accounts

* Clean up db/schema.rb
2022-03-01 16:48:58 +01:00
Claire
3d60708508
Fix crash in EmailDomainBlockRefreshScheduler (#17649) 2022-02-26 16:28:28 +01:00
Eugen Rochko
27965ce5ed
Add trending statuses (#17431)
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
Eugen Rochko
a29a982eaa
Change e-mail domain blocks to block IPs dynamically (#17635)
* Change e-mail domain blocks to block IPs dynamically

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-24 17:28:23 +01:00
Eugen Rochko
35850f8195
Fix localization of cold-start follow recommendations (#17479) 2022-02-08 01:53:49 +01:00
Claire
52c1b86964
Fix Ruby 2.5 incompatibility (#17465) 2022-02-07 19:57:06 +01:00
Claire
03d59340da
Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Eugen Rochko
14f436c457
Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Eugen Rochko
8e84ebf0cb
Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
Takeshi Umeda
bda8e4f815
Fix follow recommendation biased towards older accounts (#17126) 2021-12-13 23:21:14 +01:00
Eugen Rochko
6e50134a42
Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire
4ac78e2a06
Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Eugen Rochko
d174d12c83
Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
Claire
d9ae3db8d5
Improve performance of follow recommendation scheduler (#16159)
Express follow_recommendations in terms of account_summaries rather than
accounts, integrate filters that are unconditionally used, and materialize
the resulting view.

This should result in the bulk of the computation being performed only once
instead of **once per recommendation language**.
2021-05-05 22:04:52 +02:00
Eugen Rochko
f7117646af
Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Eugen Rochko
e89e976e92
Fix configuration for sidekiq-unique-jobs after 7.x upgrade (#15908)
Remove locks from scheduled jobs
2021-03-15 11:17:43 +01:00
ThibG
3249d35bdc
Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
Eugen Rochko
216b85b053
Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
Eugen Rochko
5e1364c448
Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00