Commit Graph

439 Commits

Author SHA1 Message Date
Claire
4e55001e65
Add a short-lived lock to trend refresh scheduler (#27253) 2023-10-03 12:21:32 +02:00
Claire
5e56f3db65
Change some worker lock TTLs (#27246) 2023-10-03 10:09:00 +02:00
Claire
1e2d4975cf
Fix import progress not updating on certain failures (#27247) 2023-10-02 16:20:04 +02:00
Claire
cab4cbfa5c
Fix “Scoped order is ignored, it's forced to be batch order.” warnings (#26793) 2023-09-05 15:37:23 +02:00
Claire
16681e0f20
Add admin notifications for new Mastodon versions (#26582) 2023-09-01 17:47:07 +02:00
Claire
5c0a9aac3b
Revert to using primary database in IndexingScheduler (#26754) 2023-09-01 15:09:44 +02:00
Eugen Rochko
e263db276f
Change indexing jobs to use database replica (#26692) 2023-08-27 22:38:01 +02:00
Eugen Rochko
f8d2fea2e6
Change queue of job when opting into search from default to pull (#26688) 2023-08-27 18:58:20 +02:00
jsgoldstein
30c191aaa0
Add new public status index (#26344)
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-08-24 16:40:04 +02:00
Claire
9ed0c91a37
Add auto-refresh of accounts we get new messages/edits of (#26510) 2023-08-21 16:09:26 +02:00
Eugen Rochko
a0fad5c8bb
Change indexing frequency from 5 minutes to 1 minute, add locks to schedulers (#26304) 2023-08-03 11:04:05 +02:00
Emelia Smith
317cd26fba
Remove queued_at value from pubsub payloads (#26173) 2023-07-27 16:19:33 +02:00
Claire
b4e739ff0f
Change interaction modal in web UI (#26075)
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
2023-07-27 16:11:17 +02:00
Claire
144a406d33
Clean up unused application records (#24871) 2023-07-21 13:13:16 +02:00
Claire
943f27f437
Remove unfollowed hashtag posts from home feed (#26028) 2023-07-17 13:56:28 +02:00
Eugen Rochko
fdc3ff7c2d
Change notifications API to use a replica (#25874) 2023-07-12 17:06:00 +02:00
Eugen Rochko
a1f5188c8c
Change feed merge, unmerge and regeneration workers to use a replica (#25849) 2023-07-10 03:06:09 +02:00
Kurtis Rainbolt-Greene
e4cfe4b3db
First pass at multi-database for read replica using Rails native adapter (#25693)
Co-authored-by: emilweth <7402764+emilweth@users.noreply.github.com>
2023-07-08 19:45:36 +02:00
Claire
4658263b4a
Fix re-activated accounts being deleted by AccountDeletionWorker (#25711) 2023-07-04 18:36:24 +02:00
Emelia Smith
3a91603b15
Prevent UserCleanupScheduler from overwhelming streaming (#25519) 2023-06-20 18:04:35 +02:00
Eugen Rochko
4eda233e09
Add webhook templating (#23289)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-06-06 10:42:47 +02:00
Nick Schonning
99e2e9b81f
Fix minor typos in comments and spec names (#21831) 2023-05-19 17:13:29 +02:00
Claire
c3c7d59072
Fix infinite loop in AccountsStatusesCleanupScheduler (#24840) 2023-05-04 10:13:00 +02:00
Claire
a2a22bad23
Fix various edge cases with local moves (#24812) 2023-05-03 19:19:25 +02:00
Claire
c98b012583
Change Move handler to also move list memberships (#24808) 2023-05-03 14:03:38 +02:00
Claire
8b636a29c6
Improve automatic post cleanup worker performances (#24785) 2023-05-03 10:31:40 +02:00
Matt Jankowski
f1c1dd0118
Rename with_lock to with_redis_lock to avoid confusion with ActiveRecord's method (#24741) 2023-05-02 18:16:07 +02:00
Claire
32a030dd74
Rewrite import feature (#21054) 2023-05-02 12:08:48 +02:00
Claire
5dc3173ef8
Fix AccountsStatusesCleanupScheduler not spreading deletes across accounts correctly (#24607) 2023-04-23 22:25:40 +02:00
Claire
501d6197c4
Change automatic post deletion thresholds and load detection (#24614) 2023-04-21 18:14:19 +02:00
Vyr Cossont
d5ad8b6422
Fix Redis client and type errors introduced in #24285 (#24342) 2023-04-01 08:28:35 +02:00
Vyr Cossont
7646ad8a2b
IndexingScheduler: fetch and import in batches (#24285)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-03-31 14:38:47 +02:00
Christian Schmidt
eb38e9df31
Requeue expiration notification (#24311) 2023-03-29 10:52:40 +02:00
Vyr Cossont
94cbd808b5
Webhooks for local status.create, status.update, account.update (#24133) 2023-03-19 07:47:54 +01:00
Claire
1d0ad558ff
Change sidekiq-bulk's batch size from 10,000 to 1,000 jobs in one Redis call (#24034) 2023-03-15 03:45:15 +01:00
Jean byroot Boussier
922837dc96
Upgrade to latest redis-rb 4.x and fix deprecations (#23616)
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
2023-03-04 16:38:28 +01:00
Nick Schonning
b00f945d92
Remove implied StandardError rescue (#23942) 2023-03-03 22:49:16 +01:00
Claire
050f1669c6
Fix original account being unfollowed on migration before the follow request could be sent (#21957) 2023-03-03 21:13:55 +01:00
Claire
ddde4e0d95
Change ActivityPub::DeliveryWorker retries to be spread out more (#21956) 2023-03-03 21:08:22 +01:00
Claire
20b80c62ff
Change auto-deletion throttling constants to better scale with server size (#23320) 2023-02-23 16:52:40 +01:00
Nick Schonning
717683d1c3
Autofix Rubocop remaining Layout rules (#23679) 2023-02-20 06:58:28 +01:00
Nick Schonning
21bf326356
Autofix Rubocop Rails/Pluck (#23730) 2023-02-20 02:28:40 +01:00
Nick Schonning
2177daeae9
Autofix Rubocop Style/RedundantBegin (#23703) 2023-02-19 07:09:40 +09:00
Nick Schonning
a6f77aa28a
Autofix Rubocop Lint/AmbiguousOperatorPrecedence (#23681) 2023-02-18 04:30:23 +01:00
Claire
0c9eac80d8
Fix unbounded recursion in post discovery (#23506)
* Add a limit to how many posts can get fetched as a result of a single request

* Add tests

* Always pass `request_id` when processing `Announce` activities

---------

Co-authored-by: nametoolong <nametoolong@users.noreply.github.com>
2023-02-10 22:16:37 +01:00
Nick Schonning
cec005068f
Apply Rubocop Performance/RedundantBlockCall (#23440) 2023-02-08 00:58:18 +01:00
Nick Schonning
ed570050c6
Autofix Rails/EagerEvaluationLogMessage (#23429)
* Autofix Rails/EagerEvaluationLogMessage

* Update spec for debug block syntax
2023-02-07 03:44:36 +01:00
Claire
9edefc779f
Fix UserCleanupScheduler crash when an unconfirmed account has a moderation note (#23318)
* Fix `UserCleanupScheduler` crash when an unconfirmed account has a moderation note

* Add tests
2023-02-07 01:14:44 +01:00
luzpaz
596923da4a
Fix typos in source documentation (#21046)
Fixed 2 source comment/documentation typos
2022-12-15 15:57:26 +01:00
Eugen Rochko
f8e8e622e5
Change incoming activity processing to happen in ingress queue (#20264) 2022-11-10 14:21:51 +01:00
Claire
5333447be0
Change account deletion requests to spread out over time (#20222) 2022-11-09 14:08:19 +01:00
Eugen Rochko
e98833748e
Fix being able to spoof link verification (#20217)
- Change verification to happen in `default` queue
- Change verification worker to only be queued if there's something to do
- Add `link` tags from metadata fields to page header of profiles
2022-11-09 08:24:21 +01:00
Claire
5925a31b78
Fix followers count not being updated when migrating follows (#19998)
Fixes #19900
2022-11-07 15:38:55 +01:00
Eugen Rochko
d0c9ac3919
Fix indexing scheduler trying to index when Elasticsearch is disabled (#19805)
Fix #19646
2022-11-05 22:31:52 +01:00
Eugen Rochko
5f9e47be34
Add caching for payload serialization during fan-out (#19642) 2022-11-04 13:21:06 +01:00
Claire
6804228fdf
Fix N+1 on mentions in PushUpdateWorker (#19637) 2022-11-01 16:03:51 +01:00
Claire
03d9618595
Fix UserCleanupScheduler crash when an unconfirmed account has a moderation note (#19629)
Fixes #19109
2022-11-01 12:59:23 +01:00
Eugen Rochko
bf0ab3e0fa
Fix vacuum scheduler missing lock, locks never expiring (#19458)
Remove vacuuming of orphaned preview cards
2022-10-26 12:10:48 +02:00
Takeshi Umeda
74ead7d106
Change featured tag updates to add/remove activity (#19409)
* Change featured tag updates to add/remove activity

* Fix to check for the existence of feature tag

* Rename service and worker

* Merge AddHashtagSerializer with AddSerializer

* Undo removal of sidekiq_options
2022-10-22 18:30:55 +02:00
Takeshi Umeda
b0e3f0312c
Add synchronization of remote featured tags (#19380)
* Add LIMIT of featured tag to instance API response

* Add featured_tags_collection_url to Account

* Add synchronization of remote featured tags

* Deliver update activity when updating featured tag

* Remove featured_tags_collection_url

* Revert "Add featured_tags_collection_url to Account"

This reverts commit cff349fc27.

* Add hashtag sync from featured collections

* Fix tag name normalize

* Add target option to fetch featured collection

* Refactor fetch_featured_tags_collection_service

* Add LIMIT of featured tag to v1/instance API response
2022-10-20 09:15:52 +02:00
Eugen Rochko
5c9abdeff1
Add retention policy for cached content and media (#19232) 2022-09-27 03:08:19 +02:00
Claire
8cf7006d4e
Refactor ActivityPub handling to prepare for non-Account actors (#19212)
* Move ActivityPub::FetchRemoteAccountService to ActivityPub::FetchRemoteActorService

ActivityPub::FetchRemoteAccountService is kept as a wrapper for when the actor is
specifically required to be an Account

* Refactor SignatureVerification to allow non-Account actors

* fixup! Move ActivityPub::FetchRemoteAccountService to ActivityPub::FetchRemoteActorService

* Refactor ActivityPub::FetchRemoteKeyService to potentially return non-Account actors

* Refactor inbound ActivityPub payload processing to accept non-Account actors

* Refactor inbound ActivityPub processing to accept activities relayed through non-Account

* Refactor how Account key URIs are built

* Refactor Request and drop unused key_id_format parameter

* Rename ActivityPub::Dereferencer `signature_account` to `signature_actor`
2022-09-21 22:45:57 +02:00
Eugen Rochko
50948b46aa
Add ability to filter followed accounts' posts by language (#19095) 2022-09-20 23:51:21 +02:00
Eugen Rochko
0b3e4fd5de
Remove digest e-mails (#17985)
* Remove digest e-mails

* Remove digest-related code
2022-08-25 23:38:22 +02:00
Eugen Rochko
c3f0621a59
Add ability to follow hashtags (#18809) 2022-07-17 13:49:29 +02:00
Jeremy Kescher
25e076505e
ip_cleanup_scheduler: Make IP and session retention configurable (#18757) 2022-07-07 03:14:28 +02:00
Eugen Rochko
a2871cd747
Add administrative webhooks (#18510)
* Add administrative webhooks

* Fix error when webhook is deleted before delivery worker runs
2022-06-09 21:57:36 +02:00
Jeong Arm
fed7380e9f
Prevent use locale with empty string (#18543)
Somehow user's locale could be an empty string, And empty string itself
are treated as true value.
2022-05-28 14:32:08 +02:00
Eugen Rochko
679b7158e3
Change search indexing to use batches to minimize resource usage (#18451) 2022-05-18 23:29:14 +02:00
Eugen Rochko
6cf57c6765
Refactor how Redis locks are created (#18400)
* Refactor how Redis locks are created

* Fix autorelease duration on account deletion lock
2022-05-13 00:02:35 +02:00
Eugen Rochko
f6d35ed57d
Remove IP matching from e-mail domain blocks (#18190)
Clear out e-mail domain blocks created from automatically resolved DNS records
2022-04-29 23:27:03 +02:00
Eugen Rochko
3917353645
Fix single Redis connection being used across all threads (#18135)
* Fix single Redis connection being used across all Sidekiq threads

* Fix tests
2022-04-28 17:47:34 +02:00
Jeong Arm
3349cbeb3e
Fix clean_discarded_statuses (#17822) 2022-03-19 09:24:08 +01:00
Eugen Rochko
bd53dd5210
Change design of federation pages in admin UI (#17704)
* Change design of federation pages in admin UI

* Fix query performance in instance media attachments measure

* Fix reblogs being included in instance languages dimension
2022-03-09 08:52:32 +01:00
Eugen Rochko
2ea754b861
Fix duplicate notifications being possible after poll expiration (#17697) 2022-03-04 01:06:33 +01:00
Eugen Rochko
50ea54b3ed
Change authorized applications page (#17656)
* Change authorized applications page

* Hide revoke button for superapps and suspended accounts

* Clean up db/schema.rb
2022-03-01 16:48:58 +01:00
Claire
3d60708508
Fix crash in EmailDomainBlockRefreshScheduler (#17649) 2022-02-26 16:28:28 +01:00
Eugen Rochko
27965ce5ed
Add trending statuses (#17431)
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
Eugen Rochko
a29a982eaa
Change e-mail domain blocks to block IPs dynamically (#17635)
* Change e-mail domain blocks to block IPs dynamically

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-24 17:28:23 +01:00
Eugen Rochko
8f03b7a2fb
Add notifications when a reblogged status has been updated (#17404)
* Add notifications when a reblogged status has been updated

* Change wording to say "edit" instead of "update" and add missing controls

* Replace previous update notifications with the most up-to-date one
2022-02-11 22:20:19 +01:00
Claire
da91b18a8b
Fix NoMethodError in StatusUpdateDistributionWorker (#17499)
* Add tests

* Fix NoMethodError in StatusUpdateDistributionWorker

* Fix tests
2022-02-10 14:57:10 +01:00
Eugen Rochko
63002cde03
Add editing for published statuses (#17320)
* Add editing for published statuses

* Fix change of multiple-choice boolean in poll not resetting votes

* Remove the ability to update existing media attachments for now
2022-02-10 00:15:30 +01:00
Eugen Rochko
35850f8195
Fix localization of cold-start follow recommendations (#17479) 2022-02-08 01:53:49 +01:00
Claire
52c1b86964
Fix Ruby 2.5 incompatibility (#17465) 2022-02-07 19:57:06 +01:00
Claire
0d2cf3cd4a
Fix errors when multiple Delete are received for a given actor (#17460) 2022-02-07 13:14:48 +01:00
Claire
03d59340da
Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Eugen Rochko
9eb775a9d1
Fix error when using raw distribution worker (#17334)
Regression from #16697
2022-01-19 23:05:59 +01:00
Eugen Rochko
1060666c58
Add support for editing for published statuses (#16697)
* Add support for editing for published statuses

* Fix references to stripped-out code

* Various fixes and improvements

* Further fixes and improvements

* Fix updates being potentially sent to unauthorized recipients

* Various fixes and improvements

* Fix wrong words in test

* Fix notifying accounts that were tagged but were not in the audience

* Fix mistake
2022-01-19 22:37:27 +01:00
Eugen Rochko
14f436c457
Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Claire
d5c9feb7b7
Add support for private pinned posts (#16954)
* Add support for private pinned toots

* Allow local user to pin private toots

* Change wording to avoid "direct message"
2022-01-17 00:49:55 +01:00
Eugen Rochko
8e84ebf0cb
Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
Claire
7f803c41e2
Add ability to purge undeliverable domains from admin interface (#16686)
* Add ability to purge undeliverable domains from admin interface

* Add tests
2021-12-17 23:01:21 +01:00
Takeshi Umeda
bda8e4f815
Fix follow recommendation biased towards older accounts (#17126) 2021-12-13 23:21:14 +01:00
Eugen Rochko
6e50134a42
Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire
87085a5152
Fix AccountNote not having a maximum length (#16942) 2021-11-06 00:12:25 +01:00
Claire
5efb1ff337
Fix followers synchronization mechanism not working when URI has empty path (#16510)
* Fix followers synchronization mechanism not working when URI has empty path

To my knowledge, there is no current implementation on the fediverse
that can use bare domains (e.g., actor is at https://example.org instead of
something like https://example.org/actor) that also plans to support the
followers synchronization mechanism. However, Mastodon's current implementation
would exclude such accounts from followers list.

Also adds tests and rename them to reflect the proper method names.

* Move url prefix regexp to its own constant
2021-08-11 17:48:42 +02:00
Claire
4ac78e2a06
Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Takeshi Umeda
590e1578bf
Fix when MoveWorker cannot get locale from remote account (#16576) 2021-08-08 15:31:02 +02:00
Claire
1c21dcfa35
Fix newlines in accout notes added by the Move handler (#16415)
* Fix newlines in account notes added by the move handler

* Make MoveWorker more robust
2021-06-23 23:55:47 +02:00
Eugen Rochko
d174d12c83
Add authentication history (#16408) 2021-06-21 17:07:30 +02:00