mastodon/app/models
Claire 02851848e9
Revamp post filtering system (#18058)
* Add model for custom filter keywords

* Use CustomFilterKeyword internally

Does not change the API

* Fix /filters/edit and /filters/new

* Add migration tests

* Remove whole_word column from custom_filters (covered by custom_filter_keywords)

* Redesign /filters

Instead of a list, present a card that displays more information and handles
multiple keywords per filter.

* Redesign /filters/new and /filters/edit to add and remove keywords

This adds a new gem dependency: cocoon, as well as a npm dependency:
cocoon-js-vanilla. Those are used to easily populate and remove form fields
from the user interface when manipulating multiple keyword filters at once.

* Add /api/v2/filters to edit filter with multiple keywords

Entities:
- `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context`
  `keywords`
- `FilterKeyword`: `id`, `keyword`, `whole_word`

API endpoits:
- `GET /api/v2/filters` to list filters (including keywords)
- `POST /api/v2/filters` to create a new filter
  `keywords_attributes` can also be passed to create keywords in one request
- `GET /api/v2/filters/:id` to read a particular filter
- `PUT /api/v2/filters/:id` to update a new filter
  `keywords_attributes` can also be passed to edit, delete or add keywords in
   one request
- `DELETE /api/v2/filters/:id` to delete a particular filter
- `GET /api/v2/filters/:id/keywords` to list keywords for a filter
- `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a
   filter
- `GET /api/v2/filter_keywords/:id` to read a particular keyword
- `PUT /api/v2/filter_keywords/:id` to edit a particular keyword
- `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword

* Change from `irreversible` boolean to `action` enum

* Remove irrelevent `irreversible_must_be_within_context` check

* Fix /filters/new and /filters/edit with update for filter_action

* Fix Rubocop/Codeclimate complaining about task names

* Refactor FeedManager#phrase_filtered?

This moves regexp building and filter caching to the `CustomFilter` class.

This does not change the functional behavior yet, but this changes how the
cache is built, doing per-custom_filter regexps so that filters can be matched
independently, while still offering caching.

* Perform server-side filtering and output result in REST API

* Fix numerous filters_changed events being sent when editing multiple keywords at once

* Add some tests

* Use the new API in the WebUI

- use client-side logic for filters we have fetched rules for.
  This is so that filter changes can be retroactively applied without
  reloading the UI.
- use server-side logic for filters we haven't fetched rules for yet
  (e.g. network error, or initial timeline loading)

* Minor optimizations and refactoring

* Perform server-side filtering on the streaming server

* Change the wording of filter action labels

* Fix issues pointed out by linter

* Change design of “Show anyway” link in accordence to review comments

* Drop “irreversible” filtering behavior

* Move /api/v2/filter_keywords to /api/v1/filters/keywords

* Rename `filter_results` attribute to `filtered`

* Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer

* Fix systemChannelId value in streaming server

* Simplify code by removing client-side filtering code

The simplifcation comes at a cost though: filters aren't retroactively
applied anymore.
2022-06-28 09:42:13 +02:00
..
account_suggestions Fix single Redis connection being used across all threads (#18135) 2022-04-28 17:47:34 +02:00
admin Add administrative webhooks (#18510) 2022-06-09 21:57:36 +02:00
concerns Revamp post filtering system (#18058) 2022-06-28 09:42:13 +02:00
form Add trending statuses (#17431) 2022-02-25 00:34:14 +01:00
trends Change algorithm of tootctl search deploy to improve performance (#18463) 2022-05-22 22:16:43 +02:00
web Add policy param to POST /api/v1/push/subscriptions (#16040) 2021-04-15 05:00:25 +02:00
account.rb Remove dependency on running Redis server for db:setup (#18560) 2022-06-01 19:23:31 +02:00
account_alias.rb Fix crash in alias settings page (#18004) 2022-04-09 20:11:06 +02:00
account_conversation.rb Fix single Redis connection being used across all threads (#18135) 2022-04-28 17:47:34 +02:00
account_deletion_request.rb Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
account_domain_block.rb Fix crash when saving invalid domain name (#11528) 2019-08-08 23:04:19 +02:00
account_filter.rb Fix /api/v1/admin/accounts (#17887) 2022-03-28 23:57:38 +02:00
account_migration.rb Refactor how Redis locks are created (#18400) 2022-05-13 00:02:35 +02:00
account_moderation_note.rb Update dependencies for Ruby (2018-04-23) (#7237) 2018-04-23 11:29:17 +02:00
account_note.rb Fix AccountNote not having a maximum length (#16942) 2021-11-06 00:12:25 +01:00
account_pin.rb Add API endpoint to list featured accounts (fixes #8315) (#8317) 2018-08-20 18:46:04 +02:00
account_stat.rb Fix follower and other counters being able to go negative (#18517) 2022-05-26 20:32:48 +02:00
account_statuses_cleanup_policy.rb Add option to automatically delete toots after one week (#17691) 2022-03-03 16:18:58 +01:00
account_statuses_filter.rb Fix performance of account timelines (#17709) 2022-03-08 09:14:39 +01:00
account_suggestions.rb Change auto-following admin-selected accounts, show in recommendations (#16078) 2021-04-24 17:01:43 +02:00
account_summary.rb Fix FollowRecommendationsScheduler failing because of unpopulated views (#16189) 2021-05-09 10:39:29 +02:00
account_warning.rb Add ability to mark statuses as sensitive from reports in admin UI (#17668) 2022-03-01 22:20:29 +01:00
account_warning_preset.rb Add titles to warning presets in admin UI (#13252) 2020-03-12 17:57:59 +01:00
admin.rb Add logging of admin actions (#5757) 2017-11-24 02:05:53 +01:00
announcement.rb Change order of announcements in admin page to sort them newest-first (#15091) 2020-11-04 21:15:22 +01:00
announcement_filter.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
announcement_mute.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
announcement_reaction.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
appeal.rb Fix issues when attempting to appeal an old strike (#17554) 2022-02-16 22:29:48 +01:00
application_record.rb Fix records not being indexed sometimes (#12024) 2019-10-01 01:19:11 +02:00
backup.rb Add announcements (#12662) 2020-01-23 22:00:13 +01:00
block.rb Store URIs of follows, follow requests and blocks for ActivityPub (#7160) 2018-05-04 21:14:34 +02:00
bookmark.rb Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915) 2021-11-18 22:02:08 +01:00
canonical_email_block.rb Add ability for admins to delete canonical email blocks (#16644) 2021-12-17 23:02:14 +01:00
context.rb Refactor JSON templates to be generated with ActiveModelSerializers instead of Rabl (#4090) 2017-07-07 04:02:06 +02:00
conversation.rb Revert "Remove conversation URI (#11423)" (#11424) 2019-07-28 17:47:37 +02:00
conversation_mute.rb Update dependencies for Ruby (2018-04-23) (#7237) 2018-04-23 11:29:17 +02:00
custom_emoji.rb Add support for webp uploads (#18506) 2022-05-27 20:06:40 +02:00
custom_emoji_category.rb Add batch actions and categories to admin UI for custom emojis (#11793) 2019-09-09 22:44:17 +02:00
custom_emoji_filter.rb Various fixes and improvements (#12878) 2020-01-20 15:55:03 +01:00
custom_filter.rb Revamp post filtering system (#18058) 2022-06-28 09:42:13 +02:00
custom_filter_keyword.rb Revamp post filtering system (#18058) 2022-06-28 09:42:13 +02:00
device.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
domain_allow.rb Add /api/v1/admin/domain_allows (#18668) 2022-06-23 23:12:01 +02:00
domain_block.rb Add /api/v1/admin/domain_blocks (#18247) 2022-06-01 17:31:36 +02:00
email_domain_block.rb Remove IP matching from e-mail domain blocks (#18190) 2022-04-29 23:27:03 +02:00
encrypted_message.rb Fix single Redis connection being used across all threads (#18135) 2022-04-28 17:47:34 +02:00
export.rb Add import/export feature for bookmarks (#14956) 2020-11-19 17:48:13 +01:00
favourite.rb Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915) 2021-11-18 22:02:08 +01:00
featured_tag.rb Fix data integrity of featured tags (#17712) 2022-03-09 08:51:12 +01:00
feed.rb allow pagination by min_id and max_id (#14776) 2020-09-12 17:09:49 +02:00
follow.rb Fix being able to import more than allowed number of follows (#15384) 2020-12-26 23:52:46 +01:00
follow_recommendation.rb Fix FollowRecommendationsScheduler failing because of unpopulated views (#16189) 2021-05-09 10:39:29 +02:00
follow_recommendation_filter.rb Fix single Redis connection being used across all threads (#18135) 2022-04-28 17:47:34 +02:00
follow_recommendation_suppression.rb Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
follow_request.rb Fix edge case where accepted follow cannot be processed because of follow limit (#16098) 2021-04-23 22:51:21 +02:00
home_feed.rb Fix rubocop config and warnings (#15503) 2021-01-07 09:40:55 +01:00
identity.rb Change identities id column to a bigint (#9371) 2018-11-27 13:56:25 +01:00
import.rb Fix follow limit preventing re-following of a moved account (#14207) 2020-12-18 09:18:31 +01:00
instance.rb Fix nil error when viewing suspended domain in admin UI (#17765) 2022-03-14 05:27:37 +01:00
instance_filter.rb Change design of federation pages in admin UI (#17704) 2022-03-09 08:52:32 +01:00
invite.rb Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
invite_filter.rb Various fixes and improvements (#12878) 2020-01-20 15:55:03 +01:00
ip_block.rb Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
list.rb Improve account deletion performances further (#15407) 2020-12-22 23:57:46 +01:00
list_account.rb Add abilityto add oneself to lists (#12271) 2019-11-04 13:02:01 +01:00
list_feed.rb Fix rubocop config and warnings (#15503) 2021-01-07 09:40:55 +01:00
login_activity.rb Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
marker.rb Add timeline read markers API (#11762) 2019-09-06 13:55:51 +02:00
media_attachment.rb Add support for webp uploads (#18506) 2022-05-27 20:06:40 +02:00
mention.rb Improve support for aspects/circles (#8950) 2018-10-17 17:13:04 +02:00
message_franking.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
mute.rb Add duration parameter to muting. (#13831) 2020-10-13 01:01:14 +02:00
notification.rb Add notifications for new reports (#18697) 2022-06-27 09:30:15 +02:00
one_time_key.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
poll.rb Fix regression in tootctl search deploy caused by unloaded attribute (#18514) 2022-05-26 18:05:47 +02:00
poll_vote.rb Add optimistic lock to avoid race conditions when handling votes (#10196) 2019-03-06 19:53:57 +01:00
preview_card.rb Add support for webp uploads (#18506) 2022-05-27 20:06:40 +02:00
preview_card_provider.rb Add trending links (#16917) 2021-11-25 13:07:38 +01:00
public_feed.rb Fix rubocop config and warnings (#15503) 2021-01-07 09:40:55 +01:00
relationship_filter.rb Fix followings list order | Issue #13538 (#13676) 2020-05-08 20:17:16 +02:00
relay.rb Fix enable/disable relay failures (#13535) 2020-04-23 22:04:18 +02:00
remote_follow.rb Remove dependency on goldfinger gem (#14919) 2020-10-08 00:34:57 +02:00
report.rb Add administrative webhooks (#18510) 2022-06-09 21:57:36 +02:00
report_filter.rb Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
report_note.rb Fix scope latest of ReportNote (#9630) 2018-12-26 06:38:59 +01:00
rule.rb Add server rules (#15769) 2021-02-21 19:50:12 +01:00
scheduled_status.rb Fix deleting a scheduled status immediately deleting media attachments (#9728) 2019-01-06 16:38:40 +01:00
search.rb Refactor JSON templates to be generated with ActiveModelSerializers instead of Rabl (#4090) 2017-07-07 04:02:06 +02:00
session_activation.rb Add Ruby 3.0 support (#16046) 2021-05-06 14:22:54 +02:00
setting.rb Use Rails' index_by where it makes sense (#15542) 2021-01-12 09:27:38 +01:00
site_upload.rb Improvements to image upload validation and creation (#10431) 2019-04-01 07:30:46 +02:00
status.rb Fix unnecessary query on status creation (#17901) 2022-05-26 00:20:30 +02:00
status_edit.rb Add polls and media attachments to edit comparison modal in web UI (#17727) 2022-03-09 21:15:24 +01:00
status_pin.rb Add feature to automatically delete old toots (#16529) 2021-08-09 23:11:50 +02:00
status_stat.rb Fix follower and other counters being able to go negative (#18517) 2022-05-26 20:32:48 +02:00
system_key.rb Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
tag.rb Fix error on trending hashtags/links pages in admin UI due to missing constant (#17044) 2021-11-26 01:12:39 +01:00
tag_feed.rb Fix rubocop config and warnings (#15503) 2021-01-07 09:40:55 +01:00
tombstone.rb Record deleted(by mod) status to prevent re-appear (#10732) 2019-05-09 22:03:02 +02:00
trends.rb Add trending statuses (#17431) 2022-02-25 00:34:14 +01:00
unavailable_domain.rb Fix performance on instances list in admin UI (#15282) 2020-12-14 09:06:34 +01:00
user.rb Add administrative webhooks (#18510) 2022-06-09 21:57:36 +02:00
user_invite_request.rb Add "why do you want to join" field to invite requests (#10524) 2019-04-09 23:06:30 +09:00
user_ip.rb Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
web.rb Add extended about page stub 2017-01-13 03:24:41 +01:00
webauthn_credential.rb Fix validates :sign_count of WebauthnCredential (#14806) 2020-09-16 20:16:46 +02:00
webhook.rb Add administrative webhooks (#18510) 2022-06-09 21:57:36 +02:00