Merge remote-tracking branch 'remotes/origin/develop' into media-preview-proxy-nostream

This commit is contained in:
Ivan Tashkinov 2020-09-15 20:29:39 +03:00
commit 2f155fb89f
17 changed files with 205 additions and 29 deletions

View File

@ -9,15 +9,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Renamed `:await_up_timeout` in `:connections_pool` namespace to `:connect_timeout`, old name is deprecated. - Renamed `:await_up_timeout` in `:connections_pool` namespace to `:connect_timeout`, old name is deprecated.
- Renamed `:timeout` in `pools` namespace to `:recv_timeout`, old name is deprecated. - Renamed `:timeout` in `pools` namespace to `:recv_timeout`, old name is deprecated.
- Minimum lifetime for ephmeral activities changed to 10 minutes and made configurable (`:min_lifetime` option).
### Removed ### Removed
- **Breaking:** `Pleroma.Workers.Cron.StatsWorker` setting from Oban `:crontab` (moved to a simpler implementation). - **Breaking:** `Pleroma.Workers.Cron.StatsWorker` setting from Oban `:crontab` (moved to a simpler implementation).
- **Breaking:** `Pleroma.Workers.Cron.ClearOauthTokenWorker` setting from Oban `:crontab` (moved to scheduled jobs). - **Breaking:** `Pleroma.Workers.Cron.ClearOauthTokenWorker` setting from Oban `:crontab` (moved to scheduled jobs).
- **Breaking:** `Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker` setting from Oban `:crontab` (moved to scheduled jobs). - **Breaking:** `Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker` setting from Oban `:crontab` (moved to scheduled jobs).
- Removed `:managed_config` option. In practice, it was accidentally removed with 2.0.0 release when frontends were
switched to a new configuration mechanism, however it was not officially removed until now.
### Changed ## unreleased-patch - ???
- Minimum lifetime for ephmeral activities changed to 10 minutes and made configurable (`:min_lifetime` option).
### Fixed
- Welcome Chat messages preventing user registration with MRF Simple Policy applied to the local instance
- Mastodon API: the public timeline returning an error when the `reply_visibility` parameter is set to `self` for an unauthenticated user
## [2.1.1] - 2020-09-08 ## [2.1.1] - 2020-09-08

View File

@ -216,7 +216,6 @@
allow_relay: true, allow_relay: true,
public: true, public: true,
quarantined_instances: [], quarantined_instances: [],
managed_config: true,
static_dir: "instance/static/", static_dir: "instance/static/",
allowed_post_formats: [ allowed_post_formats: [
"text/plain", "text/plain",

View File

@ -764,12 +764,6 @@
"*.quarantined.com" "*.quarantined.com"
] ]
}, },
%{
key: :managed_config,
type: :boolean,
description:
"Whenether the config for pleroma-fe is configured in this config or in static/config.json"
},
%{ %{
key: :static_dir, key: :static_dir,
type: :string, type: :string,

View File

@ -18,7 +18,7 @@ To add configuration to your config file, you can copy it from the base config.
* `notify_email`: Email used for notifications. * `notify_email`: Email used for notifications.
* `description`: The instances description, can be seen in nodeinfo and ``/api/v1/instance``. * `description`: The instances description, can be seen in nodeinfo and ``/api/v1/instance``.
* `limit`: Posts character limit (CW/Subject included in the counter). * `limit`: Posts character limit (CW/Subject included in the counter).
* `discription_limit`: The character limit for image descriptions. * `description_limit`: The character limit for image descriptions.
* `chat_limit`: Character limit of the instance chat messages. * `chat_limit`: Character limit of the instance chat messages.
* `remote_limit`: Hard character limit beyond which remote posts will be dropped. * `remote_limit`: Hard character limit beyond which remote posts will be dropped.
* `upload_limit`: File size limit of uploads (except for avatar, background, banner). * `upload_limit`: File size limit of uploads (except for avatar, background, banner).
@ -40,7 +40,6 @@ To add configuration to your config file, you can copy it from the base config.
* `allow_relay`: Enable Pleromas Relay, which makes it possible to follow a whole instance. * `allow_relay`: Enable Pleromas Relay, which makes it possible to follow a whole instance.
* `public`: Makes the client API in authenticated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network. Note that there is a dependent setting restricting or allowing unauthenticated access to specific resources, see `restrict_unauthenticated` for more details. * `public`: Makes the client API in authenticated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network. Note that there is a dependent setting restricting or allowing unauthenticated access to specific resources, see `restrict_unauthenticated` for more details.
* `quarantined_instances`: List of ActivityPub instances where private (DMs, followers-only) activities will not be send. * `quarantined_instances`: List of ActivityPub instances where private (DMs, followers-only) activities will not be send.
* `managed_config`: Whenether the config for pleroma-fe is configured in [:frontend_configurations](#frontend_configurations) or in ``static/config.json``.
* `allowed_post_formats`: MIME-type list of formats allowed to be posted (transformed into HTML). * `allowed_post_formats`: MIME-type list of formats allowed to be posted (transformed into HTML).
* `extended_nickname_format`: Set to `true` to use extended local nicknames format (allows underscores/dashes). This will break federation with * `extended_nickname_format`: Set to `true` to use extended local nicknames format (allows underscores/dashes). This will break federation with
older software for theses nicknames. older software for theses nicknames.

View File

@ -767,7 +767,7 @@ defp restrict_replies(query, %{exclude_replies: true}) do
end end
defp restrict_replies(query, %{ defp restrict_replies(query, %{
reply_filtering_user: user, reply_filtering_user: %User{} = user,
reply_visibility: "self" reply_visibility: "self"
}) do }) do
from( from(
@ -783,7 +783,7 @@ defp restrict_replies(query, %{
end end
defp restrict_replies(query, %{ defp restrict_replies(query, %{
reply_filtering_user: user, reply_filtering_user: %User{} = user,
reply_visibility: "following" reply_visibility: "following"
}) do }) do
from( from(

View File

@ -66,7 +66,8 @@ defp check_media_nsfw(
"type" => "Create", "type" => "Create",
"object" => child_object "object" => child_object
} = object } = object
) do )
when is_map(child_object) do
media_nsfw = media_nsfw =
Config.get([:mrf_simple, :media_nsfw]) Config.get([:mrf_simple, :media_nsfw])
|> MRF.subdomains_regex() |> MRF.subdomains_regex()

View File

@ -309,7 +309,7 @@ def fix_url(object), do: object
def fix_emoji(%{"tag" => tags} = object) when is_list(tags) do def fix_emoji(%{"tag" => tags} = object) when is_list(tags) do
emoji = emoji =
tags tags
|> Enum.filter(fn data -> data["type"] == "Emoji" and data["icon"] end) |> Enum.filter(fn data -> is_map(data) and data["type"] == "Emoji" and data["icon"] end)
|> Enum.reduce(%{}, fn data, mapping -> |> Enum.reduce(%{}, fn data, mapping ->
name = String.trim(data["name"], ":") name = String.trim(data["name"], ":")

View File

@ -87,6 +87,50 @@ def perform(:fetch, %Activity{} = activity) do
def rich_media_get(url) do def rich_media_get(url) do
headers = [{"user-agent", Pleroma.Application.user_agent() <> "; Bot"}] headers = [{"user-agent", Pleroma.Application.user_agent() <> "; Bot"}]
Pleroma.HTTP.get(url, headers, @options) head_check =
case Pleroma.HTTP.head(url, headers, @options) do
# If the HEAD request didn't reach the server for whatever reason,
# we assume the GET that comes right after won't either
{:error, _} = e ->
e
{:ok, %Tesla.Env{status: 200, headers: headers}} ->
with :ok <- check_content_type(headers),
:ok <- check_content_length(headers),
do: :ok
_ ->
:ok
end
with :ok <- head_check, do: Pleroma.HTTP.get(url, headers, @options)
end
defp check_content_type(headers) do
case List.keyfind(headers, "content-type", 0) do
{_, content_type} ->
case Plug.Conn.Utils.media_type(content_type) do
{:ok, "text", "html", _} -> :ok
_ -> {:error, {:content_type, content_type}}
end
_ ->
:ok
end
end
@max_body @options[:max_body]
defp check_content_length(headers) do
case List.keyfind(headers, "content-length", 0) do
{_, maybe_content_length} ->
case Integer.parse(maybe_content_length) do
{content_length, ""} when content_length <= @max_body -> :ok
{_, ""} -> {:error, :body_too_large}
_ -> :ok
end
_ ->
:ok
end
end end
end end

View File

@ -36,6 +36,14 @@ defp get_cached_or_parse(url) do
{:ok, _data} = res -> {:ok, _data} = res ->
res res
{:error, :body_too_large} = e ->
e
{:error, {:content_type, _}} = e ->
e
# The TTL is not set for the errors above, since they are unlikely to change
# with time
{:error, _} = e -> {:error, _} = e ->
ttl = Pleroma.Config.get([:rich_media, :failure_backoff], 60_000) ttl = Pleroma.Config.get([:rich_media, :failure_backoff], 60_000)
Cachex.expire(:rich_media_cache, url, ttl) Cachex.expire(:rich_media_cache, url, ttl)

View File

@ -3,7 +3,7 @@ msgstr ""
"Project-Id-Version: PACKAGE VERSION\n" "Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n" "Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2020-09-09 09:49+0000\n" "POT-Creation-Date: 2020-09-09 09:49+0000\n"
"PO-Revision-Date: 2020-09-09 10:52+0000\n" "PO-Revision-Date: 2020-09-11 21:26+0000\n"
"Last-Translator: tarteka <info@tarteka.net>\n" "Last-Translator: tarteka <info@tarteka.net>\n"
"Language-Team: Spanish <https://translate.pleroma.social/projects/pleroma/" "Language-Team: Spanish <https://translate.pleroma.social/projects/pleroma/"
"pleroma/es/>\n" "pleroma/es/>\n"
@ -94,52 +94,52 @@ msgid "must be less than %{number}"
msgstr "" msgstr ""
msgid "must be greater than %{number}" msgid "must be greater than %{number}"
msgstr "" msgstr "debe ser mayor que %{number}"
msgid "must be less than or equal to %{number}" msgid "must be less than or equal to %{number}"
msgstr "" msgstr "debe ser menor o igual que %{number}"
msgid "must be greater than or equal to %{number}" msgid "must be greater than or equal to %{number}"
msgstr "" msgstr "deber ser mayor o igual que %{number}"
msgid "must be equal to %{number}" msgid "must be equal to %{number}"
msgstr "" msgstr "deber ser igual a %{number}"
#: lib/pleroma/web/common_api/common_api.ex:505 #: lib/pleroma/web/common_api/common_api.ex:505
#, elixir-format #, elixir-format
msgid "Account not found" msgid "Account not found"
msgstr "" msgstr "Cuenta no encontrada"
#: lib/pleroma/web/common_api/common_api.ex:339 #: lib/pleroma/web/common_api/common_api.ex:339
#, elixir-format #, elixir-format
msgid "Already voted" msgid "Already voted"
msgstr "" msgstr "Ya has votado"
#: lib/pleroma/web/oauth/oauth_controller.ex:359 #: lib/pleroma/web/oauth/oauth_controller.ex:359
#, elixir-format #, elixir-format
msgid "Bad request" msgid "Bad request"
msgstr "" msgstr "Solicitud incorrecta"
#: lib/pleroma/web/activity_pub/activity_pub_controller.ex:426 #: lib/pleroma/web/activity_pub/activity_pub_controller.ex:426
#, elixir-format #, elixir-format
msgid "Can't delete object" msgid "Can't delete object"
msgstr "" msgstr "No se puede eliminar el objeto"
#: lib/pleroma/web/controller_helper.ex:105 #: lib/pleroma/web/controller_helper.ex:105
#: lib/pleroma/web/controller_helper.ex:111 #: lib/pleroma/web/controller_helper.ex:111
#, elixir-format #, elixir-format
msgid "Can't display this activity" msgid "Can't display this activity"
msgstr "" msgstr "No se puede mostrar esta actividad"
#: lib/pleroma/web/mastodon_api/controllers/account_controller.ex:285 #: lib/pleroma/web/mastodon_api/controllers/account_controller.ex:285
#, elixir-format #, elixir-format
msgid "Can't find user" msgid "Can't find user"
msgstr "" msgstr "No se puede encontrar al usuario"
#: lib/pleroma/web/pleroma_api/controllers/account_controller.ex:61 #: lib/pleroma/web/pleroma_api/controllers/account_controller.ex:61
#, elixir-format #, elixir-format
msgid "Can't get favorites" msgid "Can't get favorites"
msgstr "" msgstr "No se puede obtener los favoritos"
#: lib/pleroma/web/activity_pub/activity_pub_controller.ex:438 #: lib/pleroma/web/activity_pub/activity_pub_controller.ex:438
#, elixir-format #, elixir-format
@ -149,7 +149,7 @@ msgstr ""
#: lib/pleroma/web/common_api/utils.ex:563 #: lib/pleroma/web/common_api/utils.ex:563
#, elixir-format #, elixir-format
msgid "Cannot post an empty status without attachments" msgid "Cannot post an empty status without attachments"
msgstr "" msgstr "No se puede publicar un estado vacío y sin archivos adjuntos"
#: lib/pleroma/web/common_api/utils.ex:511 #: lib/pleroma/web/common_api/utils.ex:511
#, elixir-format #, elixir-format

View File

@ -4,6 +4,8 @@ defmodule Pleroma.Repo.Migrations.MoveActivityExpirationsToOban do
import Ecto.Query, only: [from: 2] import Ecto.Query, only: [from: 2]
def change do def change do
Pleroma.Config.Oban.warn()
Supervisor.start_link([{Oban, Pleroma.Config.get(Oban)}], Supervisor.start_link([{Oban, Pleroma.Config.get(Oban)}],
strategy: :one_for_one, strategy: :one_for_one,
name: Pleroma.Supervisor name: Pleroma.Supervisor

View File

@ -4,6 +4,8 @@ defmodule Pleroma.Repo.Migrations.MoveTokensExpirationIntoOban do
import Ecto.Query, only: [from: 2] import Ecto.Query, only: [from: 2]
def change do def change do
Pleroma.Config.Oban.warn()
Supervisor.start_link([{Oban, Pleroma.Config.get(Oban)}], Supervisor.start_link([{Oban, Pleroma.Config.get(Oban)}],
strategy: :one_for_one, strategy: :one_for_one,
name: Pleroma.Supervisor name: Pleroma.Supervisor

View File

@ -0,0 +1,27 @@
defmodule Pleroma.Repo.Migrations.RemoveManagedConfigFromDb do
use Ecto.Migration
import Ecto.Query
alias Pleroma.ConfigDB
alias Pleroma.Repo
def up do
config_entry =
from(c in ConfigDB,
select: [:id, :value],
where: c.group == ^:pleroma and c.key == ^:instance
)
|> Repo.one()
if config_entry do
{_, value} = Keyword.pop(config_entry.value, :managed_config)
config_entry
|> Ecto.Changeset.change(value: value)
|> Repo.update()
end
end
def down do
:ok
end
end

View File

@ -1262,4 +1262,21 @@ def post(url, query, body, headers) do
inspect(headers) inspect(headers)
}"} }"}
end end
# Most of the rich media mocks are missing HEAD requests, so we just return 404.
@rich_media_mocks [
"https://example.com/ogp",
"https://example.com/ogp-missing-data",
"https://example.com/twitter-card"
]
def head(url, _query, _body, _headers) when url in @rich_media_mocks do
{:ok, %Tesla.Env{status: 404, body: ""}}
end
def head(url, query, body, headers) do
{:error,
"Mock response not implemented for HEAD #{inspect(url)}, #{query}, #{inspect(body)}, #{
inspect(headers)
}"}
end
end end

View File

@ -440,6 +440,45 @@ test "it sends a welcome chat message if it is set" do
assert activity.actor == welcome_user.ap_id assert activity.actor == welcome_user.ap_id
end end
setup do:
clear_config(:mrf_simple,
media_removal: [],
media_nsfw: [],
federated_timeline_removal: [],
report_removal: [],
reject: [],
followers_only: [],
accept: [],
avatar_removal: [],
banner_removal: [],
reject_deletes: []
)
setup do:
clear_config(:mrf,
policies: [
Pleroma.Web.ActivityPub.MRF.SimplePolicy
]
)
test "it sends a welcome chat message when Simple policy applied to local instance" do
Pleroma.Config.put([:mrf_simple, :media_nsfw], ["localhost"])
welcome_user = insert(:user)
Pleroma.Config.put([:welcome, :chat_message, :enabled], true)
Pleroma.Config.put([:welcome, :chat_message, :sender_nickname], welcome_user.nickname)
Pleroma.Config.put([:welcome, :chat_message, :message], "Hello, this is a chat message")
cng = User.register_changeset(%User{}, @full_user_data)
{:ok, registered_user} = User.register(cng)
ObanHelpers.perform_all()
activity = Repo.one(Pleroma.Activity)
assert registered_user.ap_id in activity.recipients
assert Object.normalize(activity).data["content"] =~ "chat message"
assert activity.actor == welcome_user.ap_id
end
test "it sends a welcome email message if it is set" do test "it sends a welcome email message if it is set" do
welcome_user = insert(:user) welcome_user = insert(:user)
Pleroma.Config.put([:welcome, :email, :enabled], true) Pleroma.Config.put([:welcome, :email, :enabled], true)

View File

@ -1810,6 +1810,14 @@ test "public timeline with default reply_visibility `self`", %{users: %{u1: user
|> Enum.map(& &1.id) |> Enum.map(& &1.id)
assert activities_ids == [] assert activities_ids == []
activities_ids =
%{}
|> Map.put(:reply_visibility, "self")
|> Map.put(:reply_filtering_user, nil)
|> ActivityPub.fetch_public_activities()
assert activities_ids == []
end end
test "home timeline", %{users: %{u1: user}} do test "home timeline", %{users: %{u1: user}} do

View File

@ -56,6 +56,27 @@ defmodule Pleroma.Web.RichMedia.ParserTest do
%{method: :get, url: "http://example.com/error"} -> %{method: :get, url: "http://example.com/error"} ->
{:error, :overload} {:error, :overload}
%{
method: :head,
url: "http://example.com/huge-page"
} ->
%Tesla.Env{
status: 200,
headers: [{"content-length", "2000001"}, {"content-type", "text/html"}]
}
%{
method: :head,
url: "http://example.com/pdf-file"
} ->
%Tesla.Env{
status: 200,
headers: [{"content-length", "1000000"}, {"content-type", "application/pdf"}]
}
%{method: :head} ->
%Tesla.Env{status: 404, body: "", headers: []}
end) end)
:ok :ok
@ -144,4 +165,12 @@ test "rejects invalid OGP data" do
test "returns error if getting page was not successful" do test "returns error if getting page was not successful" do
assert {:error, :overload} = Parser.parse("http://example.com/error") assert {:error, :overload} = Parser.parse("http://example.com/error")
end end
test "does a HEAD request to check if the body is too large" do
assert {:error, :body_too_large} = Parser.parse("http://example.com/huge-page")
end
test "does a HEAD request to check if the body is html" do
assert {:error, {:content_type, _}} = Parser.parse("http://example.com/pdf-file")
end
end end