Merge branch 'develop' into 'feat/floki-fast-html-2'

# Conflicts:
#   config/config.exs
This commit is contained in:
lain 2020-08-09 10:53:58 +00:00
commit d0fc48ea67
10 changed files with 137 additions and 86 deletions

View File

@ -6,6 +6,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [unreleased] ## [unreleased]
### Changed ### Changed
- **Breaking:** Added the ObjectAgePolicy to the default set of MRFs. This will delist and strip the follower collection of any message received that is older than 7 days. This will stop users from seeing very old messages in the timelines. The messages can still be viewed on the user's page and in conversations. They also still trigger notifications.
- **Breaking:** Elixir >=1.9 is now required (was >= 1.8) - **Breaking:** Elixir >=1.9 is now required (was >= 1.8)
- **Breaking:** Configuration: `:auto_linker, :opts` moved to `:pleroma, Pleroma.Formatter`. Old config namespace is deprecated. - **Breaking:** Configuration: `:auto_linker, :opts` moved to `:pleroma, Pleroma.Formatter`. Old config namespace is deprecated.
- In Conversations, return only direct messages as `last_status` - In Conversations, return only direct messages as `last_status`
@ -15,6 +16,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Configuration: `:instance, rewrite_policy` moved to `:mrf, policies`, `:instance, :mrf_transparency` moved to `:mrf, :transparency`, `:instance, :mrf_transparency_exclusions` moved to `:mrf, :transparency_exclusions`. Old config namespace is deprecated. - Configuration: `:instance, rewrite_policy` moved to `:mrf, policies`, `:instance, :mrf_transparency` moved to `:mrf, :transparency`, `:instance, :mrf_transparency_exclusions` moved to `:mrf, :transparency_exclusions`. Old config namespace is deprecated.
- Configuration: `:media_proxy, whitelist` format changed to host with scheme (e.g. `http://example.com` instead of `example.com`). Domain format is deprecated. - Configuration: `:media_proxy, whitelist` format changed to host with scheme (e.g. `http://example.com` instead of `example.com`). Domain format is deprecated.
- **Breaking:** Configuration: `:instance, welcome_user_nickname` moved to `:welcome, :direct_message, :sender_nickname`, `:instance, :welcome_message` moved to `:welcome, :direct_message, :message`. Old config namespace is deprecated. - **Breaking:** Configuration: `:instance, welcome_user_nickname` moved to `:welcome, :direct_message, :sender_nickname`, `:instance, :welcome_message` moved to `:welcome, :direct_message, :message`. Old config namespace is deprecated.
- **Breaking:** LDAP: Fallback to local database authentication has been removed for security reasons and lack of a mechanism to ensure the passwords are synchronized when LDAP passwords are updated.
<details> <details>
<summary>API Changes</summary> <summary>API Changes</summary>

View File

@ -515,7 +515,13 @@
"user-search", "user-search",
"user_exists", "user_exists",
"users", "users",
"web" "web",
"verify_credentials",
"update_credentials",
"relationships",
"search",
"confirmation_resend",
"mfa"
], ],
email_blacklist: [] email_blacklist: []
@ -739,6 +745,8 @@
config :floki, :html_parser, Floki.HTMLParser.FastHtml config :floki, :html_parser, Floki.HTMLParser.FastHtml
config :pleroma, Pleroma.Web.Auth.Authenticator, Pleroma.Web.Auth.PleromaAuthenticator
# Import environment specific config. This must remain at the bottom # Import environment specific config. This must remain at the bottom
# of this file so it overrides the configuration defined above. # of this file so it overrides the configuration defined above.
import_config "#{Mix.env()}.exs" import_config "#{Mix.env()}.exs"

View File

@ -98,3 +98,13 @@ but should only be run if necessary. **It is safe to cancel this.**
```sh tab="From Source" ```sh tab="From Source"
mix pleroma.database vacuum full mix pleroma.database vacuum full
``` ```
## Add expiration to all local statuses
```sh tab="OTP"
./bin/pleroma_ctl database ensure_expiration
```
```sh tab="From Source"
mix pleroma.database ensure_expiration
```

View File

@ -858,9 +858,6 @@ Warning: it's discouraged to use this feature because of the associated security
### :auth ### :auth
* `Pleroma.Web.Auth.PleromaAuthenticator`: default database authenticator.
* `Pleroma.Web.Auth.LDAPAuthenticator`: LDAP authentication.
Authentication / authorization settings. Authentication / authorization settings.
* `auth_template`: authentication form template. By default it's `show.html` which corresponds to `lib/pleroma/web/templates/o_auth/o_auth/show.html.eex`. * `auth_template`: authentication form template. By default it's `show.html` which corresponds to `lib/pleroma/web/templates/o_auth/o_auth/show.html.eex`.
@ -890,6 +887,9 @@ Pleroma account will be created with the same name as the LDAP user name.
* `base`: LDAP base, e.g. "dc=example,dc=com" * `base`: LDAP base, e.g. "dc=example,dc=com"
* `uid`: LDAP attribute name to authenticate the user, e.g. when "cn", the filter will be "cn=username,base" * `uid`: LDAP attribute name to authenticate the user, e.g. when "cn", the filter will be "cn=username,base"
Note, if your LDAP server is an Active Directory server the correct value is commonly `uid: "cn"`, but if you use an
OpenLDAP server the value may be `uid: "uid"`.
### OAuth consumer mode ### OAuth consumer mode
OAuth consumer mode allows sign in / sign up via external OAuth providers (e.g. Twitter, Facebook, Google, Microsoft, etc.). OAuth consumer mode allows sign in / sign up via external OAuth providers (e.g. Twitter, Facebook, Google, Microsoft, etc.).

View File

@ -10,6 +10,7 @@ defmodule Mix.Tasks.Pleroma.Database do
alias Pleroma.User alias Pleroma.User
require Logger require Logger
require Pleroma.Constants require Pleroma.Constants
import Ecto.Query
import Mix.Pleroma import Mix.Pleroma
use Mix.Task use Mix.Task
@ -53,8 +54,6 @@ def run(["update_users_following_followers_counts"]) do
end end
def run(["prune_objects" | args]) do def run(["prune_objects" | args]) do
import Ecto.Query
{options, [], []} = {options, [], []} =
OptionParser.parse( OptionParser.parse(
args, args,
@ -94,8 +93,6 @@ def run(["prune_objects" | args]) do
end end
def run(["fix_likes_collections"]) do def run(["fix_likes_collections"]) do
import Ecto.Query
start_pleroma() start_pleroma()
from(object in Object, from(object in Object,
@ -130,4 +127,23 @@ def run(["vacuum", args]) do
Maintenance.vacuum(args) Maintenance.vacuum(args)
end end
def run(["ensure_expiration"]) do
start_pleroma()
days = Pleroma.Config.get([:mrf_activity_expiration, :days], 365)
Pleroma.Activity
|> join(:left, [a], u in assoc(a, :expiration))
|> where(local: true)
|> where([a, u], is_nil(u))
|> Pleroma.RepoStreamer.chunk_stream(100)
|> Stream.each(fn activities ->
Enum.each(activities, fn activity ->
expires_at = Timex.shift(activity.inserted_at, days: days)
Pleroma.ActivityExpiration.create(activity, expires_at, false)
end)
end)
|> Stream.run()
end
end end

View File

@ -20,11 +20,11 @@ defmodule Pleroma.ActivityExpiration do
field(:scheduled_at, :naive_datetime) field(:scheduled_at, :naive_datetime)
end end
def changeset(%ActivityExpiration{} = expiration, attrs) do def changeset(%ActivityExpiration{} = expiration, attrs, validate_scheduled_at) do
expiration expiration
|> cast(attrs, [:scheduled_at]) |> cast(attrs, [:scheduled_at])
|> validate_required([:scheduled_at]) |> validate_required([:scheduled_at])
|> validate_scheduled_at() |> validate_scheduled_at(validate_scheduled_at)
end end
def get_by_activity_id(activity_id) do def get_by_activity_id(activity_id) do
@ -33,9 +33,9 @@ def get_by_activity_id(activity_id) do
|> Repo.one() |> Repo.one()
end end
def create(%Activity{} = activity, scheduled_at) do def create(%Activity{} = activity, scheduled_at, validate_scheduled_at \\ true) do
%ActivityExpiration{activity_id: activity.id} %ActivityExpiration{activity_id: activity.id}
|> changeset(%{scheduled_at: scheduled_at}) |> changeset(%{scheduled_at: scheduled_at}, validate_scheduled_at)
|> Repo.insert() |> Repo.insert()
end end
@ -49,7 +49,9 @@ def due_expirations(offset \\ 0) do
|> Repo.all() |> Repo.all()
end end
def validate_scheduled_at(changeset) do def validate_scheduled_at(changeset, false), do: changeset
def validate_scheduled_at(changeset, true) do
validate_change(changeset, :scheduled_at, fn _, scheduled_at -> validate_change(changeset, :scheduled_at, fn _, scheduled_at ->
if not expires_late_enough?(scheduled_at) do if not expires_late_enough?(scheduled_at) do
[scheduled_at: "an ephemeral activity must live for at least one hour"] [scheduled_at: "an ephemeral activity must live for at least one hour"]

View File

@ -638,6 +638,34 @@ def force_password_reset_async(user) do
@spec force_password_reset(User.t()) :: {:ok, User.t()} | {:error, Ecto.Changeset.t()} @spec force_password_reset(User.t()) :: {:ok, User.t()} | {:error, Ecto.Changeset.t()}
def force_password_reset(user), do: update_password_reset_pending(user, true) def force_password_reset(user), do: update_password_reset_pending(user, true)
# Used to auto-register LDAP accounts which won't have a password hash stored locally
def register_changeset_ldap(struct, params = %{password: password})
when is_nil(password) do
params = Map.put_new(params, :accepts_chat_messages, true)
params =
if Map.has_key?(params, :email) do
Map.put_new(params, :email, params[:email])
else
params
end
struct
|> cast(params, [
:name,
:nickname,
:email,
:accepts_chat_messages
])
|> validate_required([:name, :nickname])
|> unique_constraint(:nickname)
|> validate_exclusion(:nickname, Config.get([User, :restricted_nicknames]))
|> validate_format(:nickname, local_nickname_regex())
|> put_ap_id()
|> unique_constraint(:ap_id)
|> put_following_and_follower_address()
end
def register_changeset(struct, params \\ %{}, opts \\ []) do def register_changeset(struct, params \\ %{}, opts \\ []) do
bio_limit = Config.get([:instance, :user_bio_length], 5000) bio_limit = Config.get([:instance, :user_bio_length], 5000)
name_limit = Config.get([:instance, :user_name_length], 100) name_limit = Config.get([:instance, :user_name_length], 100)

View File

@ -28,10 +28,6 @@ def get_user(%Plug.Conn{} = conn) do
%User{} = user <- ldap_user(name, password) do %User{} = user <- ldap_user(name, password) do
{:ok, user} {:ok, user}
else else
{:error, {:ldap_connection_error, _}} ->
# When LDAP is unavailable, try default authenticator
@base.get_user(conn)
{:ldap, _} -> {:ldap, _} ->
@base.get_user(conn) @base.get_user(conn)
@ -92,7 +88,7 @@ defp bind_user(connection, ldap, name, password) do
user user
_ -> _ ->
register_user(connection, base, uid, name, password) register_user(connection, base, uid, name)
end end
error -> error ->
@ -100,35 +96,32 @@ defp bind_user(connection, ldap, name, password) do
end end
end end
defp register_user(connection, base, uid, name, password) do defp register_user(connection, base, uid, name) do
case :eldap.search(connection, [ case :eldap.search(connection, [
{:base, to_charlist(base)}, {:base, to_charlist(base)},
{:filter, :eldap.equalityMatch(to_charlist(uid), to_charlist(name))}, {:filter, :eldap.equalityMatch(to_charlist(uid), to_charlist(name))},
{:scope, :eldap.wholeSubtree()}, {:scope, :eldap.wholeSubtree()},
{:attributes, ['mail', 'email']},
{:timeout, @search_timeout} {:timeout, @search_timeout}
]) do ]) do
{:ok, {:eldap_search_result, [{:eldap_entry, _, attributes}], _}} -> {:ok, {:eldap_search_result, [{:eldap_entry, _, attributes}], _}} ->
with {_, [mail]} <- List.keyfind(attributes, 'mail', 0) do
params = %{ params = %{
email: :erlang.list_to_binary(mail),
name: name, name: name,
nickname: name, nickname: name,
password: password, password: nil
password_confirmation: password
} }
changeset = User.register_changeset(%User{}, params) params =
case List.keyfind(attributes, 'mail', 0) do
{_, [mail]} -> Map.put_new(params, :email, :erlang.list_to_binary(mail))
_ -> params
end
changeset = User.register_changeset_ldap(%User{}, params)
case User.register(changeset) do case User.register(changeset) do
{:ok, user} -> user {:ok, user} -> user
error -> error error -> error
end end
else
_ ->
Logger.error("Could not find LDAP attribute mail: #{inspect(attributes)}")
{:error, :ldap_registration_missing_attributes}
end
error -> error ->
error error

View File

@ -127,4 +127,43 @@ test "it turns OrderedCollection likes into empty arrays" do
assert Enum.empty?(Object.get_by_id(object2.id).data["likes"]) assert Enum.empty?(Object.get_by_id(object2.id).data["likes"])
end end
end end
describe "ensure_expiration" do
test "it adds to expiration old statuses" do
%{id: activity_id1} = insert(:note_activity)
%{id: activity_id2} =
insert(:note_activity, %{inserted_at: NaiveDateTime.from_iso8601!("2015-01-23 23:50:07")})
%{id: activity_id3} = activity3 = insert(:note_activity)
expires_at =
NaiveDateTime.utc_now()
|> NaiveDateTime.add(60 * 61, :second)
|> NaiveDateTime.truncate(:second)
Pleroma.ActivityExpiration.create(activity3, expires_at)
Mix.Tasks.Pleroma.Database.run(["ensure_expiration"])
expirations =
Pleroma.ActivityExpiration
|> order_by(:activity_id)
|> Repo.all()
assert [
%Pleroma.ActivityExpiration{
activity_id: ^activity_id1
},
%Pleroma.ActivityExpiration{
activity_id: ^activity_id2,
scheduled_at: ~N[2016-01-23 23:50:07]
},
%Pleroma.ActivityExpiration{
activity_id: ^activity_id3,
scheduled_at: ^expires_at
}
] = expirations
end
end
end end

View File

@ -7,7 +7,6 @@ defmodule Pleroma.Web.OAuth.LDAPAuthorizationTest do
alias Pleroma.Repo alias Pleroma.Repo
alias Pleroma.Web.OAuth.Token alias Pleroma.Web.OAuth.Token
import Pleroma.Factory import Pleroma.Factory
import ExUnit.CaptureLog
import Mock import Mock
@skip if !Code.ensure_loaded?(:eldap), do: :skip @skip if !Code.ensure_loaded?(:eldap), do: :skip
@ -72,9 +71,7 @@ test "creates a new user after successful LDAP authorization" do
equalityMatch: fn _type, _value -> :ok end, equalityMatch: fn _type, _value -> :ok end,
wholeSubtree: fn -> :ok end, wholeSubtree: fn -> :ok end,
search: fn _connection, _options -> search: fn _connection, _options ->
{:ok, {:ok, {:eldap_search_result, [{:eldap_entry, '', []}], []}}
{:eldap_search_result, [{:eldap_entry, '', [{'mail', [to_charlist(user.email)]}]}],
[]}}
end, end,
close: fn _connection -> close: fn _connection ->
send(self(), :close_connection) send(self(), :close_connection)
@ -101,50 +98,6 @@ test "creates a new user after successful LDAP authorization" do
end end
end end
@tag @skip
test "falls back to the default authorization when LDAP is unavailable" do
password = "testpassword"
user = insert(:user, password_hash: Pbkdf2.hash_pwd_salt(password))
app = insert(:oauth_app, scopes: ["read", "write"])
host = Pleroma.Config.get([:ldap, :host]) |> to_charlist
port = Pleroma.Config.get([:ldap, :port])
with_mocks [
{:eldap, [],
[
open: fn [^host], [{:port, ^port}, {:ssl, false} | _] -> {:error, 'connect failed'} end,
simple_bind: fn _connection, _dn, ^password -> :ok end,
close: fn _connection ->
send(self(), :close_connection)
:ok
end
]}
] do
log =
capture_log(fn ->
conn =
build_conn()
|> post("/oauth/token", %{
"grant_type" => "password",
"username" => user.nickname,
"password" => password,
"client_id" => app.client_id,
"client_secret" => app.client_secret
})
assert %{"access_token" => token} = json_response(conn, 200)
token = Repo.get_by(Token, token: token)
assert token.user_id == user.id
end)
assert log =~ "Could not open LDAP connection: 'connect failed'"
refute_received :close_connection
end
end
@tag @skip @tag @skip
test "disallow authorization for wrong LDAP credentials" do test "disallow authorization for wrong LDAP credentials" do
password = "testpassword" password = "testpassword"