Commit Graph

69 Commits

Author SHA1 Message Date
Claire
8acc75435b
Change S3 checksum mode to be disabled by default (#27007) 2023-09-21 14:00:51 +02:00
Claire
4bc0dd751c
Add S3_DISABLE_CHECKSUM_MODE environment variable for compatibility with some S3-compatible providers (#26435) 2023-08-10 14:15:18 +02:00
Misty De Méo
b848ba3867
Paperclip: add support for Azure blob storage (#23607) 2023-07-19 09:02:49 +02:00
Claire
dc8f1fbd97
Merge pull request from GHSA-9928-3cp5-93fm
* Fix attachments getting processed despite failing content-type validation

* Add a restrictive ImageMagick security policy tailored for Mastodon

* Fix misdetection of MP3 files with large cover art

* Reject unprocessable audio/video files instead of keeping them unchanged
2023-07-06 15:05:05 +02:00
Nick Schonning
c0b9664a31
Autofix Rubocop spacing in config (#25022) 2023-05-22 13:17:56 +02:00
Matt Jankowski
d9a958fcf7
Fix Performance/RedundantMerge cop (#24817) 2023-05-04 05:25:43 +02:00
Matt Jankowski
d902a707a3
Fix Rails/CompactBlank cop (#24690) 2023-04-30 14:07:21 +02:00
Claire
e084b5b82d
Fix user archive takeout when using OpenStack Swift or S3 providers with no ACL support (#24200) 2023-03-27 17:07:37 +02:00
Jamie Hoyle
de137e6bb0
Added support for specifying S3 storage classes in environment (#22480) 2023-03-03 20:53:37 +01:00
Matt Corallo
9d039209cc
Add Cache-Control header to openstack-stored files (#20610)
When storing files in S3, paperclip is configured with a Cache-Control header
indicating the file is immutable, however no such header was added when using
OpenStack storage.

Luckily Paperclip's fog integration makes this trivial, with a simple
`fog_file` `Cache-Control` default doing the trick.
2022-11-14 05:26:49 +01:00
David Hewitt
290d78cea4
Allow unsetting x-amz-acl S3 Permission headers (#20510)
Some "S3 Compatible" storage providers (Cloudflare R2 is one such example) don't support setting ACLs on individual uploads with the `x-amz-acl` header, and instead just have a visibility for the whole bucket. To support uploads to such providers without getting unsupported errors back, lets use a black `S3_PERMISSION` env var to indicate that these headers shouldn't be sent.

This is tested as working with Cloudflare R2.
2022-11-13 06:57:10 +01:00
Holger
39b489ba4c
fix: s3_force_single_request not parsed (#17922) 2022-04-01 23:56:23 +02:00
Eugen Rochko
3f2533ca8e
Fix autoloading deprecation warnings from Rails 6 (#16010) 2021-04-09 02:31:20 +02:00
Eugen Rochko
1045549f85
Add stoplight for object storage failures, return HTTP 503 (#13043) 2020-12-15 12:55:29 +01:00
Eugen Rochko
df1653174b
Add cache buster feature for media files (#15155)
Nginx can be configured to bypass proxy cache when a special header
is in the request. If the response is cacheable, it will replace
the cache for that request. Proxy caching of media files is
desirable when using object storage as a way of minimizing bandwidth
costs, but has the drawback of leaving deleted media files for
a configured amount of cache time. A cache buster can make those
media files immediately unavailable. This especially makes sense
when suspending and unsuspending an account.
2020-11-19 17:38:06 +01:00
tateisu
7919418e4c
add S3_READ_TIMEOUT environment variable (#14952) 2020-10-06 21:29:22 +02:00
Eugen Rochko
7aaf2b44ec
Fix remote files not using Content-Type header, streaming (#14184) 2020-06-30 23:58:02 +02:00
Takeshi Umeda
8e056bd82e
Fix csv upload (#13835) 2020-05-24 09:15:23 +02:00
mayaeh
acc367fd14
Fix naming issue (#13551) 2020-04-27 10:32:05 +02:00
Eugen Rochko
c3ca3801f2
Add separate cache directory for non-local uploads (#12821) 2020-04-26 23:29:08 +02:00
Eugen Rochko
49b2f7c0a2
Fix base64-encoded file uploads not being possible (#12748)
Fix #3804, Fix #5776
2020-01-04 01:54:07 +01:00
Eugen Rochko
17159625b3
Add S3_OVERRIDE_PATH_STYLE environment variable (#12594)
To support Exoscale
2019-12-10 07:40:01 +01:00
Eugen Rochko
f3d232381d
Add tootctl media remove-orphans (#12568) 2019-12-08 15:37:12 +01:00
tateisu
f1ef777d40 add S3_OPEN_TIMEOUT environment variable (#12459) 2019-12-02 21:05:27 +01:00
Eugen Rochko
354fdd317e
Fix attachment not being re-downloaded even if file is not stored (#12125)
Change the behaviour of remotable concern. Previously, it would skip
downloading an attachment if the stored remote URL is identical to
the new one. Now it would not be skipped if the attachment is not
actually currently stored by Paperclip.
2019-10-09 07:10:46 +02:00
Eugen Rochko
086fc7ed77
Fix S3 adapter retrying failing uploads with exponential backoff (#12085)
The default limit of 10 retries with exponential backoff meant
that if the S3 server was timing out, you would be stuck with it
for much, much longer than the 5 second read timeout we expect.

The uploading happens within a database transaction, which means
a failing S3 server could negatively affect database performance
2019-10-06 06:20:57 +02:00
Yamagishi Kazutoshi
b02169f124 Cast multipart threshold to integer (#11944) 2019-09-24 17:32:12 +02:00
Yamagishi Kazutoshi
172eaeba3f Add config of multipart threshold for S3 (#11924) 2019-09-23 15:37:45 +02:00
Eugen Rochko
4699cf853c
Add timeouts for S3 (#9842) 2019-01-18 01:36:59 +01:00
Nolan Lawson
f05eb67081 Enable immutable caching for S3 objects (#9722)
I also added "public" here, as I can't think of a good reason not to add it. Perhaps it has some marginal benefit in that ISPs (or other proxies) can cache it for all users. The assets are certainly publicly available and the same for all users.
2019-01-05 12:29:53 -05:00
M Somerville
2bba6e582d Rename S3_CLOUDFRONT_HOST to S3_ALIAS_HOST. (#8423)
Still check for S3_CLOUDFRONT_HOST for existing installs.
2018-08-25 13:27:08 +02:00
ThibG
f06fa09962 Revert to using Paperclip's filesystem storage, and fix dangling records in remove_remote (#8339)
* Fix uncaching worker

* Revert to using Paperclip's filesystem backend instead of fog-local

fog-local has lots of concurrency issues, causing failure to delete files,
dangling file records, and spurious errors UncacheMediaWorker
2018-08-21 17:53:01 +02:00
Hugo Gameiro
ea4e243303 Improve OpenStack v3 compatibility (#7392)
* Update paperclip.rb

* Update .env.production.sample

* Update paperclip.rb
2018-05-07 02:28:28 +02:00
Yamagishi Kazutoshi
28384c1771 Revert "Revert "Upgrade Paperclip to version 6.0.0" (#6807)" (#6808)
This reverts commit 40871caa4b.
2018-03-24 12:52:45 +01:00
Eugen Rochko
40871caa4b
Revert "Upgrade Paperclip to version 6.0.0" (#6807)
* Revert "Bump version to 2.3.2rc1"

This reverts commit cdf8b92fea.

* Revert "Downgrade Dockerfile to Ruby 2.4.3 on Alpine 3.6 (#6806)"

This reverts commit 0074cad44f.

* Revert "Handle Mastodon::HostValidationError when pulling remoteable assets (#6782)"

This reverts commit 4a0a19fe54.

* Revert "Correct the reference to user's password in mastodon:add_user task (#6800)"

This reverts commit 338bff8b93.

* Revert "Upgrade Paperclip to version 6.0.0 (#6754)"

This reverts commit b88fcd53f7.
2018-03-17 14:20:35 +01:00
Yamagishi Kazutoshi
b88fcd53f7 Upgrade Paperclip to version 6.0.0 (#6754) 2018-03-17 12:37:58 +01:00
THE BOSS ♨
17e26f8afe Fix typo in paperclip.rb (#5936) 2017-12-09 13:59:59 +09:00
Yamagishi Kazutoshi
b0db4dad79 Revert fog-aws (ref #5604) (#5934) 2017-12-09 00:47:52 +01:00
MitarashiDango
cbbeec05be Fix spell miss (SWIIFT_OBJECT_URL -> SWIFT_OBJECT_URL) (#5617) 2017-11-07 19:06:30 +01:00
Yamagishi Kazutoshi
47b0c61853 Unify file upload to using fog (#5604) 2017-11-07 14:30:31 +01:00
Jeong Arm
9d97054fe6 Remove timestamps on any option (#5282) 2017-10-09 17:52:02 +02:00
Nishi, Keisuke
83ffc4dc07 Fix Paperclip::Fog always responds Not Found in OpenStack-v2 like ConoHa (#5155) 2017-09-30 14:28:29 +02:00
Patrick Figel
3018043fc2 Add OpenStack Keystone V3 support (#4889)
Keystone V2 is deprecated in favour of V3. This adds the necessary
connection parameters for establishing a V3 connection. Connections
to V2 endpoints are still possible and the configuration should
remain compatible.

This also introduces a SWIFT_REGION variable for multi-region
OpenStack environments and a SWIFT_CACHE_TTL that controls how long
tokens and other meta-data is cached for. Caching tokens avoids
rate-limiting errors that would result in media uploads becoming
unavailable during high load or when using tasks like
media:remove_remote. fog-openstack only supports token caching for
V3 endpoints, so a recommendation for using V3 was added.
2017-09-11 15:11:13 +02:00
Adam Thurlow
6994664a13 swift-enable the paperclip! 📎 (#2322) 2017-09-05 23:17:06 +02:00
alpaca-tc
ee3e0a93f4 Fixes unknown mime type (#2822) 2017-05-05 21:32:14 +02:00
Ash Furrow
66b39ccaed Removes timestamp from URLs. (#2185) 2017-04-20 03:54:24 +02:00
tmyt
6c83fec971 Make configuarable s3_permissions for paperclip (#2139) 2017-04-19 14:20:36 +02:00
Yamagishi Kazutoshi
8b8e140c71 Change to switch signature version for Amazon S3 (#2124) 2017-04-19 14:18:50 +02:00
Eugen
e09ab2c0bd Fix #1642, fix #1912 - Dictate content-type file extension (#2078)
* Fix #1642, fix #1912 - Previous change (#1718) did not modify how original file was saved on upload

* Fix for when file is missing
2017-04-18 23:15:44 +02:00
Joachim Viide
363de2dffd Leave out the "Expires" header from S3 uploads (#1886) 2017-04-16 04:01:58 +02:00