Configuring File Storage

Phorge Administrator and User Documentation (Configuration)

Setup file storage and support for large files.

Overview

This document describes how to configure Phorge to support large file uploads, and how to choose where Phorge stores files.

There are two major things to configure:

• set up PHP and your HTTP server to accept large requests;
• choose and configure a storage engine.

The following sections will guide you through this configuration.

How Phorge Stores Files

Phorge stores files in "storage engines", which are modular backends that implement access to some storage system (like MySQL, the filesystem, or a cloud storage service like Amazon S3).

Phorge stores large files by breaking them up into many chunks (a few megabytes in size) and storing the chunks in an underlying storage engine. This makes it easier to implement new storage engines and gives Phorge more flexibility in managing file data.

The first section of this document discusses configuring your install so that PHP and your HTTP server will accept requests which are larger than the size of one file chunk. Without this configuration, file chunk data will be rejected.

The second section discusses choosing and configuring storage engines, so data is stored where you want it to be.

Configuring Upload Limits

File uploads are limited by several pieces of configuration at different layers of the stack. Generally, the minimum value of all the limits is the effective one.

To upload large files, you need to increase all the limits to at least 32MB. This will allow you to upload file chunks, which will let Phorge store arbitrarily large files.

The settings which limit file uploads are:

HTTP Server: The HTTP server may set a limit on the maximum request size. If you exceed this limit, you'll see a default server page with an HTTP error. These directives limit the total size of the request body, so they must be somewhat larger than the desired maximum filesize.

• Apache: Apache limits requests with the Apache LimitRequestBody directive.
• nginx: nginx limits requests with the nginx client_max_body_size directive. This often defaults to 1M.
• lighttpd: lighttpd limits requests with the lighttpd server.max-request-size directive.

Set the applicable limit to at least 32MB. Phorge can not read these settings, so it can not raise setup warnings if they are misconfigured.

PHP: PHP has several directives which limit uploads. These directives are found in php.ini.

• post_max_size: Maximum POST request size PHP will accept. If you exceed this, Phorge will give you a useful error. This often defaults to 8M. Set this to at least 32MB. Phorge will give you a setup warning about this if it is set too low.
• memory_limit: For some uploads, file data will be read into memory before Phorge can adjust the memory limit. If you exceed this, PHP may give you a useful error, depending on your configuration. It is recommended that you set this to -1 to disable it. Phorge will give you a setup warning about this if it is set too low.

You may also want to configure these PHP options:

• max_input_vars: When files are uploaded via HTML5 drag and drop file upload APIs, PHP parses the file body as though it contained normal POST parameters, and may trigger max_input_vars if a file has a lot of brackets in it. You may need to set it to some astronomically high value.
• upload_max_filesize: Maximum file size PHP will accept in a raw file upload. This is not normally used when uploading files via drag-and-drop, but affects some other kinds of file uploads. If you exceed this, Phorge will give you a useful error. This often defaults to 2M. Set this to at least 32MB.

Once you've adjusted all this configuration, your server will be able to receive chunk uploads. As long as you have somewhere to store them, this will enable you to store arbitrarily large files.

Storage Engines

Phorge supports several different file storage engines:

You can review available storage engines and their configuration by navigating to Applications → Files → Help/Options → Storage Engines in the web UI.

By default, Phorge is configured to store files up to 1MB in MySQL, and reject files larger than 1MB. To store larger files, you can either:

• increase the MySQL limit to at least 8MB; or
• configure another storage engine.

Doing either of these will enable the chunk storage engine and support for arbitrarily large files.

The remaining sections of this document discuss the available storage engines and how to configure them.

Engine: MySQL

• Pros: Low latency, no setup required.
• Cons: Storing files in a database is a classic bad idea. May become difficult to administrate if you have a large amount of data.

MySQL storage is configured by default, for files up to (just under) 1MB. You can configure it with these keys:

• storage.mysql-engine.max-size: Change the filesize limit, in bytes. Set to 0 to disable.

For most installs, it is reasonable to leave this engine as-is and let small files (like thumbnails and profile images) be stored in MySQL, which is usually the lowest-latency filestore, even if you configure another storage engine.

To support large files, increase this limit to at least 8388608 (8MB). This will activate chunk storage in MySQL.

Engine: Local Disk

• Pros: Simple to setup.
• Cons: Doesn't scale to multiple web frontends without NFS.

To configure file storage on the local disk, set:

• storage.local-disk.path: Set to some writable directory on local disk. Make that directory.

Engine: Amazon S3

• Pros: Scales well.
• Cons: Slightly more complicated than other engines, not free.

To enable file storage in S3, set these keys:

• amazon-s3.access-key: Your AWS access key.
• amazon-s3.secret-key: Your AWS secret key.
• amazon-s3.region: Your AWS S3 region.
• amazon-s3.endpoint: Your AWS S3 endpoint.
• storage.s3.bucket: S3 bucket name where files should be stored.

Testing Storage Engines

You can test that things are correctly configured by dragging and dropping a file onto the Phorge home page. If engines have been configured properly, the file should upload.

Migrating Files Between Engines

If you want to move files between storage engines, you can use the bin/files script to perform migrations. For example, suppose you previously used MySQL but recently set up S3 and want to migrate all your files there. First, migrate one file to make sure things work:

phorge/ $ ./bin/files migrate --engine amazon-s3 F12345

If that works properly, you can then migrate everything:

phorge/ $ ./bin/files migrate --engine amazon-s3 --all

You can use --dry-run to show which migrations would be performed without taking any action. Run bin/files help for more options and information.

Next Steps

Continue by:

• reviewing at-rest encryption options with Configuring Encryption; or
• returning to the Configuration Guide.