Fixing the “Admin Media Gallery Hangs / 504 Gateway Time-out” Error in Magento 2 (S3 Remote Storage)

One of the more frustrating Magento 2 admin problems shows up only after you move pub/media to S3 and your media library has grown large: open Content > Media > Media Gallery and the folder tree just spins. On a big library — hundreds of thousands of assets, or a catalog/product tree in the millions — the AJAX call eventually dies with a 504 Gateway Time-out or 503 Service Unavailable, or PHP gives up with Maximum execution time of N seconds exceeded. No exception is logged — the request simply runs past your proxy and max_execution_time budgets.

For editors this is a hard stop. They can't pick images for CMS blocks, categories, or WYSIWYG content, which means content work grinds to a halt. We'll cover why it happens, how to confirm it, and a class override you can adapt that — on a ~1.3M-asset store we measured — drops the call from minutes to roughly 100ms. Smaller S3 stores feel this as "slower than local" rather than a hard timeout; the fix still helps, but the dramatic numbers below come from a large library.

Why Magento 2 hangs here

The slow code is in core: Magento\MediaGalleryUi\Model\Directories\GetDirectoryTree::execute(). To build the admin folder tree, it calls readRecursively() on the media DirectoryList, then loops over every returned path. On local disk that walk is cheap. Under the AWS S3 driver it is not — and the reason is different from what most write-ups assume.

• The S3 driver does ONE recursive listing, not one call per folder: readRecursively() ends at AwsS3::readDirectoryRecursively(), which calls the Flysystem adapter's listContents(prefix, deep: true). When deep is true the adapter omits the S3 Delimiter, so a single recursive ListObjects sweep returns the entire key space under the media prefix, paginated at 1000 keys per page. Request count is roughly ceil(total_objects / 1000) — so it scales with total object count, not folder count. At ~1.3M objects that's ~1,300 sequential paginated requests, plus the PHP-side cost of iterating every one of those keys.
• The expensive work is enumerating objects you'll throw away: core lists the whole prefix — including a catalog/product tree that can hold over a million files — and then discards nearly all of it to keep only directory nodes. The per-path isDirectory() metadata check in that loop is normally served from the cache primed by the recursive listing, so it isn't the dominant cost; the dominant cost is simply enumerating and iterating ~1.3M objects.
• The exclusion check is CPU-only, but it doesn't help here: each path passes through IsPathExcludedInterface (bound to Magento\MediaGallery\Model\Directory\IsExcluded), which does pure string normalization plus a preg_match against cached config — no storage I/O. It filters the result after the expensive listing has already happened, so it can't save you any S3 calls.
• The folder data is already in the DB (with caveats): Magento's media-gallery sync populates media_gallery_asset with one row per synced image file. Folder paths can be derived from those rows with SUBSTRING_INDEX, which MySQL returns in milliseconds. That's the basis of the fix below — with the trade-off that the table only knows about folders that contain at least one synced asset.

1. Confirm remote storage is the trigger

Before touching code, prove S3 is in play. A local-disk store will not reproduce this.

php bin/magento config:show | grep -iE 'remote_storage|media_storage'
grep -A6 "'remote_storage'" app/etc/env.php

You should see remote_storage configured with the aws-s3 driver. (Note the grep -iE — basic grep treats the | as a literal character, so an alternation pattern silently matches nothing and you'd wrongly conclude remote storage is off.) If the driver is File, the slow tree is something else — stop here. If it's S3, the recursive listing in GetDirectoryTree is being served by ListObjects, and that's your cost source.

2. Verify the folder paths are already indexed

The tree only needs folder names, and those are derivable from the asset paths the DB already holds.

SELECT SUBSTRING_INDEX(path, '/', 2) AS dir, COUNT(*)
FROM media_gallery_asset
GROUP BY dir ORDER BY 2 DESC LIMIT 20;

You should see your top-level gallery folders (catalog/category/..., wysiwyg/...) with asset counts. That confirms the folders that contain synced assets are derivable from MySQL — no storage walk required. Two caveats to keep in mind: media_gallery_asset stores file rows, not folder rows, so it only "knows" folders that hold at least one synced asset; and it's populated by the sync job, so empty folders and anything not yet synced won't appear — core's live walk would show them. If the table is empty or stale, run bin/magento media-gallery:sync first. Treat the table-driven tree as an accepted trade-off for a read-only image picker, not exact parity with core's live tree.

3. Override the core class with a di.xml preference

A thin module swaps the class wholesale — no core patch.

// app/code/Vendor/Module/etc/di.xml
<config>
  <preference for="Magento\MediaGalleryUi\Model\Directories\GetDirectoryTree"
              type="Vendor\Module\Model\Directories\GetDirectoryTree"/>
</config>

After cache:clean (or setup:di:compile in production), the admin gallery endpoint resolves to your implementation. Keep the same public surface — execute(): array returning the jstree node structure — so every caller is unchanged.

4. Build the tree from media_gallery_asset, not readRecursively()

Replace the storage listing with a query that SELECT DISTINCTs folder paths per depth level. The critical thing to get right: in core, IsPathExcluded enforces an admin-configurable allowlist — the media_gallery_image_folders config, which defaults to wysiwyg and catalog/category and deliberately hides catalog/product. Your WHERE clause is now what enforces that allowlist, so it must mirror your configured folders. Read the config dynamically, or at minimum match it exactly. Do not broaden to catalog/%, or you'll surface catalog/product (which can be over a million assets) and junk subdirectories that the gallery is designed to hide.

-- one row per folder, per depth, restricted to the configured allowlist
-- (default media_gallery_image_folders: catalog/category, wysiwyg)
SELECT DISTINCT SUBSTRING_INDEX(path, '/', 1) FROM media_gallery_asset
  WHERE path LIKE 'catalog/category/%' OR path LIKE 'wysiwyg/%'
UNION
SELECT DISTINCT SUBSTRING_INDEX(path, '/', 2) FROM media_gallery_asset
  WHERE path LIKE 'catalog/category/%' OR path LIKE 'wysiwyg/%'
UNION
SELECT DISTINCT SUBSTRING_INDEX(path, '/', 3) FROM media_gallery_asset
  WHERE path LIKE 'catalog/category/%' OR path LIKE 'wysiwyg/%'
UNION
SELECT DISTINCT SUBSTRING_INDEX(path, '/', 4) FROM media_gallery_asset
  WHERE path LIKE 'catalog/category/%' OR path LIKE 'wysiwyg/%';

Map each dir_path to the node shape jstree expects — and match core's shape exactly so node selection still works:

[
  'text'       => $leaf,                       // last path segment
  'id'         => $path,
  'path'       => $path,
  'path_array' => explode('/', $path),
  'li_attr'    => ['data-id' => $path],        // jstree uses this to identify the <li>
]

Then nest each node under its parent with a parent-finder. Two adaptations you'll likely need beyond this sketch: the example caps depth at four levels via the UNIONed SUBSTRING_INDEX calls, so deeper trees get truncated unless you extend it; and if your store enables additional allowed folders, add their prefixes to both the config and the WHERE. We've seen this most often when a store with a deep, heavily nested media library migrates to S3 and the gallery quietly becomes unusable for editors.

5. Clear caches and confirm it opens instantly

php bin/magento setup:di:compile && php bin/magento cache:flush

Open the gallery and time the AJAX request in your browser network panel. On a large library you should see it return in well under a second — on our ~1.3M-asset store, roughly 100ms instead of minutes — and S3 ListObjects volume on gallery open should drop to near zero. Spot-check that folder nesting and labels match the folders you intended to expose, and remember the table-driven tree won't show empty or un-synced folders.

When this points to a deeper problem

This single fix is usually a symptom of a broader pattern worth auditing:

• Magento's media abstractions hide the cost gap between local and remote storage, so a recursive listing that's free on disk becomes an expensive ListObjects sweep over the entire prefix on S3 — and the cost grows with total object count, not how many folders you actually want to show.
• The UI re-derives folder structure from live storage on every request even though media_gallery_asset already holds the synced paths it could read instead.
• Work scoped by a post-hoc allowlist (here, IsPathExcluded filtering after the full listing) still pays the full enumeration cost — the win comes from never listing the excluded bulk in the first place.
• No request-level timeout guard exists — the operation only fails when an external PHP or proxy timeout fires, which is why the symptom shows up as a 504/503 with nothing in the logs.

Once a store moves to remote storage, grep for readRecursively, getDirectoryRead()->read(), and any media-directory enumeration running synchronously inside an admin request. For each, ask whether the same data already lives in media_gallery_asset or another index, and whether you're listing far more than you'll display. Overriding one class buys you a fast gallery; the durable answer is auditing every storage walk that assumes local-disk economics.

You now have a clear way to confirm the cause and a concrete, adaptable override that makes the Admin Media Gallery fast again on S3 — without patching core, and without exposing folders the gallery is meant to hide.