from_template: cap eager coordinate allocation on the dask path#3585
Open
brendancol wants to merge 1 commit into
Open
from_template: cap eager coordinate allocation on the dask path#3585brendancol wants to merge 1 commit into
brendancol wants to merge 1 commit into
Conversation
A fine resolution on a dask backend gets past both size caps and then
runs out of memory building the coordinate vectors at construction.
The dask backends skip _MAX_CELLS because the grid data is lazy, and the
_MAX_CHUNKS guard keys on block count -- the default tiling grows its
block edge to keep that count near _DASK_MAX_BLOCKS, so a fine resolution
never trips it. But _make_output_coords builds the x/y vectors eagerly
with np.linspace, sized width + height, so
from_template("conus", resolution=0.001, backend="dask") would allocate
~72 GB of coordinates up front while the grid data stays lazy.
Add _MAX_COORD_CELLS (1e9 elements) and check width + height against it
in the dask branch. Eager backends are already bounded by _MAX_CELLS, so
this only constrains the otherwise-unbounded dask path; conus at 1 m,
the finest grid in the tests, is ~9e6 coordinate elements.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A fine resolution on a dask backend gets past both
from_templatesize caps and then runs out of memory building the coordinate vectors at construction._MAX_CELLSis skipped on the dask backends because the grid data is lazy._MAX_CHUNKSkeys on block count, and the default tiling grows its block edge to keep that count near_DASK_MAX_BLOCKS, so a fine resolution never trips it. But_make_output_coordsbuilds the x/y vectors eagerly withnp.linspace, sizedwidth + height, so:would allocate ~72 GB of coordinates up front (a 3.1e9 x 5.9e9 grid, ~9e9 coordinate elements) while the grid data stays lazy and the chunk count sits around 2e5, under the 1M cap. Neither existing guard fires.
This PR adds
_MAX_COORD_CELLS(1e9 elements, ~8 GB at float64) and checkswidth + heightagainst it inside the dask branch. Eager backends are already bounded by_MAX_CELLS, so this only constrains the otherwise-unbounded dask path. The finest grid in the tests, conus at 1 m, is ~9e6 coordinate elements, well under the limit.Test
test_over_fine_dask_coord_alloc_raisescovers the explicit-dask and chunks-promotion paths.Found by the templates security sweep (resource-exhaustion / DoS).