mlx.data.core.AWSFileFetcher.prefetch

AWSFileFetcher.prefetch(self: mlx.data._c.core.FileFetcher, filenames: List[str]) → None

Start prefetching these files.

num_prefetch_max files are downloaded with num_prefetch_threads parallelism. When one of the prefetched files is accessed by fetch then more of the prefetch file list is downloaded.

At any given point we keep num_kept_files in the local cache.

Parameters:

filenames (list[str]) – A list of filenames to be prefetched in order.