Python API reference

See Python usage for an introduction with examples.

DataLad commands

The commands added to DataLad by datalad-fuse, available from datalad.api and as methods of datalad.api.Dataset.

fusefs(mount_path[, dataset, foreground, ...])	FUSE File system providing transparent access to files under DataLad control
fsspec_head(path[, dataset, lines, bytes, ...])	Show leading lines/bytes of an annexed file by fetching its data from a remote URL
fsspec_cache_clear([dataset, recursive])	Clear fsspec cache

Opening files: datalad_fuse.adapter

class datalad_fuse.adapter.DatasetAdapter(path, caching, mode_transparent=False, backends=None)

Bases: object

Read access to the files of a single dataset.

Files that are not annexed, and annexed files whose content is present locally, are opened from disk. Annexed files without local content are opened from one of the http(s) URLs found for their git-annex key (see get_urls()), reading only the needed parts of the file.

Parameters:

• path (str or Path) – Top directory of the dataset (any git or git-annex repository).

• caching (bool) – If true, keep the data fetched from remote URLs in a sparse on-disk cache under <path>/.git/datalad/cache/ (one subdirectory per backend), to be reused by subsequent reads (for up to a week with the fsspec backend). If false, data are only buffered in memory while a file is open.

• mode_transparent (bool) – If true, paths of key files under .git/annex/objects/ (the targets of annexed symlinks) are opened as annexed content, fetched from a remote URL if not present locally.

• backends (str, optional) – Comma-separated, priority-ordered backends to try, e.g. "remfile,fsspec". Defaults to the datalad.fusefs.backends configuration option, or to DEFAULT_BACKENDS.

Notes

Call close() (or use contextlib.closing()) when done, to stop the git annex processes started for the dataset.

close()

Stop the batched git annex processes started for the dataset

Return type:

None

get_file_state(relpath)

Determine whether a file is annexed and has its content present

Results are cached (for the most recently queried files).

Parameters:

relpath (str) – Path of the file relative to the top directory of the dataset.

Returns:

The state of the file, and its git-annex key if it is annexed.

Return type:

tuple of (FileState, AnnexKey or None)

get_urls(key)

Yield candidate http(s) URLs for the content of an annex key

URLs are yielded in the order in which they are tried by open():

1. http(s) URLs recorded in git-annex for the key, as reported by git annex whereis (e.g. those of the web special remote);

2. annex/objects/... locations on the http(s) git remotes that git annex whereis lists as having the key, including the annex/objects endpoint of Forgejo-aneksajo instances.

URLs on S3 special remotes with exporttree=yes are not included; open() falls back to them via get_exporttree_urls().

Parameters:

key (str) – A git-annex key, e.g. str(AnnexKey).

Return type:

Iterator[str]

get_exporttree_urls(relpath, key)

Yield versioned URLs for relpath on S3 exporttree remotes.

Workaround for datasets lacking proper versioned URLs in git-annex metadata. Constructs URLs from the remote’s publicurl + fileprefix and resolves the correct S3 object version by matching key.size.

Parameters:

• relpath (str)

• key (AnnexKey)

Return type:

Iterator[str]

open(relpath, mode='rb', encoding='utf-8', errors=None)

Open a file of the dataset for reading

Annexed files without local content are opened by the first configured backend that can handle them (see can_handle()), trying each candidate URL in turn; if none of them works, the next backend is tried.

Parameters:

• relpath (str) – Path of the file relative to the top directory of the dataset.

• mode (str) – "rb" (default) to get a binary file object, "r" or "rt" to get a text file object.

• encoding (str) – Encoding to use in text mode.

• errors (str, optional) – How to handle encoding errors in text mode, as for open().

Returns:

A seekable, read-only file object. Files read from disk are regular Python file objects; files read from a URL are whatever the backend that opened them returns (an fsspec file object for the fsspec backend, a RemfileWrapper for remfile).

Return type:

file object

Raises:

• NotImplementedError – If mode is not one of the supported read modes.

• IOError – If the content of an annexed file is not present locally and no backend could open any of its candidate URLs.

clear()

Remove the on-disk caches of the dataset (only if caching)

Return type:

None

class datalad_fuse.adapter.RemoteFilesystemAdapter(root, caching, mode_transparent=False, backends=None)

Bases: object

Read access to the files of a dataset and its installed subdatasets.

Each path is mapped to the (sub)dataset containing it, and a DatasetAdapter is created for that dataset on first use. Use it as a context manager, so that the git annex processes started for the datasets are stopped on exit.

Parameters:

• root (str or Path) – Top directory of the (super)dataset.

• caching (bool) – Passed to each DatasetAdapter.

• mode_transparent (bool) – Passed to each DatasetAdapter.

• backends (str, optional) – Passed to each DatasetAdapter.

Notes

Use an absolute root, and absolute paths under it for the methods. Paths relative to root or to the current directory are not supported.

get_dataset_path(path)

Return the top directory of the (sub)dataset containing path

Parameters:

path (str | Path)

Return type:

Path

resolve_dataset(filepath)

Return the adapter for the dataset containing filepath

Returns:

The adapter, and the path of filepath relative to the dataset’s top directory.

Return type:

tuple of (DatasetAdapter, str)

Parameters:

filepath (str | Path)

open(filepath, mode='rb', encoding='utf-8', errors=None)

Open a file for reading; see DatasetAdapter.open()

Parameters:

• filepath (str | Path)

• mode (str)

• encoding (str)

• errors (str | None)

Return type:

IO

get_file_state(filepath)

Return state and key of a file; see DatasetAdapter.get_file_state

Parameters:

filepath (str | Path)

Return type:

tuple[FileState, AnnexKey | None]

is_under_annex(filepath)

Tell whether a file is annexed

Parameters:

filepath (str | Path)

Return type:

bool

get_commit_datetime(filepath)

Return the date of HEAD in the dataset containing filepath

Parameters:

filepath (str | Path)

Return type:

datetime

class datalad_fuse.adapter.FileState(*values)

Bases: Enum

State of a file in a dataset, as returned by get_file_state()

NOT_ANNEXED = 1

The file is not annexed (e.g. committed to git directly); it is read from disk.

NO_CONTENT = 2

The file is annexed but its content is not present locally; it is read from a remote URL.

HAS_CONTENT = 3

The file is annexed and its content is present locally; it is read from disk.

datalad_fuse.adapter.resolve_backends(backends=None, config=None)

Resolve backends spec from backends argument, config, or default.

config is a dataset’s ConfigManager, so that a per-dataset datalad.fusefs.backends in .git/config or .datalad/config is honored; it inherits the global overrides (-c), so it is a superset of the global datalad.cfg. Falls back to the global config when not given.

Returns (spec, explicit) where explicit is True when the user (or config) supplied the spec and False when falling back to DEFAULT_BACKENDS. Callers use explicit to decide whether a missing backend is a warning (explicit) or a silent skip (default).

Parameters:

• backends (str | None)

• config (Any | None)

Return type:

tuple[str, bool]

datalad_fuse.adapter.create_backends(spec, path, caching, explicit=True)

Instantiate backends from a comma-separated spec.

Backends that cannot be imported are skipped; missing backends requested via an explicit spec (user argument or config) are logged as warnings, while those missing from the default spec are logged at debug level only. Raises ValueError if no usable backend remains.

Parameters:

• spec (str)

• path (str | Path)

• caching (bool)

• explicit (bool)

Return type:

list[Backend]

Note

Up to 0.6.0 these lived in datalad_fuse.fsspec, and RemoteFilesystemAdapter was called FsspecAdapter. Importing FsspecAdapter, DatasetAdapter, FileState or is_http_url from datalad_fuse.fsspec still works, with a DeprecationWarning.

Backends: datalad_fuse.backends

Backends do the actual reading of remote files; see Backends.

class datalad_fuse.backends.Backend

Bases: ABC

Base class for remote file access backends.

abstractmethod can_handle(key, mode, relpath=None)

Return True if this backend should be used for key in mode.

relpath is the path within the dataset, for backends that dispatch on the file name when the annex key carries no suffix (URL/VURL keys).

Parameters:

• key (AnnexKey | None)

• mode (str)

• relpath (str | None)

Return type:

bool

abstractmethod open_url(url, mode='rb', **kwargs)

Open url and return a file-like object.

Parameters:

• url (str)

• mode (str)

• kwargs (Any)

Return type:

IO

clear()

Clear any caches held by this backend. Default: no-op.

Return type:

None

datalad_fuse.backends.DEFAULT_BACKENDS = 'remfile,fsspec'

Backends used when neither --backends nor the datalad.fusefs.backends configuration option is set.

class datalad_fuse.fsspec.FsspecBackend(path, caching)

Bases: Backend

Backend using fsspec’s HTTPFileSystem (optionally with disk caching).

Parameters:

• path (str | Path)

• caching (bool)

can_handle(key, mode, relpath=None)

Return True if this backend should be used for key in mode.

relpath is the path within the dataset, for backends that dispatch on the file name when the annex key carries no suffix (URL/VURL keys).

Parameters:

• key (AnnexKey | None)

• mode (str)

• relpath (str | None)

Return type:

bool

open_url(url, mode='rb', **kwargs)

Open url and return a file-like object.

Parameters:

• url (str)

• mode (str)

• kwargs (Any)

Return type:

IO

clear()

Clear any caches held by this backend. Default: no-op.

Return type:

None

class datalad_fuse.remfile.RemfileBackend(path, caching)

Bases: Backend

Backend using remfile for HDF5-structured files (.nwb, .h5, etc.).

Parameters:

• path (str | Path)

• caching (bool)

PROBE_TIMEOUT = 10.0

Timeout, in seconds, for the size probe in open_url().

can_handle(key, mode, relpath=None)

Return True if this backend should be used for key in mode.

relpath is the path within the dataset, for backends that dispatch on the file name when the annex key carries no suffix (URL/VURL keys).

Parameters:

• key (AnnexKey | None)

• mode (str)

• relpath (str | None)

Return type:

bool

open_url(url, mode='rb', **kwargs)

Open url and return a file-like object.

Parameters:

• url (str)

• mode (str)

• kwargs (Any)

Return type:

IO

clear()

Clear any caches held by this backend. Default: no-op.

Return type:

None

class datalad_fuse.remfile.RemfileWrapper(remfile_obj, url)

Bases: object

Wraps remfile.File to satisfy the contracts expected by datalad-fuse.

Adds context manager protocol, line iteration (for fsspec_head), and an info() method compatible with file_getattr in fuse_.py.

Parameters:

• remfile_obj (Any)

• url (str)

read(size=-1)

Read up to size bytes, or to the end of the file by default

Parameters:

size (int | None)

Return type:

bytes

seek(offset, whence=0)

Set the read position, and return it

Parameters:

• offset (int)

• whence (int)

Return type:

int

tell()

Return the current read position

Return type:

int

close()

Close the file

Return type:

None

info()

Minimal info dict matching the fsspec convention.

DataLadFUSE.getattr(path, fh) reaches this for every fstat() on an open handle, so it must not hit the network: a HEAD request would fail outright against presigned S3 GET URLs, which reject HEAD. remfile already determined the size when the file was opened.

Return type:

dict[str, Any]

git-annex helpers: datalad_fuse.utils

class datalad_fuse.utils.AnnexKey(backend, name, size=None, mtime=None, chunk_size=None, chunk_number=None, suffix=None)

Bases: object

A git-annex key, parsed into its fields

See <https://git-annex.branchable.com/internals/key_format/>. str() of an instance gives back the key.

Examples

>>> k = AnnexKey.parse("SHA256E-s1024--0123abcd.nwb")
>>> k.backend, k.size, k.name, k.suffix
('SHA256E', 1024, '0123abcd', '.nwb')
>>> str(k)
'SHA256E-s1024--0123abcd.nwb'

Parameters:

• backend (str)

• name (str)

• size (int | None)

• mtime (int | None)

• chunk_size (int | None)

• chunk_number (int | None)

• suffix (str | None)

classmethod parse(s)

Parse a key; raises ValueError if s is not a valid key

Parameters:

s (str)

Return type:

AnnexKey

classmethod parse_filename(s)

Parse a key from the name of a file under .git/annex/objects/

Such names escape some characters of the key (e.g. / as %).

Parameters:

s (str)

Return type:

AnnexKey

class datalad_fuse.utils.AnnexDir(topdir)

Bases: object

A (hashing) directory under .git/annex/objects/ of a repository

Parameters:

topdir (str)

topdir: str

Top directory of the repository

datalad_fuse.utils.is_annex_dir_or_key(path)

Tell whether path points into .git/annex/objects/

Returns an AnnexKey for a key file (.git/annex/objects/Xx/Yy/KEY/KEY), an AnnexDir for a directory leading to one, and None otherwise.

Parameters:

path (str | Path)

Return type:

AnnexDir | AnnexKey | None

FUSE file system: datalad_fuse.fuse_

class datalad_fuse.fuse_.DataLadFUSE(root, caching, mode_transparent=False, backends=None)

Bases: Operations

fusepy file system exposing a dataset, as used by datalad fusefs

Files are read via a RemoteFilesystemAdapter, so annexed files without local content are read from their remote URLs. Unless mode_transparent is set, annexed files appear as regular files. For files without local content, the size is taken from their annex key and the modification time is the date of the HEAD commit. Files cannot be written to.

Parameters:

• root (str) – Absolute path, without symbolic links (see os.path.realpath), to the top directory of the dataset to expose.

• caching (bool) – Whether to cache remote data on disk; see DatasetAdapter.

• mode_transparent (bool) – Whether to expose the .git directories of the datasets (hidden by default). Annexed files without local content then appear as symlinks into .git/annex/objects/.

• backends (str, optional) – Comma-separated, priority-ordered backends to read remote files with; see DatasetAdapter.

Examples

Mount a dataset with extra FUSE options:

from fuse import FUSE
FUSE(DataLadFUSE("/abs/path/to/ds", caching=False), "/mnt/point",
     foreground=True, ro=True)