Plugin reference#
This page lists all plugins that ship with DLite, together with their options and dependencies. See storage plugins for how to use storage plugins, protocol plugins for protocols and mappings for mapping plugins.
Content#
Storage plugins (drivers)#
Driver |
Language |
Kind |
Requires |
Key options |
|---|---|---|---|---|
C |
generic |
— |
|
|
C |
generic |
HDF5 (source build) |
|
|
C |
generic |
Redland librdf (source build) |
|
|
Python |
generic |
PyYAML |
|
|
Python |
generic |
pandas |
|
|
Python |
generic |
pymongo |
|
|
Python |
generic |
rdflib |
|
|
Python |
generic |
psycopg |
|
|
Python |
generic |
pymongo |
|
|
Python |
generic |
redis |
|
|
Python |
generic |
minio |
|
|
Python |
generic |
requests |
|
|
Python |
specific ( |
scikit-image |
— |
|
Python |
specific ( |
— |
— |
|
Python |
specific (template files) |
jinja2 (optional) |
|
All file-based plugins support the mode option with the values:
a: Append to existing file or create new file (default).r: Open existing file for read-only.w: Truncate existing file or create new file.
json#
Serialises instances to/from JSON. This is the built-in format used
when printing instances and by several of the dlite.Instance
convenience methods (from_json(), asjson()).
Option |
Description |
|---|---|
|
|
|
Whether to write single-entity format. The default ( |
|
Whether to use the URI as JSON key (default: false). |
|
Whether to include the UUID in output. |
|
Always include meta in output. |
|
Serialise metadata dimensions and properties as arrays. |
Example — storing multiple instances in one file:
>>> with dlite.Storage("json", "data.json", options="mode=w;single=no") as s:
... s.save(inst1)
... s.save(inst2)
hdf5#
Serialises instances to/from HDF5 files. Only available in DLite installations built from source with HDF5 enabled.
Option |
Description |
|---|---|
|
|
rdf#
Serialises instances to/from RDF, backed by the Redland librdf library. Only available in DLite installations built from source against librdf. For the RDF format when installing DLite from PyPI, use the Python-based pyrdf driver instead.
yaml#
Serialises instances to/from YAML.
Option |
Description |
|---|---|
|
|
|
Whether to save using the SOFT7 format. |
|
Whether to save in single-instance form. |
|
Whether to include UUID when saving. |
|
Whether to always include “meta” (even for metadata). |
|
Whether to include parent info for transactions. |
|
Whether the URI is the preferred key in multi-instance format. |
The option names with_uuid, with_meta and urikey are deprecated
aliases for with-uuid, with-meta and uri-key, respectively
(see issue #886).
csv#
Reads/writes CSV (and other tabular formats supported by pandas, like
Excel when openpyxl is installed). When loading, the metadata can be
inferred from the data source or provided explicitly.
Option |
Description |
|---|---|
|
|
|
URI to metadata describing the table to read. Required if |
|
Whether to infer metadata from the data source (default: true). |
|
Additional search directories for |
|
Any format supported by pandas. The default is inferred from the file extension. |
|
Comma-separated string of |
|
Explicit id of the returned instance when reading. |
Example — reading a CSV file with inferred metadata:
>>> inst = dlite.Instance.from_location("csv", "data.csv", options="mode=r")
bson#
Serialises instances to/from BSON. The BSON data is translated to JSON.
Option |
Description |
|---|---|
|
|
|
Whether to save using the SOFT7 format. |
pyrdf#
Serialises instances to/from RDF, backed by the pure-Python rdflib library.
Option |
Description |
|---|---|
|
|
|
File format, e.g. |
postgresql#
Stores instances in a PostgreSQL database.
Option |
Description |
|---|---|
|
Name of database to connect to (default: |
|
User name. |
|
Password. |
|
|
mongodb#
Stores instances in a MongoDB database. The location is a MongoDB
connection string.
Option |
Description |
|---|---|
|
|
|
Name of database to use (default: |
|
Name of collection to use (default: |
Additional options are passed to the constructor of pymongo.MongoClient.
redis#
Stores instances in a Redis database.
Option |
Description |
|---|---|
|
Port to connect to (default: 6379). |
|
Redis user name. |
|
Redis password. |
|
Enable socket keepalive (default: true). |
|
Timeout in seconds. |
|
Whether to SSL-encrypt the connection. |
|
Path to SSL certificate file ( |
|
Path to SSL private key file ( |
|
Path to SSL certificate container ( |
|
Database number (default: 0). |
|
Number of seconds before new keys expire. |
minio#
Stores instances in a MinIO (S3-compatible) object store. The location
is the hostname of the S3 service.
Option |
Description |
|---|---|
|
Name of bucket (default: |
|
Access key (aka user ID) of your account. |
|
Secret key (aka password) of your account. |
|
Session token of your account. |
|
Whether to use a secure (TLS) connection. |
|
Region name of buckets. |
|
Connection timeout in seconds. |
http#
Loads instances from a web address using requests. This is a
convenience driver; for full flexibility combine the http protocol
with any driver (see protocol plugins).
Option |
Description |
|---|---|
|
Whether the input is assumed to be in single-entity form. The default ( |
image#
Specific plugin that loads/saves instances of the entity
http://onto-ns.com/meta/0.1/Image. No options. Requires
scikit-image.
blob#
Specific plugin that loads/saves instances of the entity
http://onto-ns.com/meta/0.1/Blob. No options.
template#
Specific plugin that renders an instance to a file using a template.
Useful for generating reports, input files, etc. Requires a template
engine: the format method of the Python standard library (default) or
jinja2.
Option |
Description |
|---|---|
|
Path to the template file. |
|
|
Protocol plugins#
Protocol plugins transfer raw bytes to/from external resources. They
are combined with storage plugins via dlite.Instance.load() and
dlite.Instance.save() or used directly via the Protocol class
(imported from the dlite.protocol module). See protocol plugins
for details and examples.
Protocol |
Description |
Requires |
Key options |
|---|---|---|---|
Read/write local files (and directories) |
— |
|
|
Download/upload via HTTP(S) |
requests |
passed to |
|
Transfer via SFTP |
paramiko |
|
|
Read/write files inside zip archives (with local cache for URLs) |
— |
|
file protocol#
Opens a local file or directory.
Option |
Description |
|---|---|
|
Combination of |
|
Whether |
http protocol#
Downloads/loads data via HTTP(S). All options are passed as keyword arguments to requests. A timeout of 1 second is added by default.
sftp protocol#
Transfers data via SFTP. The location may be just a host name or fully
qualified as username:password@host:port.
Option |
Description |
|---|---|
|
User name. |
|
Password. |
|
Host name. |
|
Port number (default: 22). |
|
Key type for key-based authorisation. |
zip protocol#
Reads/writes files inside zip archives. If location is an URL, a
local cache is created and reused.
Option |
Description |
|---|---|
|
Number of seconds before timing out when downloading an URL. |
|
If true and the location is an URL, download the zip file again even if a cache exists. |
Availability of plugins#
Which plugins are available depends on how DLite was installed:
Installing DLite with
pip install dlite-pythongives a minimal installation, where the only built-in storage plugins arejson(andbson/rdfwhen compiled against the corresponding libraries).Installing the optional requirements (see
requirements_full.txtin the DLite repository) enables the full set of Python plugins listed above:yaml,csv,postgresql,mongodb,redis,minio,http,image,pyrdf,template,blob, …The C plugins
hdf5andrdfare only available in installations built from source with HDF5/librdf enabled.
Note
The single/with-uuid-style options of file-based drivers influence
the layout of the produced files. When in doubt, refer to the driver’s
docstring, which is the authoritative documentation:
>>> import dlite
>>> help(dlite.Storage("json", "data.json", "mode=r"))