{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# SOFT7 Data Source\n", "\n", "This notebook contains examples related to the SOFT7 data source.\n", "How to create, manage, resolve, and use SOFT7 data sources." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generate a SOFT7 Data Source\n", "\n", "SOFT7 Data Source instances can be generated based on information from the following parts:\n", "\n", "1. Data source (DB, File, Webpage, ...).\n", "2. Generic data source parser.\n", "3. Data source parser configuration.\n", "4. SOFT7 entity (data model).\n", "\n", "Parts 2 and 3 are together considered to produce the \"specific parser\".\n", "Parts 1 through 3 are provided as a collective, based on the [`ResourceConfig`](https://emmc-asbl.github.io/oteapi-core/latest/all_models/#oteapi.models.ResourceConfig) from [OTEAPI Core](https://emmc-asbl.github.io/oteapi-core/)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Resource configuration\n", "\n", "The resource configuration, originally based on the [Data Catalog Vocabulary (DCAT)](https://www.w3.org/TR/vocab-dcat-3/), is in this case a small set of data catalog fields mapped to resource-specific values:\n", "\n", "- `downloadUrl` or `accessUrl`:\n", " - `downloadUrl`: The URL of the downloadable file in a given format. E.g. CSV file or RDF file.\n", "\n", " Usage: `downloadURL` _SHOULD_ be used for the URL at which this distribution is available directly, typically through a HTTPS GET request or SFTP.\n", " - `accessUrl`: A URL of the resource that gives access to a distribution of the dataset. E.g. landing page, feed, SPARQL endpoint.\n", "\n", " Usage: `accessURL` _SHOULD_ be used for the URL of a service or location that can provide access to this distribution, typically through a Web form, query or API call. \n", " `downloadURL` is preferred for direct links to downloadable resources.\n", "- `mediaType`: The media type of the distribution as defined by IANA [[IANA-MEDIA-TYPES](https://www.w3.org/TR/vocab-dcat-2/#bib-iana-media-types)].\n", "\n", " Usage: This property _SHOULD_ be used when the media type of the distribution is defined in IANA [[IANA-MEDIA-TYPES](https://www.w3.org/TR/vocab-dcat-2/#bib-iana-media-types)].\n", "- `accessService`: A data service that gives access to the distribution of the dataset.\n", "\n", "It is worth noting that the resource configuration **MUST** contain either `downloadUrl` _and_ `mediaType` **or** `accessUrl` _and_ `accessService`.\n", "It may contain any combination otherwise, but a minimum of one of the two combinations is required.\n", "\n", "The part described up to now defines the \"data source\" part of the SOFT7 data source.\n", "Furthermore, based on either `mediaType` or `accessService` the \"generic parser\" part of the SOFT7 data source is determined.\n", "\n", "To supply a parser configuration, one needs to know the generic parser that will be used, as well as the specific data to be retrieved.\n", "Under \"normal\" circumstances, the parser configuration is stored alongside a reference to the data source for easy reusability.\n", "\n", "Finally, a SOFT7 entity (or data model) is required to base the generated SOFT7 Data Source instance on.\n", "Again, under \"normal\" circumstances, the SOFT7 entity is stored alongside a reference to the data source for easy reusability." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "from __future__ import annotations\n", "\n", "import contextlib\n", "\n", "with contextlib.suppress(ImportError):\n", " from rich import print" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example of generating a SOFT7 Data Source\n", "\n", "The following example shows how to generate a SOFT7 Data Source instance based on the parts described above." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import logging\n", "\n", "logging.getLogger(\"s7\").addHandler(logging.StreamHandler())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from s7.factories import create_entity\n", "\n", "OPTIMADEStructure = create_entity(\"http://onto-ns.com/meta/1.0/OPTIMADEStructure#\")\n", "# OPTIMADEStructure.model_fields[\"properties\"].annotation.model_fields[\n", "# \"attributes\"\n", "# ].annotation.model_fields[\"properties\"].annotation.model_fields\n", "\n", "optimade_structure_data_as_soft = {\n", " \"properties\": {\n", " \"id\": \"mp-1228448\",\n", " \"type\": \"structures\",\n", " \"attributes\": {\n", " \"dimensions\": {\n", " \"nsites\": 5,\n", " \"nelements\": 2,\n", " \"dimensionality\": 3,\n", " \"nspecies\": 2,\n", " \"nstructure_features\": 0,\n", " },\n", " \"properties\": {\n", " \"immutable_id\": \"645d307dbcd30f748b48eefb\",\n", " \"last_modified\": \"2021-02-10T01:38:17Z\",\n", " \"elements\": [\"Al\", \"O\"],\n", " \"elements_ratios\": [0.4, 0.6],\n", " \"chemical_formula_descriptive\": \"Al2O3\",\n", " \"chemical_formula_reduced\": \"Al2O3\",\n", " \"chemical_formula_hill\": \"Al2O3\",\n", " \"chemical_formula_anonymous\": \"A3B2\",\n", " \"dimension_types\": [1, 1, 1],\n", " \"nperiodic_dimensions\": 3,\n", " \"lattice_vectors\": [\n", " [\n", " -1.508747,\n", " -2.6132,\n", " 0.000129\n", " ],\n", " [\n", " -1.508747,\n", " 2.6132,\n", " -0.000129\n", " ],\n", " [\n", " 0,\n", " 0.000184,\n", " -7.574636\n", " ]\n", " ],\n", " \"cartesian_site_positions\": [\n", " [\n", " -1.508747,\n", " 2.6132805930160004,\n", " -5.469387346584\n", " ],\n", " [\n", " -1.508747,\n", " -2.613096593016,\n", " -2.105248653416\n", " ],\n", " [\n", " -1.508747,\n", " -0.8710047307360002,\n", " -6.063801547042\n", " ],\n", " [\n", " -1.508747,\n", " 0.8711887307360001,\n", " -1.510834452958\n", " ],\n", " [\n", " 0,\n", " 0.000092,\n", " -3.787318\n", " ]\n", " ],\n", " \"species\": [\n", " {\n", " \"dimensions\": {\n", " \"nelements\": 1,\n", " \"nattached_elements\": 0,\n", " },\n", " \"properties\": {\n", " \"name\": \"Al\",\n", " \"chemical_symbols\": [\"Al\"],\n", " \"concentration\": [1],\n", " \"mass\": None,\n", " \"original_name\": None,\n", " \"attached\": None,\n", " \"nattached\": None,\n", " }\n", " },\n", " {\n", " \"dimensions\": {\n", " \"nelements\": 1,\n", " \"nattached_elements\": 0,\n", " },\n", " \"properties\": {\n", " \"name\": \"O\",\n", " \"chemical_symbols\": [\"O\"],\n", " \"concentration\": [1],\n", " \"mass\": None,\n", " \"original_name\": None,\n", " \"attached\": None,\n", " \"nattached\": None,\n", " }\n", " },\n", " ],\n", " \"species_at_sites\": [\"Al\", \"Al\", \"O\", \"O\", \"O\"],\n", " \"assemblies\": None,\n", " \"structure_features\": [],\n", " }\n", " }\n", " }\n", "}\n", "\n", "# print(OPTIMADEStructure.model_fields[\"properties\"].annotation.model_fields[\n", "# \"attributes\"\n", "# ].annotation.__args__[0].__doc__)\n", "OPTIMADEStructure(**optimade_structure_data_as_soft)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "from s7.factories import create_datasource\n", "\n", "os.environ[\"OTELIB_DEBUG\"] = \"true\"\n", "\n", "datasource = create_datasource(\n", " oteapi_url=\"python\",\n", " entity=\"http://onto-ns.com/meta/1.0/OPTIMADEStructure\",\n", " configs={\n", " \"dataresource\": {\n", " \"resourceType\": \"resource/url\",\n", " \"downloadUrl\": \"https://optimade.materialsproject.org/v1/structures/mp-1228448\",\n", " \"mediaType\": \"application/json\",\n", " },\n", " \"parser\": {\n", " \"parserType\": \"parser/json\",\n", " \"entity\": \"http://onto-ns.com/meta/1.0/OPTIMADEStructure\",\n", " },\n", " \"mapping\": {\n", " \"mappingType\": \"triples\",\n", " \"prefixes\": {\n", " \"optimade\": \"https://optimade.materialsproject.org/v1/structures/mp-1228448#\",\n", " \"s7_top\": \"http://onto-ns.com/meta/1.0/OPTIMADEStructure#\",\n", " \"s7_attr\": \"http://onto-ns.com/meta/1.0/OPTIMADEStructureAttributes#\",\n", " \"s7_species\": \"http://onto-ns.com/meta/1.0/OPTIMADEStructureSpecies#\",\n", " },\n", " \"triples\": {\n", " # top\n", " (\"optimade:data.id\", \"\", \"s7_top:properties.id\"),\n", " (\"optimade:data.type\", \"\", \"s7_top:properties.type\"),\n", " (\"optimade:data.attributes\", \"\", \"s7_top:properties.attributes\"),\n", "\n", " # attributes - dimensions\n", " (\"optimade:data.attributes.nsites\", \"\", \"s7_attr:dimensions.nsites\"),\n", " (\n", " \"optimade:data.attributes.nelements\",\n", " \"\",\n", " \"s7_attr:dimensions.nelements\",\n", " ),\n", " (\n", " \"optimade:data.attributes.dimension_types\",\n", " \"\",\n", " \"s7_attr:dimensions.dimensionality\",\n", " ),\n", " (\"optimade:data.attributes.species\", \"\", \"s7_attr:dimensions.nspecies\"),\n", " (\n", " \"optimade:data.attributes.structure_features\",\n", " \"\",\n", " \"s7_attr:dimensions.nstructure_features\",\n", " ),\n", " # attributes - properties\n", " (\n", " \"optimade:data.attributes.immutable_id\",\n", " \"\",\n", " \"s7_attr:properties.immutable_id\",\n", " ),\n", " (\n", " \"optimade:data.attributes.last_modified\",\n", " \"\",\n", " \"s7_attr:properties.last_modified\",\n", " ),\n", " (\n", " \"optimade:data.attributes.elements\",\n", " \"\",\n", " \"s7_attr:properties.elements\",\n", " ),\n", " (\n", " \"optimade:data.attributes.elements_ratios\",\n", " \"\",\n", " \"s7_attr:properties.elements_ratios\",\n", " ),\n", " (\n", " \"optimade:data.attributes.chemical_formula_descriptive\",\n", " \"\",\n", " \"s7_attr:properties.chemical_formula_descriptive\",\n", " ),\n", " (\n", " \"optimade:data.attributes.chemical_formula_reduced\",\n", " \"\",\n", " \"s7_attr:properties.chemical_formula_reduced\",\n", " ),\n", " (\n", " \"optimade:data.attributes.chemical_formula_hill\",\n", " \"\",\n", " \"s7_attr:properties.chemical_formula_hill\",\n", " ),\n", " (\n", " \"optimade:data.attributes.chemical_formula_anonymous\",\n", " \"\",\n", " \"s7_attr:properties.chemical_formula_anonymous\",\n", " ),\n", " (\n", " \"optimade:data.attributes.dimension_types\",\n", " \"\",\n", " \"s7_attr:properties.dimension_types\",\n", " ),\n", " (\n", " \"optimade:data.attributes.nperiodic_dimensions\",\n", " \"\",\n", " \"s7_attr:properties.nperiodic_dimensions\",\n", " ),\n", " (\n", " \"optimade:data.attributes.lattice_vectors\",\n", " \"\",\n", " \"s7_attr:properties.lattice_vectors\",\n", " ),\n", " (\n", " \"optimade:data.attributes.cartesian_site_positions\",\n", " \"\",\n", " \"s7_attr:properties.cartesian_site_positions\",\n", " ),\n", " (\n", " \"optimade:data.attributes.species_at_sites\",\n", " \"\",\n", " \"s7_attr:properties.species_at_sites\",\n", " ),\n", " (\n", " \"optimade:data.attributes.structure_features\",\n", " \"\",\n", " \"s7_attr:properties.structure_features\",\n", " ),\n", " (\"optimade:data.attributes.species\", \"\", \"s7_attr:properties.species\"),\n", "\n", " # attributes.species - properties\n", " (\n", " \"optimade:data.attributes.species.name\",\n", " \"\",\n", " \"s7_species:properties.name\",\n", " ),\n", " (\n", " \"optimade:data.attributes.species.chemical_symbols\",\n", " \"\",\n", " \"s7_species:properties.chemical_symbols\",\n", " ),\n", " (\n", " \"optimade:data.attributes.species.concentration\",\n", " \"\",\n", " \"s7_species:properties.concentration\",\n", " ),\n", " (\n", " \"optimade:data.attributes.species.mass\",\n", " \"\",\n", " \"s7_species:properties.mass\",\n", " ),\n", " (\n", " \"optimade:data.attributes.species.original_name\",\n", " \"\",\n", " \"s7_species:properties.original_name\",\n", " ),\n", " (\n", " \"optimade:data.attributes.species.attached\",\n", " \"\",\n", " \"s7_species:properties.attached\",\n", " ),\n", " (\n", " \"optimade:data.attributes.species.nattached\",\n", " \"\",\n", " \"s7_species:properties.nattached\",\n", " ),\n", " # attributes.species - dimensions\n", " (\n", " \"optimade:data.attributes.species.chemical_symbols\",\n", " \"\",\n", " \"s7_species:dimensions.nelements\",\n", " ),\n", " (\n", " \"optimade:data.attributes.species.attached\",\n", " \"\",\n", " \"s7_species:dimensions.nattached_elements\",\n", " ),\n", " },\n", " },\n", " },\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "datasource.attributes.properties.species[0].properties.name" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from s7.factories.datasource_factory import CACHE\n", "\n", "print(CACHE)" ] } ], "metadata": { "kernelspec": { "display_name": "soft", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" }, "mystnb": { "execution_mode": "off" } }, "nbformat": 4, "nbformat_minor": 2 }