Uploading datasets#
Please read Downloading datasets first as it explains the general setup.
We connect to SciCat and a file server using a Client:
from scitacean import Client
from scitacean.transfer.sftp import SFTPFileTransfer
client = Client.from_token(url="https://scicat.ess.eu/api/v3",
token=...,
file_transfer=SFTPFileTransfer(
host="login.esss.dk"
))
This code is identical to the one used for downloading . As with the downloading guide, we use a fake client instead of the real one shown above.
[1]:
from scitacean.testing.docs import setup_fake_client
client = setup_fake_client()
This is especially useful here as datasets cannot be deleted from SciCat by regular users, and we don’t want to pollute the database with our test data.
First, we need to generate some data to upload:
[2]:
from pathlib import Path
path = Path("data/witchcraft.dat")
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w") as f:
f.write("7.9 13 666")
Create a new dataset#
With the totally realistic data in hand, we can construct a dataset.
[3]:
from scitacean import Dataset
dset = Dataset(
name="Spellpower of the Three Witches",
description="The spellpower of the maiden, mother, and crone.",
type="raw",
owner_group="wyrdsisters",
access_groups=["witches"],
owner="Nanny Ogg",
principal_investigators=["Esme Weatherwax"],
contact_email="nogg@wyrd.lancre",
creation_location="lancre/whichhut",
data_format="space-separated",
source_folder="/somewhere/on/remote",
)
There are many more fields that can be filled in as needed. See scitacean.Dataset.
The owner_group and access_groups fields correspond to users/usergroups in SciCat and determine who can access the files. Typically, the owner_group should be based on the proposal ID.
Now we can attach our file:
[4]:
dset.add_local_files("data/witchcraft.dat")
Now, let’s inspect the dataset.
[5]:
dset
[5]:
| Name | Type | Value | Description | |
|---|---|---|---|---|
* |
creation_time | datetime | 2026-07-15 06:36:07+0000 | Time when dataset became fully available. This can be the time when all containing files have been written, or when the dataset was created in SciCat. Local times without timezone/offset info are automatically transformed to UTC using the timezone of the API server. Inserted automatically by Scitacean on upload. |
* |
source_folder | RemotePath | RemotePath('/somewhere/on/remote') | Absolute file path on fileserver containing the files of this dataset. This is usually a POSIX path, e.g., ``/some/path/to/sourcefolder``. All files must be placed within this folder and its subfolders. |
| description | str | The spellpower of the maiden, mother, and crone. | Free text explanation of the contents of the dataset. | |
| input_datasets | list[PID] | [] | Array of input dataset identifiers used in producing this dataset. Can be identifiers in the same or another federated catalogue. | |
| name | str | Spellpower of the Three Witches | The name of the dataset. Can be set freely by the creator to help identify the dataset. | |
| pid | PID | None | Persistent identifier of the dataset. |
Advanced fields
* |
contact_email | str | nogg@wyrd.lancre | Email of the contact person for this dataset. The string may contain a list of emails, which should then be separated by semicolons. |
* |
owner | str | Nanny Ogg | Full name of the owner or custodian of the dataset. The string may contain a list of persons, which should then be separated by semicolons. |
* |
owner_group | str | wyrdsisters | Name of the group owning this item. This group must exist in SciCat and is used to control access to this dataset. |
| access_groups | list[str] | ['witches'] | List of groups which have access to this item. | |
| api_version | str | None | Version of the API used to create the dataset. | |
| classification | str | None | ACIA information about the dataset. ACIA stands for AUthenticity,COnfidentiality,INtegrity and AVailability. Example: ``'AV=medium,CO=low'`` SciCat may trigger different operations based on this value. | |
| comment | str | None | Comment about the dataset. | |
| created_at | datetime | None | Date and time when this dataset was created in the database. This field is managed by SciCat. | |
| created_by | str | None | Username who created this dataset. This field is managed by SciCat. | |
| creation_location | str | lancre/whichhut | Unique location identifier where data was taken. Usually one of these forms:: /site-name/facility-name/instrumentOrBeamline-name facility-name:instrumentOrBeamline-name | |
| data_format | str | space-separated | Format of the data files in this dataset. Example: ```Nexus Version x.y.``` | |
| data_quality_metrics | int | None | A number given by the user to rate the dataset. | |
| end_time | datetime | None | End time of data acquisition for the current dataset. Local times without timezone/offset info are automatically transformed to UTC using the timezone of the API server. | |
| instrument_group | str | None | Group of the instrument which this item was acquired on. | |
| instrument_ids | list[str] | None | IDs of the instruments where the data was created. | |
| is_published | bool | None | True if dataset is publicly available. | |
| job_log_data | str | None | The job log file. Keep the size of this log data well below 15 MB. | |
| job_parameters | dict[str, typing.Any] | None | Parameters used by the job that created this dataset. | |
| keywords | list[str] | None | Array of tags associated with the meaning or contents of this dataset. Values should ideally come from defined vocabularies, taxonomies, ontologies, or knowledge graphs. | |
| license | str | None | Name of the license under which the data can be used. | |
| lifecycle | Lifecycle | None | Current status of the dataset during its lifetime w.r.t. storage handling. | |
| orcid_of_owner | str | None | ORCID iD of the owner or custodian. The string may contain a list of ORCIDs, which should then be separated by semicolons. | |
| owner_email | str | None | Email of the owner or custodian of the dataset. The string may contain a list of emails, which should then be separated by semicolons. | |
| principal_investigators | list[str] | ['Esme Weatherwax'] | Full name of the principal investigator(s). | |
| proposal_ids | list[str] | None | The IDs of the proposals to which the dataset belongs. | |
| relationships | list[Relationship] | None | Relationships with other datasets. | |
| run_number | str | None | Run number of the data acquisition. | |
| sample_ids | list[str] | None | ID(s) of the sample(s) used when collecting the data. | |
| scientific_metadata_schema | str | None | Link to the schema for scientific Metadata validation. | |
| scientific_metadata_valid | bool | None | Whether the scientific metadata complies with the schema. | |
| shared_with | list[str] | None | List of users that the dataset has been shared with. | |
| source_folder_host | str | None | DNS host name of the fileserver hosting the files. | |
| start_time | datetime | None | Start time of data acquisition for the current dataset. Local times without timezone/offset info are automatically transformed to UTC using the timezone of the API server. | |
| techniques | list[Technique] | None | Techniques used to create the data. See Also -------- ontology: Helper module for defining techniques based on known ontologies. | |
| updated_at | datetime | None | Date and time when this record was updated last. This field is managed by SciCat. | |
| updated_by | str | None | Username who last updated this dataset. This field is managed by SciCat. | |
| used_software | list[str] | [] | Software used to create this data. Should ideally contain complete and unique identifiers such as links to software releases, DOIs, or software name + version combinations. | |
| validation_status | str | None | Level of trust. For example, a measure of how much data was verified or used by other persons. |
Files: 1 (10 B)
| Local | Remote | Size |
|---|---|---|
| data/witchcraft.dat | None | 10 B |
[6]:
len(list(dset.files))
[6]:
1
[7]:
dset.size # in bytes
[7]:
10
[8]:
file = list(dset.files)[0]
print(f"{file.remote_access_path(dset.source_folder) = }")
print(f"{file.local_path = }")
print(f"{file.size = } bytes")
file.remote_access_path(dset.source_folder) = None
file.local_path = PosixPath('data/witchcraft.dat')
file.size = 10 bytes
The file has a local_path but no remote_access_path which means that it exists on the local file system (where we put it earlier) but not on the remote file server accessible by SciCat. The location can also be queried using file.is_on_local and file.is_on_remote.
Likewise, the dataset only exists in memory on our local machine and not on SciCat. Nothing has been uploaded yet. So we can freely modify the dataset or bail out by deleting the Python object if we need to.
Upload the dataset#
Once the dataset is ready, we can upload it using
[9]:
finalized = client.upload_new_dataset_now(dset)
WARNING:
This action cannot be undone by a regular user! Contact an admin if you uploaded a dataset accidentally.
scitacean.Client.upload_new_dataset_now uploads the dataset (i.e. metadata) to SciCat and the files to the file server. And it does so in such a way that it always creates a new dataset and new files without overwriting any existing (meta) data.
It returns a new dataset that is a copy of the input with some updated information generated by SciCat and the file transfer. For example, it has been assigned a new ID:
[10]:
finalized.pid
[10]:
PID(prefix='PID.prefix.a0b1', pid='51bab98e-ebfb-4ff4-be1c-bd2f6aea1e18')
And the remote access path of our file has been set:
[11]:
list(finalized.files)[0].remote_access_path(finalized.source_folder)
[11]:
RemotePath('/somewhere/on/remote/witchcraft.dat')
Location of uploaded files#
All files associated with a dataset will be uploaded to the same folder. This folder may be at the path we specify when making the dataset, i.e. dset.source_folder. However, the folder is ultimately determined by the file transfer (in this case SFTPFileTransfer) and it may choose to override the source_folder that we set. In this example, since we don’t tell the file transfer otherwise, it respects dset.source_folder and uploads the files to that location. See the File
transfer reference for information how to control this behavior. The reason for this is that facilities may have a specific structure on their file server and Scitacean’s file transfers can be used to enforce that.
In any case, we can find out where files were uploaded by inspecting the finalized dataset that was returned by client.upload_new_dataset_now:
[12]:
finalized.source_folder
[12]:
RemotePath('/somewhere/on/remote')
Or by looking at each file individually as shown in the section above.
Attaching images to datasets#
It is possible to attach small images to datasets. In SciCat, this is done by creating ‘attachment’ objects which contain the image. Scitacean handles those via the attachments property of Dataset. For our locally created dataset, the property is an empty list and we can add an attachment like this:
[13]:
dset.add_attachment(
caption="Scitacean logo",
thumbnail="./logo.png",
)
dset.attachments[0]
[13]:
Dataset.add_attachment can load an image from a file and properly encode it for SciCat. We could also use a more manual approach and construct scitacean.Attachment and scitacean.Thumbnail objects ourselves and append them to dset.attachments.
When we then upload the dataset, the client automatically uploads all attachments as well. Note that this creates a new dataset in SciCat. If you want to add attachments to an existing dataset after upload, you need to use the lower-level API through client.scicat.create_attachment or the web interface directly.
[14]:
finalized = client.upload_new_dataset_now(dset)
In order to download the attachments again, we can pass attachments=True when downloading the dataset:
[15]:
downloaded = client.get_dataset(finalized.pid, attachments=True)
downloaded.attachments[0]
[15]: