scitacean.Dataset

class scitacean.Dataset(type=None, access_groups=None, classification=None, comment=None, contact_email=None, creation_location=None, creation_time='now', data_format=None, data_quality_metrics=None, description=None, end_time=None, input_datasets=None, instrument_group=None, instrument_ids=None, is_published=None, job_log_data=None, job_parameters=None, keywords=None, license=None, lifecycle=None, name=None, orcid_of_owner=None, owner=None, owner_email=None, owner_group=None, principal_investigators=None, proposal_ids=None, relationships=None, run_number=None, sample_ids=None, scientific_metadata_schema=None, scientific_metadata_valid=None, shared_with=None, source_folder=None, source_folder_host=None, start_time=None, techniques=None, used_software=None, validation_status=None, meta=None, checksum_algorithm='blake2b')

Metadata and linked data files for a measurement, simulation, or analysis.

Constructors

__init__([type, access_groups, ...])
from_download_model(dataset_model)	Construct a new dataset from SciCat download models.

Methods

add_attachment([thumbnail, owner_group, ...])	Create a new attachment and add it to the dataset.
add_files(*files[, datablock])	Add files to the dataset.
add_local_files(*paths[, datablock])	Add files on the local file system to the dataset.
add_orig_datablock(*, checksum_algorithm)	Append a new orig datablock to the list of orig datablocks.
as_new()	Return a new dataset with lifecycle-related fields erased.
derive(*[, keep])	Return a new dataset that is derived from self.
fields([read_only])	Iterate over dataset fields.
items()	Dict-like items(name and value pairs of fields) method.
keys()	Dict-like keys(names of fields) method.
make_attachment_upload_models()	Build models for all registered attachments.
make_datablock_upload_models()	Build models for all contained (orig) datablocks.
make_upload_fields()	Return a dict with the fields for uploading a dataset.
make_upload_model()	Construct a SciCat upload model from self.
replace(*[, _read_only, _orig_datablocks])	Return a new dataset with replaced fields.
replace_files(*files)	Return a new dataset with replaced files.
validate()	Validate the fields of the dataset.
values()	Dict-like values(values of fields) method.

Attributes

access_groups	List of groups which have access to this item.
api_version	Version of the API used to create the dataset.
attachments	List of attachments for this dataset.
classification	ACIA information about the dataset.
comment	Comment about the dataset.
contact_email	Email of the contact person for this dataset.
created_at	Date and time when this dataset was created in the database.
created_by	Username who created this dataset.
creation_location	Unique location identifier where data was taken.
creation_time	Time when dataset became fully available.
data_format	Format of the data files in this dataset.
data_quality_metrics	A number given by the user to rate the dataset.
description	Free text explanation of the contents of the dataset.
end_time	End time of data acquisition for the current dataset.
files	Files linked with the dataset.
input_datasets	Array of input dataset identifiers used in producing this dataset.
instrument_group	Group of the instrument which this item was acquired on.
instrument_id	Single instrument ID if there is at most one.
instrument_ids	IDs of the instruments where the data was created.
is_published	True if dataset is publicly available.
job_log_data	The job log file.
job_parameters	Parameters used by the job that created this dataset.
keywords	Array of tags associated with the meaning or contents of this dataset.
license	Name of the license under which the data can be used.
lifecycle	Current status of the dataset during its lifetime w.r.t.
meta	Dict of scientific metadata.
name	The name of the dataset.
number_of_files	Number of files in directly accessible storage in the dataset.
number_of_files_archived	Total number of archived files in the dataset.
orcid_of_owner	ORCID iD of the owner or custodian.
owner	Full name of the owner or custodian of the dataset.
owner_email	Email of the owner or custodian of the dataset.
owner_group	Name of the group owning this item.
packed_size	Total size of all datablock package files created for this dataset.
pid	Persistent identifier of the dataset.
principal_investigator	Full name of the single principal investigator if there is at most one.
principal_investigators	Full name of the principal investigator(s).
proposal_id	The ID of the single proposal for this dataset if there is at most one.
proposal_ids	The IDs of the proposals to which the dataset belongs.
relationships	Relationships with other datasets.
run_number	Run number of the data acquisition.
sample_id	ID of the sample used when collecting the data if there is at most one.
sample_ids	ID(s) of the sample(s) used when collecting the data.
scientific_metadata_schema	Link to the schema for scientific Metadata validation.
scientific_metadata_valid	Whether the scientific metadata complies with the schema.
shared_with	List of users that the dataset has been shared with.
size	Total size of files in directly accessible storage in the dataset.
source_folder	Absolute file path on fileserver containing the files of this dataset.
source_folder_host	DNS host name of the fileserver hosting the files.
start_time	Start time of data acquisition for the current dataset.
techniques	Techniques used to create the data.
type	The type of this dataset.
updated_at	Date and time when this record was updated last.
updated_by	Username who last updated this dataset.
used_software	Software used to create this data.
validation_status	Level of trust.

__init__(type=None, access_groups=None, classification=None, comment=None, contact_email=None, creation_location=None, creation_time='now', data_format=None, data_quality_metrics=None, description=None, end_time=None, input_datasets=None, instrument_group=None, instrument_ids=None, is_published=None, job_log_data=None, job_parameters=None, keywords=None, license=None, lifecycle=None, name=None, orcid_of_owner=None, owner=None, owner_email=None, owner_group=None, principal_investigators=None, proposal_ids=None, relationships=None, run_number=None, sample_ids=None, scientific_metadata_schema=None, scientific_metadata_valid=None, shared_with=None, source_folder=None, source_folder_host=None, start_time=None, techniques=None, used_software=None, validation_status=None, meta=None, checksum_algorithm='blake2b')

add_attachment(thumbnail=None, *, caption, owner_group=None, access_groups=None, instrument_group=None)

Create a new attachment and add it to the dataset.

Parameters:

• thumbnail (str | PathLike[str] | Thumbnail | None, default: None) – If a scitacean.thumbnail.Thumbnail object, it is added to the attachment. If a string or path, a thumbnail is loaded from that path.

• caption (str) – Caption of the attachment.

• owner_group (str | None, default: None) – Owner group of the attachment. Defaults to self.owner_group.

• access_groups (list[str] | None, default: None) – Access groups of the attachment. Defaults to self.access_groups.

• instrument_group (str | None, default: None) – Instrument group of the attachment. Defaults to self.instrument_group.

Return type:

None

add_files(*files, datablock=None)

Add files to the dataset.

Parameters:

• files (File) – File object to add.

• datablock (int | None, default: None) –

  Advanced feature, do not set unless you know what this is!

  Select the orig datablock to store the file in.

  • None: Use the last datablock in the list if possible or add a new one if needed.

  • Otherwise, use the datablock with that index.

Return type:

None

add_local_files(*paths, datablock=None)

Add files on the local file system to the dataset.

The files are set up to be uploaded to the dataset’s source folder without preserving the local directory structure. That is, given

dataset.source_folder = "remote/source"
dataset.add_local_files("/path/to/file1", "other_path/file2")

and uploading this dataset to SciCat, the files will be uploaded to:

remote/source/file1
remote/source/file2

Parameters:

• paths (str | PathLike[str]) – Local paths to the files.

• datablock (int | None, default: None) –

  Advanced feature, do not set unless you know what this is!

  Select the orig datablock to store the file in.

  • None: Use the last datablock in the list if possible or add a new one if needed.

  • Otherwise, use the datablock with that index.

Return type:

None

add_orig_datablock(*, checksum_algorithm)

Append a new orig datablock to the list of orig datablocks.

Parameters:

checksum_algorithm (str | None) – Use this algorithm to compute checksums of files associated with this datablock.

Returns:

OrigDatablock – The newly added datablock.

as_new()

Return a new dataset with lifecycle-related fields erased.

The returned dataset has the same fields as self. But fields that indicate when the dataset was created or by who are set to None. This if, for example, created_at, history, and lifecycle.

Returns:

Dataset – A new dataset without lifecycle-related fields.

property attachments: list[Attachment] | None

List of attachments for this dataset.

This property can be in two distinct ‘falsy’ states:

• dset.attachments is None: It is unknown whether there are attachments. This happens when datasets are downloaded without downloading the attachments.

• dset.attachments == []: It is known that there are no attachments. This happens either when downloading datasets or when initializing datasets locally without assigning attachments.

derive(*, keep=('contact_email', 'orcid_of_owner', 'owner', 'owner_email', 'principal_investigators', 'techniques'))

Return a new dataset that is derived from self.

The returned dataset has most fields set to None. But a number of fields can be carried over from self. By default, this assumes that the owner of the derived dataset is the same as the owner of the original. This can be customized with the keep argument.

Parameters:

keep (Iterable[str], default: ('contact_email', 'orcid_of_owner', 'owner', 'owner_email', 'principal_investigators', 'techniques')) – Fields to copy over to the derived dataset.

Returns:

Dataset – A new derived dataset.

Raises:

ValueError – If self has no PID. The derived dataset requires a PID in order to link back to self.

classmethod fields(read_only=None)

Iterate over dataset fields.

This is similar to dataclasses.fields().

Parameters:

read_only (bool | None, default: None) – If true or false, return only fields which are read-only or allow write-access, respectively. If unset, do not filter fields.

Returns:

Generator[Field, None, None] – Iterable over the fields of datasets.

property files: tuple[File, ...]

Files linked with the dataset.

classmethod from_download_model(dataset_model)

Construct a new dataset from SciCat download models.

Parameters:

dataset_model (DownloadDataset) – Model of the dataset. Must contain orig datablocks and attachments if those should be added to the dataset.

Returns:

Dataset – A new Dataset instance.

items()

Dict-like items(name and value pairs of fields) method.

Returns:

Iterable[tuple[str, Any]] – Generator of (Name, Value) pairs of all fields corresponding to self.type and other fields that are not None.

Added in version 23.10.0.

keys()

Dict-like keys(names of fields) method.

Returns:

Iterable[str] – Generator of names of all fields corresponding to self.type and other fields that are not None.

Added in version 23.10.0.

make_attachment_upload_models()

Build models for all registered attachments.

Raises:

ValueError – If self.attachments is None, i.e., the attachments are uninitialized. Or, if self.pid is None in which case the attachments cannot be associated with this dataset in SciCat.

Returns:

list[UploadAttachment] – List of attachment models.

make_datablock_upload_models()

Build models for all contained (orig) datablocks.

Returns:

DatablockUploadModels – Structure with datablock and orig datablock models.

make_upload_fields()

Return a dict with the fields for uploading a dataset.

Returns:

dict[str, Any] – The writable fields of this dataset. Nested models are converted such that the following is valid:

model.UploadDataset(**dataset.make_upload_fields())

make_upload_model()

Construct a SciCat upload model from self.

Return type:

UploadDataset

property number_of_files: int

Number of files in directly accessible storage in the dataset.

This includes files on both the local and remote filesystems.

Corresponds to OrigDatablocks.

property number_of_files_archived: int

Total number of archived files in the dataset.

Corresponds to Datablocks.

property packed_size: int

Total size of all datablock package files created for this dataset.

replace(*, _read_only=None, _orig_datablocks=None, **replacements)

Return a new dataset with replaced fields.

Parameters starting with an underscore are for internal use. Using them may result in a broken dataset.

Parameters:

replacements (Any) – New field values.

Returns:

Dataset – The new dataset has the same fields as the input but all fields given as keyword arguments are replaced by the given values.

replace_files(*files)

Return a new dataset with replaced files.

For each argument, if the input dataset has a file with the same remote path, that file is replaced. Otherwise, a new file is added. Other existing files are kept in the returned dataset.

Parameters:

files (File) – New files for the dataset.

Returns:

Dataset – A new dataset with given files.

property size: int

Total size of files in directly accessible storage in the dataset.

This includes files on both the local and remote filesystems.

Corresponds to OrigDatablocks.

validate()

Validate the fields of the dataset.

Raises pydantic.ValidationError if validation fails. Returns normally if it passes.

Return type:

None

values()

Dict-like values(values of fields) method.

Returns:

Iterable[Any] – Generator of values of all fields corresponding to self.type and other fields that are not None.

Added in version 23.10.0.