Downloading datasets

All communication with SciCat is handled by a client object. Normally, one would construct one using something like

from scitacean import Client
from scitacean.transfer.sftp import SFTPFileTransfer
client = Client.from_token(url="https://scicat.ess.eu/api/v3",
                           token=...,
                           file_transfer=SFTPFileTransfer(
                               host="login.esss.dk"
                           ))

In this example, we use ESS’s SciCat. If you want to use a different one, you need to figure out its URL. Note that this is not the same URL that you open in a browser but typically ends in a suffix like /api/v3.

Here, we authenticate using a token. You can find your token in the web interface by logging in and opening the settings. Alternatively, we could use username and password via Client.from_credentials.

WARNING:

Do not hard code secrets like tokens or passwords in notebooks or scripts! There is a high risk of exposing them when code is under version control or uploaded to SciCat.

Scitacean currently requires secrets to be passed as function arguments. So you will have to find your own solution for now.

While the client itself is responsible for talking to SciCat, a file_transfer object is required to download data files. Here, we use SFTPFileTransfer which downloads / uploads files via SFTP.

The file transfer needs to authenticate separately from the SciCat connection. By default, it requires an SSH agent to be running an set up for the selected host.

For the purposes of this guide, we don’t want to connect to a real SciCat server in order to avoid the complications associated with that. So we set up a fake client that only pretends to connect to SciCat and file servers. Everything else in this guide works in the same way with a real client. See Developer Documentation/Testing if you are interested in the details.

from scitacean.testing.docs import setup_fake_client

client = setup_fake_client()

Metadata

We need the ID (pid) of a dataset in order to download it. The fake client provides a dataset with id 20.500.12269/72fe3ff6-105b-4c7f-b9d0-073b67c90ec3. We can download it using

dset = client.get_dataset("20.500.12269/72fe3ff6-105b-4c7f-b9d0-073b67c90ec3")

Datasets can easily be inspected in Jupyter notebooks:

dset

RawDataset

	Name	Type	Value	Description
*	creation_time	datetime	2022-06-29 14:01:05+0000	Time when dataset became fully available. This can be the time when all containing files have been written, or when the dataset was created in SciCat. Local times without timezone/offset info are automatically transformed to UTC using the timezone of the API server. Inserted automatically by Scitacean on upload.
*	source_folder	RemotePath	RemotePath('/hex/ps/thaum')	Absolute file path on fileserver containing the files of this dataset. This is usually a POSIX path, e.g., ``/some/path/to/sourcefolder``. All files must be placed within this folder and its subfolders.
	description	str	Measured the thaum flux	Free text explanation of the contents of the dataset.
	input_datasets	list[PID]	[]	Array of input dataset identifiers used in producing this dataset. Can be identifiers in the same or another federated catalogue.
	name	str	Thaum flux	The name of the dataset. Can be set freely by the creator to help identify the dataset.
	pid	PID	20.500.12269/72fe3ff6-105b-4c7f-b9d0-073b67c90ec3	Persistent identifier of the dataset.

Advanced fields

*	contact_email	str	p.stibbons@uu.am	Email of the contact person for this dataset. The string may contain a list of emails, which should then be separated by semicolons.
*	owner	str	Ponder Stibbons	Full name of the owner or custodian of the dataset. The string may contain a list of persons, which should then be separated by semicolons.
*	owner_group	str	uu	Name of the group owning this item. This group must exist in SciCat and is used to control access to this dataset.
	access_groups	list[str]	['faculty']	List of groups which have access to this item.
	api_version	str	None	Version of the API used to create the dataset.
	classification	str	None	ACIA information about the dataset. ACIA stands for AUthenticity,COnfidentiality,INtegrity and AVailability. Example: ``'AV=medium,CO=low'`` SciCat may trigger different operations based on this value.
	comment	str	None	Comment about the dataset.
	created_at	datetime	2022-08-17 14:20:23+0000	Date and time when this dataset was created in the database. This field is managed by SciCat.
	created_by	str	Ponder Stibbons	Username who created this dataset. This field is managed by SciCat.
	creation_location	str	UnseenUniversity	Unique location identifier where data was taken. Usually one of these forms:: /site-name/facility-name/instrumentOrBeamline-name facility-name:instrumentOrBeamline-name
	data_format	str	None	Format of the data files in this dataset. Example: ```Nexus Version x.y.```
	data_quality_metrics	int	None	A number given by the user to rate the dataset.
	end_time	datetime	None	End time of data acquisition for the current dataset. Local times without timezone/offset info are automatically transformed to UTC using the timezone of the API server.
	instrument_group	str	None	Group of the instrument which this item was acquired on.
	instrument_ids	list[str]	None	IDs of the instruments where the data was created.
	is_published	bool	None	True if dataset is publicly available.
	job_log_data	str	None	The job log file. Keep the size of this log data well below 15 MB.
	job_parameters	dict[str, typing.Any]	None	Parameters used by the job that created this dataset.
	keywords	list[str]	None	Array of tags associated with the meaning or contents of this dataset. Values should ideally come from defined vocabularies, taxonomies, ontologies, or knowledge graphs.
	license	str	None	Name of the license under which the data can be used.
	lifecycle	Lifecycle	None	Current status of the dataset during its lifetime w.r.t. storage handling.
	orcid_of_owner	str	None	ORCID iD of the owner or custodian. The string may contain a list of ORCIDs, which should then be separated by semicolons.
	owner_email	str	None	Email of the owner or custodian of the dataset. The string may contain a list of emails, which should then be separated by semicolons.
	principal_investigators	list[str]	['p.stibbons@uu.am']	Full name of the principal investigator(s).
	proposal_ids	list[str]	None	The IDs of the proposals to which the dataset belongs.
	relationships	list[Relationship]	None	Relationships with other datasets.
	run_number	str	None	Run number of the data acquisition.
	sample_ids	list[str]	None	ID(s) of the sample(s) used when collecting the data.
	scientific_metadata_schema	str	None	Link to the schema for scientific Metadata validation.
	scientific_metadata_valid	bool	None	Whether the scientific metadata complies with the schema.
	shared_with	list[str]	None	List of users that the dataset has been shared with.
	source_folder_host	str	None	DNS host name of the fileserver hosting the files.
	start_time	datetime	None	Start time of data acquisition for the current dataset. Local times without timezone/offset info are automatically transformed to UTC using the timezone of the API server.
	techniques	list[Technique]	None	Techniques used to create the data. See Also -------- ontology: Helper module for defining techniques based on known ontologies.
	updated_at	datetime	2022-11-01 13:22:08+0000	Date and time when this record was updated last. This field is managed by SciCat.
	updated_by	str	anonymous	Username who last updated this dataset. This field is managed by SciCat.
	used_software	list[str]	[]	Software used to create this data. Should ideally contain complete and unique identifiers such as links to software releases, DOIs, or software name + version combinations.
	validation_status	str	None	Level of trust. For example, a measure of how much data was verified or used by other persons.

Files: 2 (95 B)

Local	Remote	Size
None	RemotePath('flux.dat')	20 B
None	RemotePath('logs/measurement.log')	75 B

Scientific Metadata

Name	Value
data_type	histogram
temperature	123 [K]

All attributes listed above can be accessed directly:

dset.type

'raw'

dset.name

'Thaum flux'

dset.owner

'Ponder Stibbons'

See Dataset for a list of available fields.

In addition, datasets can have free form scientific metadata which we can be accessed using

dset.meta

{'data_type': 'histogram', 'temperature': {'value': '123', 'unit': 'K'}}

Files

The data files associated with this dataset can be accessed using

for f in dset.files:
    print(f"{f.remote_access_path(dset.source_folder) = }")
    print(f"{f.local_path = }")
    print(f"{f.size = } bytes")
    print("----")

f.remote_access_path(dset.source_folder) = RemotePath('/hex/ps/thaum/flux.dat')
f.local_path = None
f.size = 20 bytes
----
f.remote_access_path(dset.source_folder) = RemotePath('/hex/ps/thaum/logs/measurement.log')
f.local_path = None
f.size = 75 bytes
----

Note that the local_path for both files is None. This indicates that the files have not been downloaded. Indeed, client.get_dataset downloads only the metadata from SciCat, not the files.

We can download the first file using

dset_with_local_file = client.download_files(dset, target="download", select="flux.dat")

for f in dset_with_local_file.files:
    print(f"{f.remote_access_path(dset.source_folder) = }")
    print(f"{f.local_path = }")
    print(f"{f.size = } bytes")
    print("----")

f.remote_access_path(dset.source_folder) = RemotePath('/hex/ps/thaum/flux.dat')
f.local_path = PosixPath('download/flux.dat')
f.size = 20 bytes
----
f.remote_access_path(dset.source_folder) = RemotePath('/hex/ps/thaum/logs/measurement.log')
f.local_path = None
f.size = 75 bytes
----

Which populates the local_path:

file = list(dset_with_local_file.files)[0]

file.local_path

PosixPath('download/flux.dat')

We can use it to read the file:

with file.local_path.open("r") as f:
    print(f.read())

5 4 9 11 15 12 7 6 1

If we wanted to download all files, we could pass select=True (or nothing, True is the default) to client.download_files. See Client.download_files for more options to select files.