Ingestor ¶
The OpenEM ingestor service extracts metadata and integrates it with SciCat.
The ingestor service can be enabled by setting
INGESTOR_ENABLED
in
.env
.
Configuration options ¶
The ingestor configuration is defined by
openem-ingestor-config.yaml
in
compose.base.yaml
and in
config/openem-ingestor-config.yaml
.
To run the service, set
INGESTOR_ENABLED
and
INGESTOR_DATA_DIR
in
.env
.
Default configuration ¶
In the default setup in compose.base.yaml , the ingestor:
- depends on a healthy backend ,
-
uses
${BACKEND_HTTPS_URL:-http://backend.localhost}/api/v3as SciCat backend host, -
uses
${FRONTEND_HTTPS_URL:-http://localhost}as SciCat frontend host, -
mounts
${INGESTOR_DATA_DIR}as/data(read-only), -
exposes the service on port
8080.
Enable additional features ¶
By default, transfer method is
None
. To enable Globus transfer, set
TRANSFER_METHOD=ExtGlobus
and configure:
-
GLOBUS_TRANSFER_PROXY_URL -
GLOBUS_SOURCE_FACILITY -
GLOBUS_DESTINATION_FACILITY -
GLOBUS_COLLECTION_ROOT_PATH
Authentication ¶
The ingestor can authenticate users based on OIDC claims. If
OIDC_ENABLED
is true,
this will be enabled by default. It can also specifically be enabled or disabled by
setting
AUTH_ENABLED=true
in
.env
.
When ingestor authentication is disabled any scicat user can access the ingestor and
browse data. Otherwise, only users with the keycloak roles
scicat-ingestor/ingestor-read
and
scicat-ingestor/ingestor-write
will be able to use
the ingestor. These roles are granted to the default
oidc-user
, but should be added to
other users in keycloak with 'User > Role mapping > Assign role > Client roles'.
Jobs ¶
Setting
JOBS_ENABLED=true
is recommended to use the ingestor.
An
archive
jobType should be configured to enable the 'autoArchive' feature of the ingestor.
(This is provided in the default
jobConfig.yaml
configuration of scicatlive.)
DEV configuration ¶
With
INGESTOR_DEV=true
(or
DEV=true
), the ingestor uses
compose.dev.yaml
, extending
services/compose.dev.yaml
, and builds from source with target
builder
.