Skip to content

SciCatLive

Get set up with an instance of SciCat to explore the metadata catalog. SciCatlive provides a flexible and easy way to learn about SciCat and its features for people who are looking to integrate SciCat into their environment. For a user guide please see original documentation .

This project requires docker and docker compose. The docker version must be later than 2.29.0 to support this project.

First stable version

Release v3.0 is the first stable and reviewed version of SciCatLive.

Steps

Installing via the OCI packages

Starting from release v3.10 , SciCatLive is published as OCI packages, which can be found on the GitHub Container Registry .

For every new release, these tags are added to the registry:

  • latest(|version) : the latest release with default configuration;
  • latest(|version)-full : the latest release with all features enabled;
  • latest(|version)-v3 : the latest release with the v3 backend and minimal configuration.
  • latest(|version)-v3-full : the latest release with the v3 backend and all features enabled.

To use the OCI packages, simply pull the desired tag from the registry and run it with docker compose :

docker compose -f oci://ghcr.io/scicatproject/scicatlive:<tag> up -d

Please note that, when running the OCI packages, the configuration options are limited to the ones that do not require changes in the compose.yaml files, as these are included in the package as they are. For example, the v3 backend is available as a separate tag, as it requires changes in the compose.yaml file to be used. For more information on the available configuration options, please refer to the configuration options table .

One can still select the services to run, as described in the select the services section, and make use of the native docker compose overrides by including additional compose files in the command, for example, to only run the backend with modified configurations:

docker compose -f oci://ghcr.io/scicatproject/scicatlive:<tag> -f compose.override.yaml up -d backend

where compose.override.yaml contains the override configuration for example for the backend service:

services:
  backend:
    environment:
      SITE: <CUSTOM_SITE>
configs:
   backend_v4_functional_accounts_json:
       file: <CUSTOM_PATH>/functional_accounts.json

For a more flexible configuration, please refer to the next section on running from the source code.

Running from the source code

When running from the source code, the user has more flexibility in choosing the configuration and features to run, but they need to have a local copy of the repository. The following instructions are for Linux and MacOS users.

Windows specific instructions (click to expand)


⚠ Running this project on Windows is not officialy supported, you should use Windows Subsystem for Linux (WSL).

However, if you want to run it on Windows you have to be careful about:

  • This project makes use of symbolic links, Windows and git for Windows have to be configured to handle them .
  • End of lines, specifically in shell scripts. If you have the git config parameter auto.crlf set to true , git will replace LF by CRLF causing shell scripts and maybe other things to fail.
  • This project uses the variable ${PWD} to ease path resolution in bind mounts. In PowerShell/Command Prompt, the PWD environment variable doesn't exist so you would need to set in manually before running any docker compose command.


  1. Clone the repository
git clone https://github.com/SciCatProject/scicatlive.git
  1. Run with the following command inside the directory
docker compose up -d

Default setup

By running docker compose up -d these steps take place:

  1. the SciCat backend v4 container is created and connected to a mongo DB .
  2. the SciCat frontend container is created and connected to (1).
  3. a reverse proxy container is created and routes traffic to (1) and (2) through localhost subdomains, in the form: http://${service}.localhost . The frontend is available at simply http://localhost .
  4. Some services have additional endpoints that can be explored in SciCatLive which would follow http://${service}.localhost/${prefix} . For example, the backend API can be explored through a Swagger UI at http://backend.localhost/explorer . For more information on the paths used by these routes see the original documentation for these services.

Extra services

SciCat has extra features as part of its core as well as integrating with external services.

SciCat features that extend the backend are:

Services that can be integrated with SciCat are:

To simply enable one or more of these extra services configure them by setting the proper environment variable(s) and/or compose profile(s) from this table .

For a complete guide on how to customise or configure any service, including the default ones, please refer to these sections:

For a guide on how to add a new service, please refer to this section .

Dependencies

Here below we show the dependencies, including the ones of the extra services (if B depends on A , then we visualize it as A --> B ):

graph TD
   subgraph services
      subgraph backend
         backends[v3*/v4*]
      end
      backend --> frontend
      backend --> searchapi
      backend --> landingpage
      backend --> oaipmh
      backend --> jupyter
   end

   proxy -.- services

   %% CSS Styling
   linkStyle 5 marker-end:none

We flag with * the services which have extra internal dependencies, which are not shared.

Select the services

The user can selectively decide the containers to spin up and the dependencies will be resolved accordingly. The available services are in the services folder and are called consistently.

For example, one could decide to only run the backend by running (be aware that this will not run the proxy , so the service will not be available at backend.localhost ):

docker compose up -d backend

(or a list of services, for example, with the proxy docker compose up -d backend proxy )

This will run, from the previous section , (1) and (2) but skip the rest.

Accordingly (click to expand)...
docker compose up -d frontend

Will run, from the previous section , (1), (2) and (4) but skip (5).

And

docker compose --profile search up -d searchapi

Will run, from the previous section , (1) and (2), skip (3) and (4), and add the searchapi service.

Make sure to check the backend compatibility when choosing services and setting docker compose env vars and profiles .

Features

Docker compose env variables

They are used to modify existing services where whenever enabling the feature requires changes in multiple services. They also have the advantage, compared to docker profiles, of not needing to define a new profile when a new combination of features becomes available. To set an env variable for docker compose, either assign it in the shell or change the .env file. To later unset it, either unset it from the shell or assign it an empty value, either in the shell or in the .env file.

For example, to use the Jobs functionality of SciCat change JOBS_ENABLED to true before running your docker compose command or simply export it in the shell. For all env configuration options see this section .

Docker compose profiles

They are used when adding new services or grouping services together (and do not require changes in multiple services). To enable any, run docker compose --profile <PROFILE> up -d , or export the COMPOSE_PROFILES env variable as described by the docker docs . If needed, the user can specify more than one profile in the CLI by using the flag as --profile <PROFILE1> --profile <PROFILE2> .

For example docker compose --profile analysis sets up a jupyter hub with some notebooks for ingesting data into SciCat, as well as the related services (backend, proxy). For more information on profiles available in SciCat live see the following table .

Docker compose profiles and env variables configuration options

Type Env key Value: Service/Feature Default Backend Compatibility Description Other impacted services
profile COMPOSE_PROFILES
  • analysis : jupyter
  • search : searchapi,landingpage,oaipmh
  • '*' : jupyter,searchapi,landingpage,oaipmh
  • '' *
  • analysis: enables additional jupyter notebook with python SciCat SDK installed and example notebooks
  • search: enables a SciCat interface for standardized search and a public interface for published datasets
  • env BE_VERSION
  • v3 : backend/v3
  • v4 : backend/v4
  • v4 as set Sets the BE version to use in (2) of default setup to v3 mongodb,frontend
    env JOBS_ENABLED true : rabbitmq,archivemock (v3 only),jobs feature '' * Creates a RabbitMQ message broker which the BE posts to and the archivemock listens to and enables the frontend features. Archivemock emulates the data long-term archive/retrieve workflow
    env OPENSEARCH_ENABLED true : opensearch, opensearch feature '' v4 Creates an opensearch service and sets the BE to use it for full-text searches
    env LDAP_ENABLED true : ldap auth '' * Creates an LDAP service and sets the BE to use it as authentication backend
    env OIDC_ENABLED true : oidc auth '' * Creates an OIDC identity provider and sets the BE to use it as authentication backend
    env DEV true : backend,frontend,searchapi,archivemock,oaipmh,landingpage in DEV mode '' * The SciCat services' environment is prepared to ease the development in a standardized environment
    env BACKEND_DEV true : backend,archivemock in DEV mode '' * Same as DEV=true but limited to the backend service
    env FRONTEND_DEV true : frontend in DEV mode '' * Same as DEV=true but limited to the frontend service
    env SEARCHAPI_DEV true : searchapi in DEV mode '' * Same as DEV=true but limited to the searchapi service
    env LANDINGPAGE_DEV true : landingpage in DEV mode '' * Same as DEV=true but limited to the landingpage service
    env OAIPMH_DEV true : oaipmh in DEV mode '' * Same as DEV=true but limited to the oaipmh service
    env SCICATLIVE_DEV true : this repo's own docs rendered in the docs service '' * Same as DEV=true but limited to mounting this repository's own documentation in the docs service
    env USER_DOCS_DEV true : external user documentation rendered in the docs service '' * Same as DEV=true but limited to cloning and mounting the external user-documentation repo in the docs service
    env <SERVICE>_HTTPS_URL <URL> : HTTPS termination '' * Requests the TLS certificate for the URL to LetsEncrypt through the proxy
    env DEV_BBACKUP true : bidirectional synchronization of DEV volume '' * Enables DEV bidirectional synchronization between ${PWD}/bbackup/${APP} on the host and the dev volume
    env DEV_AI_STATE <path> : AI coding assistant (e.g. Claude Code) state directory, inside the container /root/.claude * Path used to persist the state of the AI service (Claude Code) across container restarts, backed by a dedicated volume. It is shared across all DEV-mode node services so that the conversation history/config is preserved and visible from any of them
    env TRAEFIK_HTTP_PORT <port> : host port for HTTP traffic 80 * Host port the proxy binds to for HTTP. Change it if port 80 is already in use on your host, or to run multiple SciCatLive stacks side by side. See Traefik ports frontend,backend,keycloak
    env TRAEFIK_HTTPS_PORT <port> : host port for HTTPS traffic 443 * Host port the proxy binds to for HTTPS. Change it if port 443 is already in use on your host, or to run multiple SciCatLive stacks side by side. See Traefik ports frontend,backend,keycloak

    After optionally setting any configuration option, one can still select the services to run as described by the select the services section.

    Computed environment variables

    Some of the env variables above have a default value that is computed from another one - for example BACKEND_DEV should also be enabled whenever DEV=true , and BACKEND_HTTPS_URL falls back to Traefik's local routing ( http://backend.localhost ) when left unset. Rather than duplicating that resolution logic at every place a compose file needs the final value, it is computed once, in a dedicated section at the bottom of .env , into a _ -prefixed variable of the same name (e.g. _BACKEND_DEV , _BACKEND_HTTPS_URL ). Compose files read the _ -prefixed variable, never the plain one.

    These _ -prefixed variables are computed automatically and must not be edited directly - to change a value, set the plain variable above it instead (e.g. BACKEND_DEV=true , not _BACKEND_DEV=true ); leaving the plain variable unset/commented keeps the documented default.

    DEV configuration

    (click to expand)

    To provide a consistent environment where developers can work, the DEV=true option creates the SciCat services (see DEV from the env vars section for the list), but instead of running them, it just creates the base environment that each service requires. For example, for the backend , instead of running the web server, it creates a NODE environment with git where one can develop and run the unit tests. This is useful as often differences in environments create collaboration problems. It should also provide an example of the configuration for running tests. Please refer to the services' README for additional information, or to the Dockerfile CMD of the components' GitHub repo if not specified otherwise. The DEV=true affects the SciCat services only. It's also possible to only run some services in development mode by using their respective variables (eg. BACKEND_DEV=true )

    Please be patient when using DEV as each container sets the env for dev, including the requirements for testing, which might take a little to finish. To see if any special precaution is required to run the tests, refer to the compose.dev.test.yaml file where tests files are referenced and refer to their content. When DEV=true , if you want to run tests when the containers start, you can do so by including the compose.dev.test.yaml compose file.

    docker compose -f compose.yaml -f .github/compose.dev.test.yaml ...
    

    It is very convenient if using VSCode , as, after the docker services are running, one can attach to it and start developing using all VSCode features, including version control and debugging.

    The openapigenerator container is used to generate SDKs for containers when in DEV mode and using the V4 backend . A script is provided to download the generated SDKs in the DEV environment, and each service that needs the SDKs has a script to install them as well, which can be executed running generate_sdk in the shell. The generation is based on the OpenAPI specification of the backend running in scicatlive (make sure to have the backend running whenever calling generate_sdk ), so it is always up to date with the current state of the API, and it is generated in a way that allows to easily install it in the DEV environment. For more details see the openapigenerator README , and for an example of how to use it, see the frontend README .

    When DEV=true (or any of BACKEND_DEV , FRONTEND_DEV , SCICATLIVE_DEV , USER_DOCS_DEV individually), a docs service also becomes available, rendering a live view of the enabled DEV-mode services' own upstream documentation (their docs/ folder), this repository's own top-level documentation, and an external user-documentation repository, using MkDocs. See the docs README for details, including how to add another service to it.

    Please note that entrypoints when DEV=true are only run when the component's container is created for the first time. This is done to avoid clashes with local changes.

    To ease writing DEV configuration, a dev template is provided at ./services/compose.dev.yaml and each component inhearits from it. As you can see in the file ./services/frontend/compose.dev. , setting the componenent specific variables from the relative .env file . ⚠ Docker compose applies a precedence mechanism whenever the same variable is defined in .env files in nested folders, with precedence to the folder where the default COMPOSE_FILE lives. This means that the current template cannot be used in case of nested components, at least for the parts where local variables are used. There is no conflict with variables defined multiple times in .env files at the same level.

    ⚠ To prevent git unpushed changes from being lost when a container is restarted, the work folder of each service, when in DEV mode, is mounted to a docker volume, with naming convention ${COMPOSE_PROJECT_NAME}_<service>_dev . Make sure, to commit and push frequently, especially before removing docker volumes to push the relevant changes.

    ⚠ As the DEV containers pull from upstream/latest, there is no guarantee of their functioning outside of releases. If they fail to start, try, as a first option, to build the image from a tag (e.g. build context ) using the TAG and then git checkout to that tag (e.g. set GITHUB_REPO including the branch using the same syntax and value as the build context). You can achieve this, by setting the GITHUB_REPO env variable in the component .env file (e.g. the frontend env file ) as follows:

    -  GITHUB_REPO=https://github.com/SciCatProject/frontend.git
    +  GITHUB_REPO=https://github.com/SciCatProject/frontend.git#v4.4.1
    

    The repo is checkout at that particular commit only if the docker volume does not yet exist.

    DEV bidirectional synchronization

    Setting DEV_BBACKUP=true in the .env file enables bidirectional synchronization between the DEV volume of each component (e.g. frontend_dev ) and a directory on the host placed at ${PWD}/bbackup/${APP} (e.g. ${PWD}/bbackup/${APP} ). This is sometimes convenient both to have a backup of the volume and to enable the use of additional tools installed on the host, which require file access.

    TLS configuration

    You can enable TLS termination of desired services by setting the <SERVICE>_HTTPS_URL , by setting the full URL, including https:// . The specified HTTPS URL will get a letsencrypt generated certificate through the proxy setting. For more details see the proxy instructions . After setting some URLs, the required changes in dependent services are automatically resolved, as explained for example in the frontend docs . Whenever possible, we use either the docker internal network or the localhost subdomains.

    ⚠ Please make sure to set all required <SERVICE>_HTTPS_URL whenever enabling one, as mixing public URLs and localhost ones might be tricky. See, for example, what is described in the frontend documentation and the backend documentation .

    Traefik ports

    By default the proxy binds to the host's ports 80 (HTTP) and 443 (HTTPS). Set TRAEFIK_HTTP_PORT and/or TRAEFIK_HTTPS_PORT in the .env file whenever those ports are unavailable, for example:

    • another service on your machine (or another SciCatLive stack) is already using 80/443
    • you are running multiple SciCatLive stacks side by side and need each proxy on its own ports
    • you don't have permission to bind to privileged ports (<1024) on your host

    The default local routing ( http://<service>.localhost , used when no <SERVICE>_HTTPS_URL is set) is served through TRAEFIK_HTTP_PORT , so if you change it you must also include the new port when browsing to a service, e.g. http://backend.localhost:8080 . TRAEFIK_HTTPS_PORT only comes into play once a <SERVICE>_HTTPS_URL is configured, as described in TLS configuration .

    Service-specific config

    It can be changed whenever needing to configure a service independently from the others.

    Every service folder (inside the services parent directory) contains its configuration and some instructions, at least for the non-third-party containers.

    For example, to configure the frontend , the user can change any file in the frontend config folder, for which instructions are available in the README file.

    After any configuration change, docker compose up -d must be rerun, to allow loading the changes.

    Entrypoints

    Sometimes, it is useful to run init scripts (entrypoints) before the service starts. For example, for the landingpage composability, it is useful to specify its configuration through multiple JSON files, with different scopes, which are then merged by a init script . For this reason, one can define common entrypoints and service-specific ones (e.g. backend v4 ones ) which can be run inside the container, before the service starts (i.e. before the docker compose command is executed). Whenever these entrypoints are shared between services, it is recommended to place them in an entrypoints folder below the outermost service (e.g. this one ).

    To ease the iterative execution of multiple init scripts, one can leverage the loop_entrypoints utility, which loops alphabetically over /docker-entrypoinst/*.sh and executes each. This is in use in some services (e.g. in the frontend ), so one can add additional init steps by mounting them, one by one, as docker compose configs inside the container in the /docker-entrypoints folder and naming them depending on the desired order (eventually rename the existing ones as well).

    If the service does not support entrypoints yet, one needs to

    (click to expand):
    1. mount the loop_entrypoint.sh as a docker compose config inside the container
    2. mount any service-specific init script as a docker compose config in the container in the folder /docker-entrypoints/*.sh , naming them sequentially, depending on the desired execution order
    3. override the entrypoint field in the service
    4. specify the service command

    See for example the frontend compose file .

    Development

    Some tooling to help while developing or contributing to SciCatLive.

    Running CI locally

    The full CI workflow - changed-file detection, linting , and the docker compose up matrix - can be run locally with nektos/act . lintci depends_on the lint service (see Running linting locally ), so running it also runs the local lint/test tools first:

    docker compose -f .github/compose.lint.yaml run --rm lintci
    

    This is handy for checking whether CI will pass without waiting on it. The JOB env var selects which top-level compose_test.yaml job it runs (defaults to lint ); since test and tests-status both need lint (and test also needs changes ), JOB=tests-status pulls in the whole workflow, including the heavy docker compose up matrix:

    JOB=tests-status docker compose -f .github/compose.lint.yaml run --rm lintci
    

    ⚠ Use JOB=tests-status with caution: it runs the full test matrix - every combination of BE_VERSION , OPENSEARCH_ENABLED , JOBS_ENABLED , LDAP_ENABLED , OIDC_ENABLED and DEV - each spinning up its own docker compose up stack, so it is very resource-heavy (CPU, RAM, disk and network) and can overwhelm a laptop. If it does, consider adding --concurrent-jobs 1 to the lintci command in compose.lint.yaml to run the matrix one job at a time instead of in parallel.

    Running linting locally

    lintci (above) runs lint.yaml 's three jobs as a check, without fixing anything. The checks among them that support auto-fixing - ruff , eslint and markdownlint-cli2 - can instead fix what they find, via the lint service in compose.lint.yaml . The same service also runs publish-oci.js 's unit tests (no fixing, just pass/fail):

    FIX=true docker compose -f .github/compose.lint.yaml run --rm lint
    

    Omit FIX to only report issues without fixing them.

    Developing services in DEV mode

    Setting DEV=true (or a per-service variant, e.g. BACKEND_DEV=true ) boots the SciCat services into a development environment instead of running them normally, so you can develop and test against the same dependencies used in this project. See DEV configuration for the full list of *_DEV variables and how they behave.

    Previewing documentation locally

    Setting SCICATLIVE_DEV=true and starting the docs service renders a live, searchable preview of this repository's own documentation (including this README) using MkDocs:

    SCICATLIVE_DEV=true docker compose up -d docs
    

    See DEV configuration and the docs README for more, including previewing individual DEV-mode services' own documentation or the external user-documentation repository ( USER_DOCS_DEV ).

    Add a new service

    Please note that services should, in general, be defined by their responsibility, rather than by their underlying technology, and should be named so.

    ⚠ When adding a new service, please use docker compose configs for mounting files inside the container, unless the mounted file needs to be modified from within the container, in which case use a bind-mount volume instead, since configs are always mounted read-only. This distinction is mostly relevant for the published OCI packages: bind mounts pointing at directories, or at files that no longer exist on disk, require the user to have those in their local environment, which might not be the case, so they are left untouched by the release workflow ; single-file bind mounts, however, are converted to configs by that same workflow, so they work in the published package too. The generated config is named <service>_<target-path> , with the mount's target path stripped of its leading slash and any remaining non-alphanumeric character replaced by _ (e.g. a proxy service mount targeting /config/traefik.yaml becomes a config named proxy_config_traefik.yaml ).

    Basic

    To add a new service (see the jupyter service for a minimal example):

    1. create a dedicated folder in the services one *
    2. name it as the service
    3. create the compose.yaml file
    4. eventually, add a README.md file in the service
    5. eventually, add the platform field, as described by the supperted OSs
    6. include the reference to (3) to the global compose include list *
    7. eventually, update the main README.md

    * if the service to add is not shared globally, but specific to one particular service or another implementation of the same component, add it to the services folder relative to the affected service, and in (6) add it to its inclusion list. See an example of a service relative services folder here and a relative inclusion list here .

    Supported OS architectures

    Since some images are not built with multi-arch, in particular the SciCat ones, make sure to specify the platform of the service in the compose, when needed, to avoid possible issues when running docker compose up on different platforms, for example on MAC with arm64 architecture. See for example the searchapi compose .

    Advanced

    (click to expand)

    To add a new service, with advanced configuration (see the backend for an extensive example, or/and this PR which added the landingpage ):

    1. follow the steps from the basic section
    2. eventually, include any service, in the service-specific folder which is specific to the service and not shared by other, more general services, e.g. here: ./services/backend/services/ . This folder should also include different versions of the same service, e.g. v3 and v4
    3. eventually, if the service supports ENVs , leverage the include override feature from docker compose. For this:
    4. create a compose.base.yaml file, e.g. ./services/backend/services/v4/compose.base.yaml , which should contain the base configuration, i.e. the one where all ENVs are unset, i.e. the features are disabled
    5. create the ENV-specific (e.g. OPENSEARCH_ENABLED ) compose.<ENV>.yaml file, e.g. backend v4 compose.opensearch.yaml , with the additional/override config, specific to the enabled feature
    6. create a symlink from .empty.yaml to each .compose.<ENV>.yaml , e.g. ./services/backend/services/v4/.compose.opensearch.yaml . This is used whenever the ENV is unset, as described in the next step
    7. use compose.yaml to merge the compose*.yaml files together, making sure to default to .compose.<ENV>.yaml whenever the ENV is not set. See an example ./services/backend/services/v4/compose.yaml . If the ENV overrides are simple, prefer limiting the compose.*.yaml files, as done in the frontend
    8. if the service is another version of an existing one, e.g. v3 and v4 versions of the backend service, add the selective include in the parent compose.yaml, e.g. ./services/backend/compose.yaml
    9. eventually, modify the compose workflow to add the toggle to the matrix (see Running CI locally for how to exercise it without waiting on CI). If the toggle should only run when relevant files changed (as done for the existing opensearch , jobs , ldap , oidc and dev toggles):

      1. add a path group for it, e.g. opensearch , to .github/changed_files.yaml
      2. add a matching <toggle>_values output to the changes job, which turns that group's <group>_all_modified_files output into the matrix values to test, e.g.: opensearch_values: ${{ steps.changed-files.outputs.opensearch_all_modified_files && '["", "opensearch"]' || '[""]' }}
      3. reference that output as the matrix axis in the test job, e.g.: OPENSEARCH_ENABLED: ${{ fromJson(needs.changes.outputs.opensearch_values) }}
      4. only add an exclude rule if the new toggle is genuinely incompatible with another matrix value

      (linting is not part of this matrix: it lives in .github/workflows/lint.yaml , called as a single reusable workflow from the lint job. It has three jobs: static-lint (YAML, JSON, shell, Python, JavaScript and HTML - checks that only ever look at the repo's own committed files), docs-lint (Markdown link checking, Markdown style, and the MkDocs link check), and release-lint (runs publish-oci.js 's unit tests, then dry-runs publish-oci.js and semantic-release , so a broken release pipeline is caught on every PR instead of only when a real release runs). test needs the lint job, so the heavy compose matrix never starts if any of the three lint jobs fails. If the new service introduces a file extension none of these tools already cover, add a step for it to whichever job fits, or a new job if it doesn't. If that tool supports auto-fixing, wire it up the way ruff , eslint and markdownlint-cli2 are: a shared script under .github/lint/ , referenced from both the workflow step and compose.lint.yaml 's lint service - see Running linting locally for how to use it) 7. if the ENV's default should fall back to another variable (e.g. to DEV , or to a localhost URL), compute it once as a _ -prefixed variable instead of duplicating the fallback at each usage site - see Computed environment variables

    10. eventually, add entrypoints for init logics, as described by the section to enable entrypoints , e.g. like ./services/backend/services/v4/compose.base.yaml , including any ENVs specific logic. Remember to set the environment variable in the compose.yaml file.

    General use of SciCat

    To use SciCat, please refer to the original documentation .