SciCatLive ¶
Get set up with an instance of SciCat to explore the metadata catalog. SciCatlive provides a flexible and easy way to learn about SciCat and its features for people who are looking to integrate SciCat into their environment. For a user guide please see original documentation .
This project requires docker and docker compose. The docker version must be later than 2.29.0 to support this project.
First stable version ¶
Release
v3.0
is the first stable and reviewed version of SciCatLive.
Steps ¶
Installing via the OCI packages ¶
Starting from release
v3.10
, SciCatLive is published as OCI packages, which can be found on the GitHub
Container Registry
.
For every new release, these tags are added to the registry:
-
latest(|version): the latest release with default configuration; -
latest(|version)-full: the latest release with all features enabled; -
latest(|version)-v3: the latest release with the v3 backend and minimal configuration. -
latest(|version)-v3-full: the latest release with the v3 backend and all features enabled.
To use the OCI packages, simply pull the desired tag from the registry and run it with
docker compose
:
docker compose -f oci://ghcr.io/scicatproject/scicatlive:<tag> up -d
Please note that, when running the OCI packages, the configuration options are limited to the ones that do not require
changes in the
compose.yaml
files, as these are included in the package as they are. For example, the
v3
backend
is available as a separate tag, as it requires changes in the
compose.yaml
file to be used. For more information on
the available configuration options, please refer to the
configuration options table
.
One can still select the services to run, as described in the select the services section, and make use of the native docker compose overrides by including additional compose files in the command, for example, to only run the backend with modified configurations:
docker compose -f oci://ghcr.io/scicatproject/scicatlive:<tag> -f compose.override.yaml up -d backend
where
compose.override.yaml
contains the override configuration for example for the backend service:
services:
backend:
environment:
SITE: <CUSTOM_SITE>
configs:
backend_v4_functional_accounts_json:
file: <CUSTOM_PATH>/functional_accounts.json
For a more flexible configuration, please refer to the next section on running from the source code.
Running from the source code ¶
When running from the source code, the user has more flexibility in choosing the configuration and features to run, but they need to have a local copy of the repository. The following instructions are for Linux and MacOS users.
Windows specific instructions (click to expand)
Running this project on Windows is not officialy supported, you should use Windows Subsystem for Linux (WSL).
However, if you want to run it on Windows you have to be careful about:
- This project makes use of symbolic links, Windows and git for Windows have to be configured to handle them .
-
End of lines, specifically in shell scripts. If you have the git config parameter
auto.crlfset totrue, git will replace LF by CRLF causing shell scripts and maybe other things to fail. -
This project uses the variable
${PWD}to ease path resolution in bind mounts. In PowerShell/Command Prompt, thePWDenvironment variable doesn't exist so you would need to set in manually before running anydocker composecommand.
- Clone the repository
git clone https://github.com/SciCatProject/scicatlive.git
- Run with the following command inside the directory
docker compose up -d
Default setup ¶
By running
docker compose up -d
these steps take place:
- the SciCat backend v4 container is created and connected to a mongo DB .
- the SciCat frontend container is created and connected to (1).
-
a reverse
proxy
container is created and routes traffic to (1) and (2) through localhost
subdomains, in the form:
http://${service}.localhost. The frontend is available at simplyhttp://localhost. -
Some services have additional endpoints that can be explored in SciCatLive which would follow
http://${service}.localhost/${prefix}. For example, the backend API can be explored through a Swagger UI athttp://backend.localhost/explorer. For more information on the paths used by these routes see the original documentation for these services.
Extra services ¶
SciCat has extra features as part of its core as well as integrating with external services.
SciCat features that extend the backend are:
- Jobs - this mechanism posts to a message broker . In v3 it can then trigger down stream processes . To use this a RabbitMQ server is enabled.
- OpenSearch - creates an opensearch service to provide full text search in the backend.
Services that can be integrated with SciCat are:
- LDAP - authentication and authorization from an LDAP server
- OIDC - authentication and authorization using an OIDC provider
- SearchAPI - for better free text search in the metadata based on the PANOSC search-api
- LandingPage - a public interface for published datasets: landingpage
- JupyterHub - Adds an instance of JupyterHub which demonstrates ingestion and extraction of metadata using pyscicat .
- OAIPMH - a service for published metadata via the OAI-PMH protocol : oaipmh
To simply enable one or more of these extra services configure them by setting the proper environment variable(s) and/or compose profile(s) from this table .
For a complete guide on how to customise or configure any service, including the default ones, please refer to these sections:
- manually select the services
- use docker compose env variables to enable features (supported values from this table )
- use docker compose profiles to enable extra services (supported values from this table )
- modify the service-specific config to customise specific services
- add entrypoints to control startup logic
For a guide on how to add a new service, please refer to this section .
Dependencies ¶
Here below we show the dependencies, including the ones of the
extra services
(if
B
depends on
A
,
then we visualize it as
A --> B
):
graph TD
subgraph services
subgraph backend
backends[v3*/v4*]
end
backend --> frontend
backend --> searchapi
backend --> landingpage
backend --> oaipmh
backend --> jupyter
end
proxy -.- services
%% CSS Styling
linkStyle 5 marker-end:none
We flag with
*
the services which have extra internal dependencies, which are not shared.
Select the services ¶
The user can selectively decide the containers to spin up and the dependencies will be resolved accordingly. The available services are in the services folder and are called consistently.
For example, one could decide to only run the
backend
by running (be aware that this will not run the
proxy
, so the
service will not be available at
backend.localhost
):
docker compose up -d backend
(or a list of services, for example, with the proxy
docker compose up -d backend proxy
)
This will run, from the previous section , (1) and (2) but skip the rest.
Accordingly (click to expand)...
docker compose up -d frontend
Will run, from the previous section , (1), (2) and (4) but skip (5).
And
docker compose --profile search up -d searchapi
Will run, from the
previous section
, (1) and (2), skip (3) and (4), and add the
searchapi
service.
Make sure to check the
backend compatibility
when
choosing services and setting
docker compose env vars and profiles
.
Features ¶
Docker compose env variables ¶
They are used to modify existing services where whenever enabling the feature requires changes in multiple services. They also have the advantage, compared to docker profiles, of not needing to define a new profile when a new combination of features becomes available. To set an env variable for docker compose, either assign it in the shell or change the .env file. To later unset it, either unset it from the shell or assign it an empty value, either in the shell or in the .env file.
For example, to use the Jobs functionality of SciCat change
JOBS_ENABLED
to true before running your
docker compose
command or simply export it in the shell. For all env configuration options see this
section
.
Docker compose profiles ¶
They are used when adding new services or grouping services together (and do not require changes in multiple services).
To enable any, run
docker compose --profile <PROFILE> up -d
, or export the
COMPOSE_PROFILES
env variable as
described by the
docker docs
. If needed,
the user can specify more than one profile in the CLI by using the flag as
--profile <PROFILE1> --profile <PROFILE2>
.
For example
docker compose --profile analysis
sets up a jupyter hub with some notebooks for ingesting data into
SciCat, as well as the related services (backend, proxy). For more information on profiles available in SciCat live see
the following
table
.
Docker compose profiles and env variables configuration options ¶
| Type | Env key | Value: Service/Feature | Default | Backend Compatibility | Description | Other impacted services |
|---|---|---|---|---|---|---|
| profile |
COMPOSE_PROFILES
|
analysis
: jupyter
search
: searchapi,landingpage,oaipmh
'*'
: jupyter,searchapi,landingpage,oaipmh
|
''
|
* |
|
|
| env |
BE_VERSION
|
v3
: backend/v3
v4
: backend/v4
|
v4
|
as set | Sets the BE version to use in (2) of default setup to v3 | mongodb,frontend |
| env |
JOBS_ENABLED
|
true
: rabbitmq,archivemock (v3 only),jobs feature
|
''
|
* | Creates a RabbitMQ message broker which the BE posts to and the archivemock listens to and enables the frontend features. Archivemock emulates the data long-term archive/retrieve workflow | |
| env |
OPENSEARCH_ENABLED
|
true
: opensearch, opensearch feature
|
''
|
v4 | Creates an opensearch service and sets the BE to use it for full-text searches | |
| env |
LDAP_ENABLED
|
true
: ldap auth
|
''
|
* | Creates an LDAP service and sets the BE to use it as authentication backend | |
| env |
OIDC_ENABLED
|
true
: oidc auth
|
''
|
* | Creates an OIDC identity provider and sets the BE to use it as authentication backend | |
| env |
DEV
|
true
: backend,frontend,searchapi,archivemock,oaipmh,landingpage in DEV mode
|
''
|
* | The SciCat services' environment is prepared to ease the development in a standardized environment | |
| env |
BACKEND_DEV
|
true
: backend,archivemock in DEV mode
|
''
|
* |
Same as
DEV=true
but limited to the backend service
|
|
| env |
FRONTEND_DEV
|
true
: frontend in DEV mode
|
''
|
* |
Same as
DEV=true
but limited to the frontend service
|
|
| env |
SEARCHAPI_DEV
|
true
: searchapi in DEV mode
|
''
|
* |
Same as
DEV=true
but limited to the searchapi service
|
|
| env |
LANDINGPAGE_DEV
|
true
: landingpage in DEV mode
|
''
|
* |
Same as
DEV=true
but limited to the landingpage service
|
|
| env |
OAIPMH_DEV
|
true
: oaipmh in DEV mode
|
''
|
* |
Same as
DEV=true
but limited to the oaipmh service
|
|
| env |
SCICATLIVE_DEV
|
true
: this repo's own docs rendered in the
docs
service
|
''
|
* |
Same as
DEV=true
but limited to mounting this repository's own documentation in the docs service
|
|
| env |
USER_DOCS_DEV
|
true
: external user documentation rendered in the
docs
service
|
''
|
* |
Same as
DEV=true
but limited to cloning and mounting the external user-documentation repo in the docs service
|
|
| env |
<SERVICE>_HTTPS_URL
|
<URL>
: HTTPS termination
|
''
|
* | Requests the TLS certificate for the URL to LetsEncrypt through the proxy | |
| env |
DEV_BBACKUP
|
true
: bidirectional synchronization of DEV volume
|
''
|
* | Enables DEV bidirectional synchronization between ${PWD}/bbackup/${APP} on the host and the dev volume | |
| env |
DEV_AI_STATE
|
<path>
: AI coding assistant (e.g. Claude Code) state directory, inside the container
|
/root/.claude
|
* | Path used to persist the state of the AI service (Claude Code) across container restarts, backed by a dedicated volume. It is shared across all DEV-mode node services so that the conversation history/config is preserved and visible from any of them | |
| env |
TRAEFIK_HTTP_PORT
|
<port>
: host port for HTTP traffic
|
80
|
* | Host port the proxy binds to for HTTP. Change it if port 80 is already in use on your host, or to run multiple SciCatLive stacks side by side. See Traefik ports | frontend,backend,keycloak |
| env |
TRAEFIK_HTTPS_PORT
|
<port>
: host port for HTTPS traffic
|
443
|
* | Host port the proxy binds to for HTTPS. Change it if port 443 is already in use on your host, or to run multiple SciCatLive stacks side by side. See Traefik ports | frontend,backend,keycloak |
After optionally setting any configuration option, one can still select the services to run as described by the select the services section.
Computed environment variables ¶
Some of the env variables above have a default value that is computed from another one - for example
BACKEND_DEV
should also be enabled whenever
DEV=true
, and
BACKEND_HTTPS_URL
falls back to Traefik's local routing
(
http://backend.localhost
) when left unset. Rather than duplicating that resolution logic at every place a compose
file needs the final value, it is computed once, in a dedicated section at the bottom of
.env
, into a
_
-prefixed variable of the same name (e.g.
_BACKEND_DEV
,
_BACKEND_HTTPS_URL
). Compose files read the
_
-prefixed variable, never the plain one.
These
_
-prefixed variables are
computed automatically and must not be edited directly
- to change a value, set
the plain variable above it instead (e.g.
BACKEND_DEV=true
, not
_BACKEND_DEV=true
); leaving the plain variable
unset/commented keeps the documented default.
DEV configuration ¶
(click to expand)
To provide a consistent environment where developers can work, the
DEV=true
option creates the SciCat services (see
DEV from the
env vars section
for the list), but instead of running them, it just
creates the base environment that each service requires. For example, for the
backend
, instead of running the web
server, it creates a NODE environment with
git
where one can develop and run the unit tests. This is useful as often
differences in environments create collaboration problems. It should also provide an example of the configuration for
running tests. Please refer to the services' README for additional information, or to the Dockerfile
CMD
of the
components' GitHub repo if not specified otherwise. The
DEV=true
affects the SciCat services only.
It's also possible to only run some services in development mode by using their respective variables (eg.
BACKEND_DEV=true
)
Please be patient when using DEV as each container sets the env for dev, including the requirements for testing, which
might take a little to finish. To see if any special precaution is required to run the tests, refer to the
compose.dev.test.yaml
file where tests files are referenced and refer to their content.
When DEV=true
, if you want to run tests when the containers start, you can do so by including the
compose.dev.test.yaml
compose file.
docker compose -f compose.yaml -f .github/compose.dev.test.yaml ...
It is very convenient if using VSCode , as, after the docker services are running, one can attach to it and start developing using all VSCode features, including version control and debugging.
The
openapigenerator container
is used to generate SDKs
for containers when in DEV mode and using the
V4 backend
.
A script is provided to download the generated SDKs in the DEV environment, and each service that needs the SDKs has a
script to install them as well, which can be executed running
generate_sdk
in the shell. The generation is based on
the OpenAPI specification of the backend running in scicatlive (make sure to have the backend running whenever calling
generate_sdk
), so it is always up to date with the current state of the API, and it is generated in a way that allows
to easily install it in the DEV environment. For more details see the
openapigenerator README
, and for an example of how
to use it, see the
frontend README
.
When
DEV=true
(or any of
BACKEND_DEV
,
FRONTEND_DEV
,
SCICATLIVE_DEV
,
USER_DOCS_DEV
individually), a
docs
service also becomes available, rendering a live view of the enabled DEV-mode services' own
upstream documentation (their
docs/
folder), this repository's own top-level documentation, and an external
user-documentation repository, using MkDocs. See the
docs README
for details, including
how to add another service to it.
Please note that
entrypoints
when
DEV=true
are only run when the component's container is created for
the first time. This is done to avoid clashes with local changes.
To ease writing DEV configuration, a dev template is provided at
./services/compose.dev.yaml
and each component inhearits from it. As you can see in the
file
./services/frontend/compose.dev.
, setting the componenent specific
variables from the relative
.env file
.
Docker compose applies a
precedence mechanism
whenever the same variable is defined in
.env
files in nested folders, with precedence to the folder where the default
COMPOSE_FILE
lives. This means that the current template cannot be used in case of nested components, at least for the
parts where local variables are used. There is no conflict with variables defined multiple times in
.env
files at the
same level.
To prevent git unpushed changes from being lost when a container is restarted, the work folder of each
service, when in DEV mode, is mounted to a docker volume, with naming convention
${COMPOSE_PROJECT_NAME}_<service>_dev
. Make sure, to commit and push frequently, especially before removing docker
volumes to push the relevant changes.
As the DEV containers pull from upstream/latest, there is no guarantee of their functioning outside of
releases. If they fail to start, try, as a first option, to build the image from a tag (e.g.
build context
) using the
TAG
and then git checkout to that tag
(e.g. set
GITHUB_REPO
including the branch using the same syntax and value as
the build context). You can achieve this, by setting the
GITHUB_REPO
env variable in the component
.env
file (e.g.
the
frontend env file
) as follows:
- GITHUB_REPO=https://github.com/SciCatProject/frontend.git
+ GITHUB_REPO=https://github.com/SciCatProject/frontend.git#v4.4.1
The repo is checkout at that particular commit only if the docker volume does not yet exist.
DEV bidirectional synchronization ¶
Setting
DEV_BBACKUP=true
in the
.env
file enables bidirectional synchronization between the DEV volume of
each component (e.g.
frontend_dev
) and a directory on the host placed at
${PWD}/bbackup/${APP}
(e.g.
${PWD}/bbackup/${APP}
). This is sometimes convenient both to have a backup of the volume and to enable the use of
additional tools installed on the host, which require file access.
TLS configuration ¶
You can enable TLS termination of desired services by setting the
<SERVICE>_HTTPS_URL
, by setting the full URL,
including
https://
. The specified HTTPS URL will get a
letsencrypt
generated certificate through the proxy setting.
For more details see the
proxy instructions
. After setting some URLs, the required changes
in dependent services are automatically resolved, as explained for example in the
frontend docs
. Whenever possible, we use either the docker internal network or the
localhost subdomains.
Please make sure to set all required
<SERVICE>_HTTPS_URL
whenever enabling one, as mixing public URLs and
localhost
ones might be tricky. See, for example, what is described in the
frontend documentation
and the
backend documentation
.
Traefik ports ¶
By default the
proxy
binds to the host's ports 80 (HTTP) and 443 (HTTPS). Set
TRAEFIK_HTTP_PORT
and/or
TRAEFIK_HTTPS_PORT
in the
.env
file whenever those ports are unavailable, for example:
- another service on your machine (or another SciCatLive stack) is already using 80/443
- you are running multiple SciCatLive stacks side by side and need each proxy on its own ports
- you don't have permission to bind to privileged ports (<1024) on your host
The default local routing (
http://<service>.localhost
, used when no
<SERVICE>_HTTPS_URL
is set) is served through
TRAEFIK_HTTP_PORT
, so if you change it you must also include the new port when browsing to a service, e.g.
http://backend.localhost:8080
.
TRAEFIK_HTTPS_PORT
only comes into play once a
<SERVICE>_HTTPS_URL
is configured,
as described in
TLS configuration
.
Service-specific config ¶
It can be changed whenever needing to configure a service independently from the others.
Every service folder (inside the services parent directory) contains its configuration and some instructions, at least for the non-third-party containers.
For example, to configure the frontend , the user can change any file in the frontend config folder, for which instructions are available in the README file.
After any configuration change,
docker compose up -d
must be rerun, to allow loading the changes.
Entrypoints ¶
Sometimes, it is useful to run init scripts (entrypoints) before the service starts. For example, for the
landingpage
composability, it is useful to specify its configuration through multiple JSON files, with different scopes, which are
then merged by a
init script
. For this reason, one can define
common entrypoints
and service-specific ones (e.g.
backend v4 ones
) which can be run inside the container, before the
service starts (i.e. before the docker compose
command
is executed). Whenever these entrypoints are shared between
services, it is recommended to place them in an
entrypoints
folder below the outermost service (e.g.
this one
).
To ease the iterative execution of multiple init scripts, one can leverage the
loop_entrypoints
utility, which loops alphabetically over
/docker-entrypoinst/*.sh
and executes each. This is in use in some services (e.g. in the
frontend
), so one can add additional init steps by mounting them, one by one, as
docker compose configs
inside the container in
the
/docker-entrypoints
folder and naming them depending on the desired order (eventually rename the existing ones
as well).
If the service does not support entrypoints yet, one needs to ¶
(click to expand):
- mount the loop_entrypoint.sh as a docker compose config inside the container
-
mount any service-specific init script as a
docker compose config
in the container in the folder
/docker-entrypoints/*.sh, naming them sequentially, depending on the desired execution order -
override the
entrypointfield in the service -
specify the service
command
See for example the frontend compose file .
Development ¶
Some tooling to help while developing or contributing to SciCatLive.
Running CI locally ¶
The full
CI workflow
- changed-file detection,
linting
,
and the
docker compose up
matrix - can be run locally with
nektos/act
.
lintci
depends_on
the
lint
service (see
Running linting locally
), so running it also runs
the local lint/test tools first:
docker compose -f .github/compose.lint.yaml run --rm lintci
This is handy for checking whether CI will pass without waiting on it. The
JOB
env var selects which top-level
compose_test.yaml
job it runs (defaults to
lint
); since
test
and
tests-status
both
need
lint
(and
test
also needs
changes
),
JOB=tests-status
pulls in the whole workflow, including the heavy
docker compose up
matrix:
JOB=tests-status docker compose -f .github/compose.lint.yaml run --rm lintci
Use
JOB=tests-status
with caution: it runs the full
test
matrix - every combination of
BE_VERSION
,
OPENSEARCH_ENABLED
,
JOBS_ENABLED
,
LDAP_ENABLED
,
OIDC_ENABLED
and
DEV
- each spinning up its own
docker compose up
stack, so it is very resource-heavy (CPU, RAM, disk and network) and can overwhelm a laptop.
If it does, consider adding
--concurrent-jobs 1
to the
lintci
command in
compose.lint.yaml
to run the matrix one job at a time instead of in parallel.
Running linting locally ¶
lintci
(above) runs
lint.yaml
's three jobs as a check, without fixing anything.
The checks among them that support auto-fixing -
ruff
,
eslint
and
markdownlint-cli2
- can instead fix what
they find, via the
lint
service in
compose.lint.yaml
. The same service also runs
publish-oci.js
's unit tests (no fixing, just pass/fail):
FIX=true docker compose -f .github/compose.lint.yaml run --rm lint
Omit
FIX
to only report issues without fixing them.
Developing services in DEV mode ¶
Setting
DEV=true
(or a per-service variant, e.g.
BACKEND_DEV=true
) boots the SciCat services into a development
environment instead of running them normally, so you can develop and test against the same dependencies used in
this project. See
DEV configuration
for the full list of
*_DEV
variables and how they behave.
Previewing documentation locally ¶
Setting
SCICATLIVE_DEV=true
and starting the
docs
service renders a live, searchable preview
of this repository's own documentation (including this README) using MkDocs:
SCICATLIVE_DEV=true docker compose up -d docs
See
DEV configuration
and the
docs README
for more, including
previewing individual DEV-mode services' own documentation or the external user-documentation repository
(
USER_DOCS_DEV
).
Add a new service ¶
Please note that services should, in general, be defined by their responsibility, rather than by their underlying technology, and should be named so.
When adding a new service, please use
docker compose configs
for mounting files inside the container, unless the mounted file needs to be modified from within the container, in
which case use a bind-mount volume instead, since configs are always mounted read-only. This distinction is mostly
relevant for the published OCI packages: bind mounts pointing at directories, or at files that no longer exist on
disk, require the user to have those in their local environment, which might not be the case, so they are left
untouched by the
release workflow
; single-file bind mounts, however, are
converted to configs by that same workflow, so they work in the published package too. The generated config is named
<service>_<target-path>
, with the mount's target path stripped of its leading slash and any remaining non-alphanumeric
character replaced by
_
(e.g. a
proxy
service mount targeting
/config/traefik.yaml
becomes a config named
proxy_config_traefik.yaml
).
Basic ¶
To add a new service (see the jupyter service for a minimal example):
- create a dedicated folder in the services one *
- name it as the service
-
create the
compose.yamlfile -
eventually, add a
README.mdfile in the service - eventually, add the platform field, as described by the supperted OSs
- include the reference to (3) to the global compose include list *
- eventually, update the main README.md
* if the service to add is not shared globally, but specific to one particular service or another implementation of the
same component, add it to the
services
folder relative to the affected service, and in (6) add it to its inclusion
list. See an example of a service relative
services folder here
and a
relative inclusion list here
.
Supported OS architectures ¶
Since some images are not built with multi-arch, in particular the SciCat ones, make sure to specify the platform of the
service in the compose, when needed, to avoid possible issues when running
docker compose up
on different platforms,
for example on MAC with arm64 architecture. See for example the
searchapi compose
.
Advanced ¶
(click to expand)
To add a new service, with advanced configuration (see the backend for an extensive example, or/and this PR which added the landingpage ):
- follow the steps from the basic section
- eventually, include any service, in the service-specific folder which is specific to the service and not shared by other, more general services, e.g. here: ./services/backend/services/ . This folder should also include different versions of the same service, e.g. v3 and v4
- eventually, if the service supports ENVs , leverage the include override feature from docker compose. For this:
-
create a
compose.base.yamlfile, e.g. ./services/backend/services/v4/compose.base.yaml , which should contain thebaseconfiguration, i.e. the one where all ENVs are unset, i.e. the features are disabled -
create the ENV-specific (e.g.
OPENSEARCH_ENABLED)compose.<ENV>.yamlfile, e.g. backend v4 compose.opensearch.yaml , with the additional/override config, specific to the enabled feature -
create a symlink from
.empty.yaml
to each
.compose.<ENV>.yaml, e.g. ./services/backend/services/v4/.compose.opensearch.yaml . This is used whenever theENVis unset, as described in the next step -
use
compose.yamlto merge thecompose*.yamlfiles together, making sure to default to.compose.<ENV>.yamlwhenever theENVis not set. See an example ./services/backend/services/v4/compose.yaml . If the ENV overrides are simple, prefer limiting thecompose.*.yamlfiles, as done in the frontend -
if the service is another version of an existing one, e.g. v3 and v4 versions of the
backendservice, add the selective include in the parent compose.yaml, e.g. ./services/backend/compose.yaml -
eventually, modify the compose workflow to add the toggle to the matrix (see Running CI locally for how to exercise it without waiting on CI). If the toggle should only run when relevant files changed (as done for the existing
opensearch,jobs,ldap,oidcanddevtoggles):-
add a path group for it, e.g.
opensearch, to .github/changed_files.yaml -
add a matching
<toggle>_valuesoutput to thechangesjob, which turns that group's<group>_all_modified_filesoutput into the matrix values to test, e.g.:opensearch_values: ${{ steps.changed-files.outputs.opensearch_all_modified_files && '["", "opensearch"]' || '[""]' }} -
reference that output as the matrix axis in the
testjob, e.g.:OPENSEARCH_ENABLED: ${{ fromJson(needs.changes.outputs.opensearch_values) }} - only add an exclude rule if the new toggle is genuinely incompatible with another matrix value
(linting is not part of this matrix: it lives in .github/workflows/lint.yaml , called as a single reusable workflow from the
lintjob. It has three jobs:static-lint(YAML, JSON, shell, Python, JavaScript and HTML - checks that only ever look at the repo's own committed files),docs-lint(Markdown link checking, Markdown style, and the MkDocs link check), andrelease-lint(runs publish-oci.js 's unit tests, then dry-runspublish-oci.jsandsemantic-release, so a broken release pipeline is caught on every PR instead of only when a real release runs).testneedsthelintjob, so the heavy compose matrix never starts if any of the three lint jobs fails. If the new service introduces a file extension none of these tools already cover, add a step for it to whichever job fits, or a new job if it doesn't. If that tool supports auto-fixing, wire it up the wayruff,eslintandmarkdownlint-cli2are: a shared script under .github/lint/ , referenced from both the workflow step and compose.lint.yaml 'slintservice - see Running linting locally for how to use it) 7. if the ENV's default should fall back to another variable (e.g. toDEV, or to alocalhostURL), compute it once as a_-prefixed variable instead of duplicating the fallback at each usage site - see Computed environment variables -
add a path group for it, e.g.
-
eventually, add entrypoints for init logics, as described by the section to enable entrypoints , e.g. like ./services/backend/services/v4/compose.base.yaml , including any ENVs specific logic. Remember to set the environment variable in the compose.yaml file.
General use of SciCat ¶
To use SciCat, please refer to the original documentation .