🛡️ Security

🗺️ Overview

All core CogStack-NiFi services — including NiFi, Elasticsearch/OpenSearch, Kibana/OpenSearch Dashboards, JupyterHub, NGINX and Gitea — are now deployed with HTTPS enabled by default.
Each component is provisioned with its own X.509 certificates issued by the shared root CA generated via the create_root_ca_cert.sh script.

This ensures full end-to-end encryption across the stack for essential operations, including service-to-service communication and user-facing endpoints.

Security is achieved through:

A unified root Certificate Authority (CA),
Per-service certificate generation and signing scripts,
Environment variable management for secrets and credentials, and
Optional reverse-proxy enforcement via NGINX.

⚠️ Important: Always generate unique certificates and credentials for each deployment.
The repository provides sample certificates for demonstration only.

🧩 Components secured with HTTPS

Service	HTTPS/TLS Enabled	Certificate Location	Script(s) Used
NiFi	✅	`security/certificates/nifi/`	`nifi_toolkit_security.sh`
Elasticsearch / OpenSearch	✅	`security/certificates/elastic/(elasticsearch or opensearch)/`	`create_es_native_certs.sh`, `create_opensearch_node_cert.sh`
Kibana / OpenSearch Dashboards	✅	`security/certificates/elastic/(elasticsearch or opensearch)/`	`create_opensearch_client_admin_certs.sh`
JupyterHub	✅	`security/certificates/root/`	`create_root_ca_cert.sh`
Gitea	✅	`security/certificates/root/`	`create_root_ca_cert.sh`
NGINX	✅	`security/certificates/root/`	`create_root_ca_cert.sh`

📂 Folder structure

The security/ directory centralizes all certificate, credential, and role management for CogStack-NiFi.
Below is the high-level structure with explanations for each sub-folder.

security/
├── certificates/                               # All generated certificates and keystores
│   ├── elastic/                                # Elasticsearch / OpenSearch + Kibana certs
│   ├── nifi/                                   # Apache NiFi certificates (generated via NiFi Toolkit)
│   └── root/                                   # Root CA files and truststores
│               
├── env/                                        # Environment variable definitions for certs and users
│   ├── certificates_*.env                      # Variables controlling certificate generation
│   └── users_*.env                             # Default credentials for each service
│               
├── es_roles/                                   # Role and role mapping definitions for ES / OpenSearch
│   ├── elasticsearch/                          # Native Elasticsearch roles
│   └── opensearch/                             # OpenSearch Security Plugin configs
│           
├── scripts/                                    # Shell utilities for creating certs and credentials
│   ├── create_root_ca_cert.sh                  # Generates the shared root CA (trust anchor)
│   ├── create_es_native_certs.sh               # Elasticsearch node and client certs
│   ├── create_es_native_credentials.sh         # Runs post-deployment to create default Elasticsearch system users and tokens
│   ├── create_opensearch_node_cert.sh          # Generates certificates and JKS stores for each OpenSearch node
│   ├── create_opensearch_admin_certs.sh        # Creates admin + client certificates for OpenSearch Dashboards (Kibana equivalent)
│   ├── create_opensearch_internal_passwords.sh # Generates bcrypt password hashes for OpenSearch internal_users.yml
│   ├── update_opensearch_users.sh              # Updates password hashes for every user in OpenSearch internal_users.yml
│   ├── create_opensearch_users.sh              # Creates OpenSearch internal users and role mappings (manual execution post-startup)
│   ├── nifi_toolkit_security.sh                # Generates NiFi HTTPS certs using NiFi Toolkit (for NiFi < 2.0, no longer used for certs as of 2.0+)
│   ├── nifi_init_create_user_auth.sh           # Bootstraps a temporary NiFi container to create a single-user authentication file
│   ├── nifi_create_single_user_auth.sh         # Helper script executed inside the container to generate NiFi single-user credentials
│   ├── es_native_cert_generator.sh             # Helper called by create_es_native_certs.sh to assemble ES cert bundles
│   └── create_keystore.sh                      # Builds Java KeyStores (JKS) from PEM or PKCS#12 certificates
│
└── templates/                                  # OpenSSL / X.509 configuration templates
    └── ssl-extensions-x509.cnf                 # SAN extensions used across certificate scripts

🏛️ Certificates and Root CA

This section describes the full structure of the security/certificates/ directory and explains how certificates are generated, organized, and used across CogStack-NiFi services.

All certificates originate from the Root Certificate Authority (CA), generated via create_root_ca_cert.sh.

This Root CA signs all service certificates (NiFi, OpenSearch, Kibana, JupyterHub, Gitea, etc.), ensuring consistent trust across the stack, with the exception of ElasticSearch (Native), we use Elastic’s built-in cert generation scripts for it instead.

📂 Certificate directory structure

security/
└── certificates/
    ├── elastic/                                        # Certificates for Elasticsearch / OpenSearch clusters
    │   ├── elasticsearch/                              # Native Elasticsearch certificates
    │   │   ├── elastic-stack-ca.*                      # CA for Elasticsearch (self-signed or derived from root)
    │   │   ├── elasticsearch/                          # Node certificates for Elasticsearch instances
    │   │   │   ├── elasticsearch-1,2,3/ and *-dev/ variants
    │   │   │   │   ├── *.crt, *.key, *.p12             # Node certs for each instance
    │   │   │   │   ├── http-elasticsearch-*.csr/key    # HTTP service certs for HTTPS APIs
    │   │   │   │   ├── sample-elasticsearch.yml        # Example ES configuration
    │   │   │   │   └── README.txt                      # Node-level info
    │   │   ├── elasticsearch-ssl-http.zip              # Bundled certs for HTTP layer
    │   │   ├── es_native_certs_bundle*.zip             # Bundled native ES certs
    │   │   ├── instances.yml                           # Defines node names and hostnames
    │   │   └── kibana/                                 # Certificates for Kibana dashboard
    │   │       ├── sample-kibana.yml
    │   │       └── README.txt
    │   │
    │   └── opensearch/                                 # OpenSearch and OpenSearch Dashboard certs
    │       ├── admin.*, es_kibana_client.*, root-ca.*  # Admin + dashboard + CA certs
    │       ├── elasticsearch/                          # Node certs for OpenSearch nodes
    │       │   ├── elasticsearch-{1,2,3}/              # Per-node certs, keystore/truststore
    │       │   │   ├── *.crt, *.key, *.p12, *.csr  
    │       │   │   ├── elasticsearch-*-keystore.jks    # Keystores for OpenSearch nodes
    │       │   │   ├── elasticsearch-*-truststore.key  # Truststores
    │       │   │   └── http-elasticsearch-*.csr/key    # HTTP layer certs
    │       ├── es_kibana_client.{pem,key,p12,csr}      # Kibana client certs
    │       ├── elastic-stack-ca.*                      # OpenSearch cluster CA
    │       └── root-ca.*                               # Root CA reference for OpenSearch
    │   
    ├── nifi/                                           # NiFi HTTPS and toolkit certificates
    │   ├── nifi.{crt,key,p12,pem,csr}                  # Primary NiFi node certificates
    │   ├── nifi-keystore.jks                           # Java keystore for NiFi server
    │   ├── nifi-truststore.jks                         # Truststore for verifying other services
    │   
    └── root/                                           # Root Certificate Authority (CA)
        ├── root-ca.key, root-ca.pem                    # Private key and public cert
        ├── root-ca.p12, root-ca.keystore.jks           # PKCS#12 and Java Keystore formats
        ├── root-ca-truststore.jks                      # Truststore for client-side verification
        └── root-ca.csr, root-ca.srl                    # Certificate signing request and serial

⚙️ Environment configuration

All certificate-generation scripts source variables from .env files under security/env/:

File	Description
`certificates_general.env`	Global Root CA options (CN, expiry, key size).
`certificates_elasticsearch.env`	Node names, SAN hostnames, version control for ES/OS.
`certificates_nifi.env`	NiFi keystore/truststore names and passwords.
`users_*.env`	Default credentials used by generation scripts.

📜 openssl-x509.conf

Set up a reusable certificate config to define SANs and subject. This is used globally for all services except ES native. Feel free to add custom DNS Note that the settings here impact services that rely on Distinguished Names (DN) attributes for authentication.

# =========================================================================================
# 📜 OpenSSL X.509 v3 Extensions Configuration
# For: Root CA and Node/Client Certificates
# =========================================================================================

[v3_ca]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:TRUE
keyUsage = critical, keyCertSign, cRLSign
subjectAltName=DNS:nifi,DNS:elasticsearch-1,DNS:elasticsearch-2,DNS:elasticsearch-3,DNS:cogstack,DNS:*.cogstack

[v3_leaf]
basicConstraints = critical, CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer

[alt_names]
DNS.1 = nifi
DNS.2 = nifi-nginx
DNS.3 = elasticsearch-1
DNS.4 = elasticsearch-2
DNS.5 = elasticsearch-3
DNS.6 = ocr-service
DNS.7 = ocr-service-text-only
DNS.8 = medcat-trainer-nginx
DNS.9 = medcat-trainer-ui 
DNS.10 = nlp-medcat-service-production
DNS.11 = nlp-medcat-service-production-deid
DNS.12 = cogstack-kibana
DNS.13 = cogstack-cohort
DNS.14 = cogstack-elasticsearch-1
DNS.15 = cogstack-elasticsearch-2
DNS.16 = cogstack-elasticsearch-3
DNS.17 = cogstack-nifi
DNS.18 = cogstack-nifi-nginx
DNS.19 = cogstack-auth-service
DNS.20 = cogstack
DNS.21 = *.cogstack
DNS.22 = localhost
IP.1 = 127.0.0.1
email.1 = admin@cogstack.net

[req]
default_bits       = 4096
string_mask        = utf8only
prompt = no
distinguished_name = req_distinguished_name
x509_extensions    = v3_leaf
default_md         = sha256

[req_distinguished_name]
CN = cogstack
C  = UK
ST = London
L  = UK
O  = cogstack
OU = cogstack
CN = cogstack

💡 Tip:
Always reload environment variables before running any script:
cd ../deploy
source export_env_vars.sh
cd ../security
or manually if you just want to test out one file:
source file.env

🛠️ Generation workflow

Generate Root CA

cd security/scripts
bash create_root_ca_cert.sh

Generate service certificates

# Elasticsearch
bash create_es_native_certs.sh

# OpenSearch
bash create_opensearch_node_cert.sh elasticsearch-1 elasticsearch-2 elasticsearch-3

# Kibana / Dashboards
bash create_opensearch_client_admin_certs.sh

# NiFi
bash nifi_toolkit_security.sh (not needed as of version 2.0+, use only for NiFi versions < 2.0) make sure to change $NIFI_TOOLKIT_VERSION env var in `../deploy/nifi.env`.

(Optional) Create custom JKS keystores

bash create_keystore.sh mycert.pem mystore.jks mypassword

Re-export environment variables and restart services

cd ../deploy
source export_env_vars.sh
make start-<SERVICE_NAME>

🧠 Best practices

Do not commit private keys (*.key, *.p12, *.jks) to version control.
Back up the Root CA files securely — they’re your trust anchor.
Rotate certificates regularly (every 2 years) or whenever hostnames change.
Use unique CN/SANs per environment (dev, staging, prod).
Verify certificate chains before deployment (e.g):

  openssl verify -CAfile security/certificates/root/root-ca.pem security/certificates/elastic/opensearch/elasticsearch/elasticsearch-1/elasticsearch-1.crt

🗄️ Security for CogStack provided other services

General services TLS configuration

All services in cogstack that are not listed in deploy/services.ymland use TLS will be described on this page. For services that are still part of the stack but in dervices/<SERVICE_NAME> (again, service not also present in deploy/services.yml) the TLS setup is handled differently, and the setup is described in each service’s README.md. Generally, most should just use the root-ca certs from security/certificates/root/.

Gitea TLS Configuration

This section describes how Gitea is secured using the shared Root Certificate Authority (CA) generated by create_root_ca_cert.sh.

Unlike other services (such as NiFi or Elastic), Gitea does not require its own dedicated certificate pair or an NGINX reverse proxy. It operates directly with the Root CA to provide HTTPS encryption and mutual trust within the CogStack-NiFi stack.

📁 Certificate source

All certificates used by Gitea originate from:

security/certificates/root/

File	Purpose
`root-ca.pem`	Public CA certificate used by Gitea for HTTPS trust
`root-ca.key`	Root CA private key (used only when generating new certificates)
`root-ca.p12`	Optional PKCS#12 keystore (not required by Gitea)

🧠 Notes

The Root CA (root-ca.pem) is shared across all CogStack services for internal TLS trust.
You do not need to create a new gitea.crt or gitea.key; the Root CA cert/key pair is sufficient.
Ensure root-ca.key remains private and is not committed to version control.
The same CA also secures NiFi, ElasticSearch, OpenSearch, Kibana, and JupyterHub.

✅ Verification

To confirm Gitea is serving HTTPS correctly:

curl -vk --cacert ./security/certificates/root/root-ca.pem https://gitea.local:2222/

You should see a valid TLS handshake and an HTTP 200 response.

🔐 NiFi TLS & Admin Access Setup (with NGINX)

This guide documents how to configure TLS and certificate-based admin access for Apache NiFi behind NGINX. For background on how certificates are generated, see Certificates and Root CA.

📁 Folder Structure

security/certificates
├── elastic
├── nifi
│   ├── nifi-keystore.jks
│   ├── nifi-truststore.jks
│   ├── nifi.crt
│   ├── nifi.csr
│   ├── nifi.key
│   ├── nifi.p12
│   └── nifi.pem
└── root
    ├── root-ca-keystore.jks
    ├── root-ca-truststore.jks
    ├── root-ca.crt
    ├── root-ca.csr
    ├── root-ca.key
    ├── root-ca.p12
    ├── root-ca.pem
    └── root-ca.srl

For securing Apache NiFi endpoints with certificates, see the official documentation.

Before starting the NiFi container:

(optional if already done) run create_root_ca_cert.sh to generate root CA certs used across services.
set nifi.sensitive.props.key to a stable value (minimum 12 characters).

Example (nifi/conf/nifi.properties):

nifi.security.keystorePasswd=example-keystore-password
nifi.security.keyPasswd=example-key-password
nifi.security.truststore=./conf/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=example-truststore-password

Setting up access via user account (single user credentials)

Default:

username: admin
password: cogstackNiFi

login-identity-providers.xml in nifi/conf/ stores the account settings.
to generate credentials inside the container:

/opt/nifi/nifi-current/bin/nifi.sh set-single-user-credentials USERNAME PASSWORD

alternatively:
- set credentials in security/env/users_nifi.env
- stop NiFi (docker stop cogstack-nifi)
- run bash security/scripts/nifi_init_create_user_auth.sh

URL: https://localhost:8443/nifi/login

`nifi-nginx`

NGINX provides secure reverse-proxy access to NiFi at:

https://localhost:8443/nifi

Reference: services/nginx/config/nginx.conf.template.

🔐 `authorizers.xml` initial admin identity

Ensure your certificate identity is present in NiFi authorizers.xml:

<property name="Initial Admin Identity">C=UK, ST=London, L=UK, O=cogstack, OU=cogstack, CN=cogstack</property>

If you use DN mapping (nifi.security.identity.mapping.pattern.dn), ensure the mapped identity matches the configured admin identity.

🌐 `nifi.properties` proxy settings

nifi.web.proxy.host=localhost:8443,nginx.local:8443
nifi.web.proxy.context.path=/nifi
nifi.security.identity.mapping.pattern.dn=^.*?CN=(.*?)(,|$)

🌍 NGINX reverse proxy example (NiFi)

server {
    listen 8443 ssl;
    server_name nginx.local;

    ssl_certificate           /certificates/nifi/nifi.pem;
    ssl_certificate_key       /certificates/nifi/nifi.key;
    ssl_client_certificate    /certificates/root/root-ca.pem;
    ssl_trusted_certificate   /certificates/root/root-ca.pem;

    location / {
        proxy_set_header Host nifi;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-ProxyHost $host;
        proxy_set_header X-ProxyPort 8443;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-ProxyScheme $scheme;
        proxy_pass https://nifi;
    }

    location /nifi {
        proxy_set_header Host nifi;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-ProxyHost $host;
        proxy_set_header X-ProxyPort 8443;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-ProxyScheme $scheme;
        proxy_pass https://nifi;
    }

    location ^~ /nifi-api/ {
        proxy_set_header Host nifi;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-ProxyHost $host;
        proxy_set_header X-ProxyPort 8443;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-ProxyScheme $scheme;
        proxy_pass https://nifi/nifi-api/;
    }
}

✅ Final checklist

[x] Certificate CN matches the NiFi admin identity
[x] NGINX uses the expected certificates and root CA
[x] nifi.web.proxy.* settings match exposed URL and context
[x] Clear cookies if browser UI shows Anonymous

🧪 Test connectivity

curl -vk --cert ./nifi.pem --key ./nifi.key https://localhost:8443/nifi-api/flow/about

🛠 Troubleshooting

Verify Initial Admin Identity in authorizers.xml matches the certificate identity.
Recreate authorizations.xml if permissions are stuck.
Restart NiFi after configuration updates.

Maintained by: admin@cogstack.org

🌐 Elasticsearch / OpenSearch Security

This section describes how to secure both Elasticsearch (native) and OpenSearch clusters used in the CogStack-NiFi stack, including certificate setup, user management, and role configuration.

All related certificates are stored in security/certificates/elastic/, and are generated from the shared Root CA created via create_root_ca_cert.sh.

🔒 Overview

Both Elasticsearch and OpenSearch deployments require:

TLS certificates for all nodes and HTTPS endpoints,
secure credentials for built-in and custom users,
properly configured roles and role mappings.

Certificates and credentials are generated using the scripts provided in security/scripts/ and are controlled through the .env files under security/env/.

📄 Environment files used

All scripts reference the following environment configuration files:

File	Purpose
`certificates_elasticsearch.env`	Hostnames, instance names, and certificate parameters for ES / OpenSearch nodes
`certificates_general.env`	Root CA configuration
`users_elasticsearch.env`	Internal user credentials

Reload them before running any security-related script:

cd ../deploy
source export_env_vars.sh
cd ../security

⚙️ Version variable

Set the ES/OS version in deploy/elasticsearch.env before launching containers:

ELASTICSEARCH_VERSION=opensearch
# or
ELASTICSEARCH_VERSION=elasticsearch

This ensures the correct certificate directory (elasticsearch or opensearch) is mounted into containers.

🧩 Common certificate layout

Certificate naming and folder structure are consistent across both ES and OpenSearch:

security/certificates/elastic/
├── elasticsearch/
│   ├── elastic-stack-ca.crt.pem
│   ├── elastic-stack-ca.key.pem
│   ├── elasticsearch/
│   │   ├── elasticsearch-{1,2,3}/
│   │   │   ├── http-elasticsearch-*.crt
│   │   │   ├── http-elasticsearch-*.key
│   │   │   ├── http-elasticsearch-*.p12
│   │   │   ├── elasticsearch-*.crt
│   │   │   ├── elasticsearch-*.key
│   │   │   └── elasticsearch-*.p12
│   └── kibana/
│       ├── sample-kibana.yml
│       └── README.txt
└── opensearch/
    ├── admin.*, es_kibana_client.*, elastic-stack-ca.*, root-ca.*
    └── elasticsearch/{1,2,3}/...

Each version has its own generation scripts, but they all depend on the same .env configuration and naming patterns.

🏗️ Generating certificates

Elasticsearch (native)

To generate certificates for Elasticsearch:

bash ./create_es_native_certs.sh

This script creates all required node and HTTP certificates under:

security/certificates/elastic/elasticsearch/elasticsearch-{1,2,3}/

The script uses variables such as:

ES_INSTANCE_NAME_* — Node names (match ELASTICSEARCH_NODE_*_NAME in /deploy/elasticsearch.env)
ES_INSTANCE_ALTERNATIVE_*_NAME — Alternative hostnames
ES_HOSTNAMES — List of all node hostnames
ES_CLIENT_SUBJ_ALT_NAMES / ES_NODE_SUBJ_ALT_NAMES — Additional domain aliases for SAN fields

Make sure the environment variables are set correctly before running the script.

OpenSearch

For OpenSearch nodes:

bash ./create_opensearch_node_cert.sh elasticsearch-1 elasticsearch-2 elasticsearch-3

Then generate the admin and client certificates:

bash ./create_opensearch_client_admin_certs.sh

This produces:

File	Purpose
`admin.crt`, `admin.key.pem`	Admin dashboard certificate
`es_kibana_client.pem`, `es_kibana_client.key`	Client certificate for Kibana/OpenDashboard
`*.jks`	Node keystores/truststores for HTTPS and inter-node encryption

The resulting certificates are placed in:

security/certificates/elastic/opensearch/

📁 Kibana / OpenDashboard certificates

Platform	Required Certificates	Source Folder
Kibana	`elasticsearch-{1,2,3}.crt`, `elasticsearch-{1,2,3}.key`, `elastic-stack-ca.crt.pem`	`security/certificates/elastic/elasticsearch/`
OpenDashboard (OpenSearch)	`admin.crt`, `admin.key.pem`, `es_kibana_client.pem`, `es_kibana_client.key`, `elastic-stack-ca.crt.pem`	`security/certificates/elastic/opensearch/`

All certificate references in services/kibana/config/opensearch.yml or services.yml must point to these locations.

🔐 Users and roles

OpenSearch

Edit security/es_roles/opensearch/internal_users.yml to define users.

Optionally generate password hashes:

bash ./create_opensearch_internal_passwords.sh

Apply changes by recreating containers:

docker compose down -v
docker compose up -d

Populate tenants, roles, users, and role mappings:
```
cd security/scripts
bash create_opensearch_users.sh elasticsearch-1 --use-ssl
```
- <es_hostname> is the OpenSearch node hostname (for this stack, usually elasticsearch-1).
- --use-ssl switches the script endpoint from http to https (recommended for this stack).
- Run the script from security/scripts/ because it loads env files via relative paths.

`create_opensearch_users.sh` reference

Script: security/scripts/create_opensearch_users.sh
Usage:

bash create_opensearch_users.sh <es_hostname> [--use-ssl]

Required inputs are loaded from:

deploy/general.env
deploy/elasticsearch.env
security/env/certificates_elasticsearch.env
security/env/certificates_general.env
security/env/users_elasticsearch.env

What it creates/updates:

Tenants: nifi_tenant, cogstack_tenant
Roles: cogstack_ingest, cogstack_access
Internal users: cogstack_user, cogstack_pipeline, nifi
Role mappings for cogstack_access and cogstack_ingest
Passwords for built-in users: logstash, kibanaro, readall, snapshotrestore

Verification example:

curl -k -u admin:${ELASTIC_PASSWORD} \
  https://elasticsearch-1:9200/_opendistro/_security/api/roles/cogstack_ingest

Troubleshooting:

If you see authentication failures, confirm ELASTIC_PASSWORD in security/env/users_elasticsearch.env matches the running OpenSearch admin password.
If you see 404 on security API paths, your OpenSearch version may require _plugins/_security/api/... instead of _opendistro/_security/api/....
The script is idempotent (PUT calls), so it can be re-run safely after credential or role changes.

OpenSearch includes default roles (admin, kibanaserver, readall, snapshotrestore, etc.) — always change their passwords after first run.

`update_opensearch_users.sh` reference

Use this script when you need to rotate passwords for every user already defined in security/es_roles/opensearch/internal_users.yml, including the reserved admin user.

The script updates password hashes in internal_users.yml using OpenSearch’s hash.sh, then applies only the internalusers config with securityadmin.sh. It does not create tenants, roles, role mappings, or new internal users. For those tasks, use create_opensearch_users.sh.

Script: security/scripts/update_opensearch_users.sh
Usage:

cd security/scripts

# Dry run: validates env vars and generates hashes, but does not write files.
./update_opensearch_users.sh <opensearch_container_name>

# Update internal_users.yml and push the internal users config to OpenSearch.
./update_opensearch_users.sh <opensearch_container_name> --apply

# Update only the local YAML file; do not run securityadmin.sh.
./update_opensearch_users.sh <opensearch_container_name> --apply --skip-securityadmin

To find the container name:

docker ps --format '{{.Names}}' | grep elasticsearch

Required inputs are loaded from:

deploy/elasticsearch.env
security/env/certificates_elasticsearch.env
security/env/certificates_general.env
security/env/users_elasticsearch.env

Password env var resolution:

OpenSearch user	Env var used
`admin`	`ADMIN_PASSWORD` or `OPENSEARCH_ADMIN_PASSWORD` if set, otherwise `ELASTIC_PASSWORD`
`kibanaserver`	`KIBANA_PASSWORD` when `KIBANA_USER=kibanaserver`
`logstash`	`ES_LOGSTASH_PASS`
`kibanaro`	`ES_KIBANARO_PASS`
`readall`	`ES_READALL_PASS`
`snapshotrestore`	`ES_SNAPSHOTRESTORE_PASS`
`anomalyadmin`	`ANOMALYADMIN_PASSWORD`
`new-user`	`NEW_USER_PASSWORD`
Any other user	`<USERNAME>_PASSWORD`, uppercased with non-alphanumeric characters converted to underscores

Example: a user named data-loader expects DATA_LOADER_PASSWORD unless it is covered by one of the explicit mappings above.

Recommended workflow:

Edit the relevant password values in security/env/users_elasticsearch.env.
Run the script without --apply to validate all users and generate hashes.
Run the script with --apply to back up and update internal_users.yml, then apply it to the running OpenSearch cluster.
Re-run any dependent setup scripts or update dependent service credentials if the rotated user is used elsewhere.

The script writes a timestamped backup next to the source file:

security/es_roles/opensearch/internal_users.yml.bak.YYYYMMDDHHMMSS

Important behavior:

The dry run still needs a running OpenSearch container because hashes are generated with the version of hash.sh bundled in that container.
securityadmin.sh -f ... -t internalusers replaces the internal users config stored in the OpenSearch security index. Keep every required internal user in internal_users.yml before applying.
The running cluster is not updated if you use --skip-securityadmin; only the local YAML file changes.
If you rotate admin, update any scripts or services that still authenticate with admin:${ELASTIC_PASSWORD}.

Verification example after rotating admin, from the repository root:

source security/env/users_elasticsearch.env
curl -k -u "admin:${ADMIN_PASSWORD:-$ELASTIC_PASSWORD}" \
  https://elasticsearch-1:9200/_plugins/_security/api/account

OpenSearch Dashboards post-login setup (Global tenant + workspace)

After your first login to https://localhost:5601 (default: admin / admin):

Open the user menu (top-right) and select Switch tenant.
Select Global, then apply the change.
Open Stack Management → Workspaces (or the workspace switcher in the header) and click Create workspace.
Enter a workspace name (for example cogstack-main) and keep it in the Global tenant.
Save, then create data views and dashboards inside this workspace.

Use Global for shared dashboards and saved objects.
Private tenant content is user-specific and not visible to other users.

If tenant switching or workspace creation is not available in the UI, verify these settings in services/kibana/config/opensearch.yml:

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.preferred: ["Global"]
workspace.enabled: true

Elasticsearch (native)

Run after containers start:

bash ./create_es_native_credentials.sh

This script creates system users, roles, and a service account token for Kibana.

You can modify credentials in security/env/users_elasticsearch.env.

New roles created:

ingest — for NiFi and pipeline ingestion (cogstack_*, nifi_* indices)
cogstack_access — read-only access to cogstack_* and nifi_*

New users:

nifi → ingest
cogstack_user → cogstack_access

⚠️ Notes

The security/certificates/ folder is also mounted inside NiFi so NiFi processors can access ES/OS securely without restarting.
For OpenSearch role details, see the OpenSearch Security Plugin documentation.
For Elasticsearch, refer to the official Elastic Security docs.

✅ Verification

To verify HTTPS access and trust:

curl -vk --cacert ./root-ca.pem https://elasticsearch-1:9200

To check inter-node encryption (inside a container):

openssl s_client -connect elasticsearch-1:9300 -CAfile ./root-ca.pem

🛡️ Security

🗺️ Overview

🧩 Components secured with HTTPS

📂 Folder structure

🏛️ Certificates and Root CA

📂 Certificate directory structure

⚙️ Environment configuration

📜 openssl-x509.conf

🛠️ Generation workflow

🧠 Best practices

🗄️ Security for CogStack provided other services

General services TLS configuration

Gitea TLS Configuration

📁 Certificate source

🧠 Notes

✅ Verification

🔐 NiFi TLS & Admin Access Setup (with NGINX)

📁 Folder Structure

Setting up access via user account (single user credentials)

nifi-nginx

🔐 authorizers.xml initial admin identity

🌐 nifi.properties proxy settings

🌍 NGINX reverse proxy example (NiFi)

✅ Final checklist

🧪 Test connectivity

🛠 Troubleshooting

🌐 Elasticsearch / OpenSearch Security

🔒 Overview

📄 Environment files used

⚙️ Version variable

🧩 Common certificate layout

🏗️ Generating certificates

Elasticsearch (native)

OpenSearch

📁 Kibana / OpenDashboard certificates

🔐 Users and roles

OpenSearch

create_opensearch_users.sh reference

update_opensearch_users.sh reference

OpenSearch Dashboards post-login setup (Global tenant + workspace)

Elasticsearch (native)

⚠️ Notes

✅ Verification

`nifi-nginx`

🔐 `authorizers.xml` initial admin identity

🌐 `nifi.properties` proxy settings

`create_opensearch_users.sh` reference

`update_opensearch_users.sh` reference