Skip to content

Marketplace Core — Design Specification

Approved 2026-03-28. Covers Sub-project 4 (Marketplace Core) plus prerequisites: OpenAPI wiring, Org/IAM hierarchy, billing, licensing, plugin engine, OPA, Keycloak changes, and two built-in modules (GitHub connector, GDPR policy pack).

1. Architecture Overview

1.1 Four Backend Modules

The marketplace is split into 4 focused backend modules following SOLID principles:

Module Responsibility
marketplace Catalog browsing, module install/configure/uninstall, .substrate bundle upload
billing Payment provider abstraction (Stripe), tier subscriptions, invoices
licensing ECDSA license signing/verification, tier enforcement, offline license import
plugins Plugin registry, dynamic loading, sync/evaluate triggers

1.2 Module Interaction

Frontend
  │
  ├── marketplace API ──→ catalog, install, configure, upload
  ├── billing API     ──→ payment, invoices, usage
  ├── plugins API     ──→ sync triggers, evaluate triggers
  │
  └── All check ──────→ licensing API (entitlement verification, tier limits)

1.3 Call Flow — Purchase + Install

1. User browses    → marketplace.list_catalog()
2. Admin purchases → billing.process_payment() → licensing.create_entitlement()
3. User installs   → marketplace.install_module() → licensing.verify_entitlement()
4. User configures → marketplace.configure_installation() → plugins.validate_config()
5. Module activates→ plugins.load_module() → licensing.verify_license()
6. Module runs     → plugins.sync() or plugins.evaluate()

1.4 Air-Gapped Flow

1. Admin uploads .substrate file → marketplace.upload_bundle()
2. Backend verifies signature + hashes → registers in catalog
3. Admin uploads license.jwt file → licensing.import_license()
4. User installs from local catalog → same flow as steps 3-6 above

1.5 Built-in Bundles

Shipped with the app in backend/bundles/, auto-registered on first boot:

backend/bundles/
├── connector-github-1.0.0.substrate
└── policy-pack-gdpr-1.0.0.substrate

2. Org/IAM Hierarchy

2.1 Structure

Organization (license tier, billing account)
├── Project 1 (scoped workspace)
│   ├── Team Alpha
│   │   ├── User Group: "Backend Engineers"
│   │   │   ├── alice (role: Developer)
│   │   │   └── bob (role: SRE)
│   │   └── User Group: "QA"
│   │       └── carol (role: QA)
│   └── Team Beta
│       └── ...
├── Project 2
│   └── ...
└── Billing
    ├── License tier
    ├── Module entitlements
    └── Payment method

2.2 Rules

  • A user can belong to multiple projects and teams with different roles in each
  • User groups are functional groupings within a team (Backend Engineers, QA, SRE, etc.)
  • The 10 canonical roles (Developer, Admin, SRE, etc.) are assigned per team context
  • org_role is org-level: admin | member | billing
  • Org admin controls billing + module purchases
  • Non-admins can browse and request modules, admins approve/deny
  • Module installations are per-org (all projects in the org can use installed modules)

2.3 License Tiers

Tier Projects Users Billing
Free 1 10 None (still interacts with marketplace for free modules)
Organization 5 Capped (tier-based) Monthly or Yearly
Enterprise Unlimited Capped (negotiated) Monthly or Yearly

Module purchases are one-time per module per org, separate from tier billing.

2.4 Tier Enforcement

Check When Enforced by
User count <= tier limit User added to org licensing module
Project count <= tier limit Project created licensing module
Module entitlement exists Module install licensing module
Org admin role Purchase/billing action billing module

2.5 Keycloak Mapping

  • Organization → Keycloak top-level group (/orgs/acme-corp)
  • Project → Sub-group (/orgs/acme-corp/projects/platform-v2)
  • Team → Sub-group (/orgs/acme-corp/projects/platform-v2/teams/alpha)
  • User Group → Sub-group (/orgs/acme-corp/projects/platform-v2/teams/alpha/backend-engineers)
  • Role → Keycloak group role mapping at the team level

2.6 New JWT Claims

Added to substrate-claims scope:

{
  "org_id": "uuid",
  "org_slug": "acme-corp",
  "org_role": "admin",
  "projects": [
    {
      "id": "uuid",
      "slug": "platform-v2",
      "teams": [
        {
          "id": "uuid",
          "slug": "alpha",
          "role": "Developer",
          "user_group": "backend-engineers"
        }
      ]
    }
  ]
}

3. Data Model

3.1 PostgreSQL Migrations

V5__organizations_and_teams.sql:

CREATE TABLE organizations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL UNIQUE,
    slug VARCHAR(100) NOT NULL UNIQUE,
    settings JSONB DEFAULT '{}',
    tier VARCHAR(20) NOT NULL DEFAULT 'free'
        CHECK (tier IN ('free', 'organization', 'enterprise')),
    max_users INTEGER NOT NULL DEFAULT 10,
    max_projects INTEGER NOT NULL DEFAULT 1,
    created_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE org_projects (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    slug VARCHAR(100) NOT NULL,
    description TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    UNIQUE(org_id, slug)
);

CREATE TABLE org_teams (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    project_id UUID NOT NULL REFERENCES org_projects(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    slug VARCHAR(100) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT now(),
    UNIQUE(project_id, slug)
);

CREATE TABLE org_user_groups (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    team_id UUID NOT NULL REFERENCES org_teams(id) ON DELETE CASCADE,
    name VARCHAR(255) NOT NULL,
    slug VARCHAR(100) NOT NULL,
    description TEXT,
    UNIQUE(team_id, slug)
);

CREATE TABLE org_memberships (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    user_id VARCHAR(255) NOT NULL,
    org_role VARCHAR(20) NOT NULL DEFAULT 'member'
        CHECK (org_role IN ('admin', 'member', 'billing')),
    joined_at TIMESTAMPTZ DEFAULT now(),
    UNIQUE(org_id, user_id)
);

CREATE TABLE team_memberships (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    team_id UUID NOT NULL REFERENCES org_teams(id) ON DELETE CASCADE,
    user_id VARCHAR(255) NOT NULL,
    user_group_id UUID REFERENCES org_user_groups(id),
    role VARCHAR(50) NOT NULL,
    joined_at TIMESTAMPTZ DEFAULT now(),
    UNIQUE(team_id, user_id)
);

V6__marketplace_modules.sql:

CREATE TABLE marketplace_modules (
    id VARCHAR(100) PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    version VARCHAR(20) NOT NULL,
    type VARCHAR(50) NOT NULL CHECK (type IN ('data_connector', 'policy_pack')),
    category VARCHAR(100),
    pricing VARCHAR(20) NOT NULL CHECK (pricing IN ('free', 'paid')),
    price_cents INTEGER DEFAULT 0,
    description TEXT,
    author VARCHAR(255),
    manifest JSONB NOT NULL,
    sha256 VARCHAR(64) NOT NULL,
    icon_url TEXT,
    config_schema JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

V7__module_installations.sql:

CREATE TABLE module_installations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    module_id VARCHAR(100) NOT NULL REFERENCES marketplace_modules(id),
    installed_by VARCHAR(255) NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'installed'
        CHECK (status IN ('installed', 'configured', 'active', 'disabled', 'uninstalled')),
    config JSONB DEFAULT '{}',
    version_installed VARCHAR(20) NOT NULL,
    installed_at TIMESTAMPTZ DEFAULT now(),
    configured_at TIMESTAMPTZ,
    updated_at TIMESTAMPTZ DEFAULT now(),
    UNIQUE(org_id, module_id)
);

CREATE TABLE module_requests (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    module_id VARCHAR(100) NOT NULL REFERENCES marketplace_modules(id),
    requested_by VARCHAR(255) NOT NULL,
    reason TEXT,
    status VARCHAR(20) NOT NULL DEFAULT 'pending'
        CHECK (status IN ('pending', 'approved', 'denied')),
    reviewed_by VARCHAR(255),
    reviewed_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ DEFAULT now()
);

V8__billing.sql:

CREATE TABLE billing_accounts (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE UNIQUE,
    stripe_customer_id VARCHAR(255),
    billing_email VARCHAR(255),
    billing_cycle VARCHAR(20) CHECK (billing_cycle IN ('monthly', 'yearly')),
    payment_method_last4 VARCHAR(4),
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE transactions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    billing_account_id UUID NOT NULL REFERENCES billing_accounts(id),
    module_id VARCHAR(100) REFERENCES marketplace_modules(id),
    amount_cents INTEGER NOT NULL,
    currency VARCHAR(3) NOT NULL DEFAULT 'USD',
    stripe_payment_intent_id VARCHAR(255),
    status VARCHAR(20) NOT NULL DEFAULT 'pending'
        CHECK (status IN ('pending', 'completed', 'failed', 'refunded')),
    created_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE invoices (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    billing_account_id UUID NOT NULL REFERENCES billing_accounts(id),
    amount_cents INTEGER NOT NULL,
    currency VARCHAR(3) NOT NULL DEFAULT 'USD',
    period_start TIMESTAMPTZ,
    period_end TIMESTAMPTZ,
    stripe_invoice_id VARCHAR(255),
    pdf_url TEXT,
    status VARCHAR(20) NOT NULL DEFAULT 'draft'
        CHECK (status IN ('draft', 'open', 'paid', 'void')),
    created_at TIMESTAMPTZ DEFAULT now()
);

V9__licensing.sql:

CREATE TABLE org_licenses (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE UNIQUE,
    tier VARCHAR(20) NOT NULL CHECK (tier IN ('free', 'organization', 'enterprise')),
    max_users INTEGER NOT NULL DEFAULT 10,
    max_projects INTEGER NOT NULL DEFAULT 1,
    license_token TEXT,
    issued_at TIMESTAMPTZ DEFAULT now(),
    expires_at TIMESTAMPTZ,
    status VARCHAR(20) NOT NULL DEFAULT 'active'
        CHECK (status IN ('active', 'expired', 'revoked'))
);

CREATE TABLE module_entitlements (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    module_id VARCHAR(100) NOT NULL REFERENCES marketplace_modules(id),
    license_token TEXT NOT NULL,
    transaction_id UUID REFERENCES transactions(id),
    granted_at TIMESTAMPTZ DEFAULT now(),
    expires_at TIMESTAMPTZ,
    status VARCHAR(20) NOT NULL DEFAULT 'active'
        CHECK (status IN ('active', 'revoked')),
    UNIQUE(org_id, module_id)
);

V10__connector_sync_and_policy_eval.sql:

CREATE TABLE connector_sync_runs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    installation_id UUID NOT NULL REFERENCES module_installations(id) ON DELETE CASCADE,
    started_at TIMESTAMPTZ DEFAULT now(),
    completed_at TIMESTAMPTZ,
    status VARCHAR(20) NOT NULL DEFAULT 'running'
        CHECK (status IN ('running', 'completed', 'failed')),
    items_synced INTEGER DEFAULT 0,
    errors JSONB DEFAULT '[]',
    next_run_at TIMESTAMPTZ
);

CREATE TABLE policy_evaluation_runs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    installation_id UUID NOT NULL REFERENCES module_installations(id) ON DELETE CASCADE,
    evaluated_at TIMESTAMPTZ DEFAULT now(),
    scope JSONB NOT NULL,
    input_summary JSONB,
    result VARCHAR(20) NOT NULL CHECK (result IN ('compliant', 'non_compliant', 'error')),
    violations_found INTEGER DEFAULT 0,
    duration_ms INTEGER
);

3.2 Neo4j Schema Additions

// Organization hierarchy
CREATE CONSTRAINT org_id IF NOT EXISTS FOR (o:Organization) REQUIRE o.id IS UNIQUE;
CREATE CONSTRAINT org_name IF NOT EXISTS FOR (o:Organization) REQUIRE o.name IS UNIQUE;
CREATE CONSTRAINT project_id IF NOT EXISTS FOR (p:Project) REQUIRE p.id IS UNIQUE;
CREATE CONSTRAINT orgteam_id IF NOT EXISTS FOR (t:OrgTeam) REQUIRE t.id IS UNIQUE;
CREATE CONSTRAINT usergroup_id IF NOT EXISTS FOR (g:UserGroup) REQUIRE g.id IS UNIQUE;
CREATE CONSTRAINT module_id IF NOT EXISTS FOR (m:MarketplaceModule) REQUIRE m.id IS UNIQUE;

// Relationships
// (:Project)-[:BELONGS_TO]->(:Organization)
// (:OrgTeam)-[:PART_OF]->(:Project)
// (:UserGroup)-[:WITHIN]->(:OrgTeam)
// (:User)-[:MEMBER_OF {role, since}]->(:OrgTeam)
// (:User)-[:IN_GROUP]->(:UserGroup)
// (:Organization)-[:INSTALLED {status, since}]->(:MarketplaceModule)
// (:MarketplaceModule)-[:PROVIDES]->(:Policy)
// (:MarketplaceModule)-[:CONNECTS_TO {type}]->(:ExternalSource)

// Connector-produced nodes (GitHub connector)
CREATE CONSTRAINT repo_id IF NOT EXISTS FOR (r:Repository) REQUIRE r.id IS UNIQUE;
CREATE CONSTRAINT workflow_id IF NOT EXISTS FOR (w:Workflow) REQUIRE w.id IS UNIQUE;
CREATE CONSTRAINT package_id IF NOT EXISTS FOR (p:Package) REQUIRE p.id IS UNIQUE;
CREATE CONSTRAINT ghproject_id IF NOT EXISTS FOR (p:GitHubProject) REQUIRE p.id IS UNIQUE;

// (:Repository)-[:OWNED_BY]->(:Organization)
// (:GitHubPages)-[:HOSTED_BY]->(:Repository)
// (:GitHubProject)-[:MANAGED_BY]->(:Organization)
// (:Workflow)-[:DEFINED_IN]->(:Repository)
// (:Package)-[:PUBLISHED_FROM]->(:Repository)
// (:PolicyEvaluation)-[:EVALUATED]->(:Service)
// (:PolicyEvaluation)-[:USING]->(:Policy)
// (:PolicyEvaluation)-[:FOUND]->(:Violation)

3.3 Redis Key Design

# Marketplace
marketplace:catalog                          → cached catalog JSON (TTL 5m)
marketplace:module:{module_id}               → module detail cache (TTL 5m)

# Licensing
license:org:{org_id}                         → org license summary (TTL 1h)
license:entitlement:{org_id}:{module_id}     → entitlement valid? (TTL 1h)

# Plugin state
plugin:status:{installation_id}              → running | idle | error
plugin:lock:{installation_id}                → sync mutex (TTL 30m)

# Billing
billing:org:{org_id}:usage                   → current seat/project count (TTL 15m)

4. .substrate Bundle Format

4.1 Structure

connector-github-1.0.0.substrate     ← tar.gz with .substrate extension
├── manifest.json
├── signature.json
├── license.jwt                      ← only for paid modules, absent for free
└── module/
    ├── __init__.py
    ├── connector.py                 ← entry point (or pack.py for policy packs)
    ├── requirements.txt             ← optional extra pip deps
    └── policies/                    ← only for policy packs
        ├── data_classification.rego
        ├── retention.rego
        ├── consent.rego
        └── cross_border.rego

4.2 manifest.json

{
  "id": "connector-github",
  "name": "GitHub Data Connector",
  "version": "1.0.0",
  "type": "data_connector",
  "category": "source_control",
  "pricing": "free",
  "description": "Ingest repos, pages, projects v2, actions, packages from GitHub.",
  "author": "Substrate",
  "entry_point": "module.connector:GitHubConnector",
  "config_schema": {
    "type": "object",
    "properties": {
      "github_token": { "type": "string", "format": "password" },
      "org_name": { "type": "string" },
      "sync_interval_minutes": { "type": "integer", "default": 60 },
      "include_repos": { "type": "array", "items": { "type": "string" } },
      "exclude_repos": { "type": "array", "items": { "type": "string" } }
    },
    "required": ["github_token", "org_name"]
  },
  "capabilities": ["repos", "pages", "projects_v2", "actions", "packages"],
  "min_substrate_version": "0.2.0",
  "file_hashes": {
    "module/__init__.py": "sha256:abc...",
    "module/connector.py": "sha256:def..."
  }
}

4.3 signature.json

{
  "algorithm": "ECDSA-P256-SHA256",
  "signed_at": "2026-03-28T00:00:00Z",
  "signer": "substrate-release-authority",
  "manifest_hash": "sha256:...",
  "signature": "base64-encoded-ECDSA-signature"
}

4.4 Verification Flow

1. Extract .substrate archive (tar.gz)
2. Read manifest.json
3. Hash manifest.json → compare against signature.json.manifest_hash
4. Verify ECDSA signature using Substrate public key
5. Hash each file in module/ → compare against manifest.file_hashes
6. If paid: verify license.jwt signature + expiry + module_id match
7. All pass → register in marketplace_modules table
8. Any fail → reject with specific error

4.5 GDPR Policy Pack Bundle

policy-pack-gdpr-1.0.0.substrate
├── manifest.json
├── signature.json
└── module/
    ├── __init__.py
    ├── pack.py
    └── policies/
        ├── data_classification.rego
        ├── retention.rego
        ├── consent.rego
        └── cross_border.rego

manifest.json for GDPR:

{
  "id": "policy-pack-gdpr",
  "name": "GDPR Compliance Pack",
  "version": "1.0.0",
  "type": "policy_pack",
  "category": "compliance",
  "pricing": "free",
  "description": "Evaluate GDPR compliance: data classification, retention, consent, cross-border transfers.",
  "author": "Substrate",
  "entry_point": "module.pack:GDPRPolicyPack",
  "config_schema": {
    "type": "object",
    "properties": {
      "eu_regions": {
        "type": "array",
        "items": { "type": "string" },
        "default": ["eu-west-1", "eu-central-1", "eu-north-1", "eu-south-1"]
      },
      "max_retention_days": { "type": "integer", "default": 2555 },
      "require_dpo": { "type": "boolean", "default": true }
    }
  },
  "capabilities": ["data_classification", "retention", "consent", "cross_border"],
  "policies": ["GDPR-001", "GDPR-002", "GDPR-003", "GDPR-004"],
  "requires_opa": true,
  "min_substrate_version": "0.2.0",
  "file_hashes": {
    "module/__init__.py": "sha256:...",
    "module/pack.py": "sha256:...",
    "module/policies/data_classification.rego": "sha256:...",
    "module/policies/retention.rego": "sha256:...",
    "module/policies/consent.rego": "sha256:...",
    "module/policies/cross_border.rego": "sha256:..."
  }
}

5. API Endpoints

5.1 Marketplace Module (/api/v1/marketplace)

Method Path Auth Description
GET /marketplace/catalog any user List catalog (filter by type, category, pricing)
GET /marketplace/catalog/{module_id} any user Module detail
GET /marketplace/installations any user List org's installed modules
GET /marketplace/installations/{id} any user Installation detail
POST /marketplace/installations org admin Install a module
PATCH /marketplace/installations/{id}/configure any user Configure module
PATCH /marketplace/installations/{id}/status org admin Enable/disable module
DELETE /marketplace/installations/{id} org admin Uninstall module
POST /marketplace/requests any user Request a module (non-admin)
GET /marketplace/requests any user List requests (admin: all, user: own)
PATCH /marketplace/requests/{id} org admin Approve/deny request
POST /marketplace/upload org admin Upload .substrate bundle (air-gapped)

5.2 Billing Module (/api/v1/billing)

Method Path Auth Description
GET /billing/account org admin/billing Billing config
PUT /billing/account org admin Configure billing
POST /billing/purchase org admin Purchase module (one-time)
GET /billing/transactions org admin/billing Purchase history
GET /billing/invoices org admin/billing Invoice list
GET /billing/invoices/{id}/pdf org admin/billing Download invoice
GET /billing/usage any user Seat/project usage vs tier limits
POST /billing/webhooks/stripe Stripe signature Webhook handler

5.3 Licensing Module (/api/v1/licensing)

Method Path Auth Description
GET /licensing/license org admin Current org platform license
POST /licensing/license/import org admin Upload offline license (air-gapped)
GET /licensing/entitlements any user List module entitlements
GET /licensing/entitlements/{module_id} any user Check specific entitlement
POST /licensing/entitlements/import org admin Upload offline entitlement (air-gapped)
POST /licensing/verify internal Verify a license token

5.4 Plugins Module (/api/v1/plugins)

Method Path Auth Description
POST /plugins/connectors/{id}/sync any user Trigger connector sync
GET /plugins/connectors/{id}/sync-history any user Past sync runs
GET /plugins/connectors/{id}/health any user Connector health check
POST /plugins/policies/{id}/evaluate any user Trigger policy evaluation
GET /plugins/policies/{id}/evaluation-history any user Past evaluations
GET /plugins/policies/{id}/policies any user List policies in pack
GET /plugins/registry org admin All loaded plugins + status

5.5 Org/IAM Module (/api/v1/orgs)

Method Path Auth Description
GET /orgs/current any user Current user's org
PATCH /orgs/current org admin Update org settings
GET /orgs/current/projects any user List projects
POST /orgs/current/projects org admin Create project (tier-limited)
GET /orgs/current/projects/{id}/teams any user List teams
POST /orgs/current/projects/{id}/teams org admin Create team
GET /orgs/current/teams/{id}/groups any user List user groups
POST /orgs/current/teams/{id}/groups org admin Create user group
GET /orgs/current/members any user List org members
POST /orgs/current/members org admin Invite user (seat-limited)
POST /orgs/current/teams/{id}/members org admin Add user to team with role
PATCH /orgs/current/teams/{id}/members/{user_id} org admin Change role in team

Total: ~42 new endpoints.

6. Billing & Licensing

6.1 Payment Provider Abstraction

# backend/app/modules/billing/providers/base.py
class PaymentProvider(ABC):
    async def create_customer(self, org_id, email) -> str: ...
    async def charge(self, customer_id, amount_cents, metadata) -> PaymentResult: ...
    async def create_subscription(self, customer_id, plan_id, cycle) -> SubscriptionResult: ...
    async def cancel_subscription(self, subscription_id) -> None: ...
    async def get_invoices(self, customer_id) -> list[Invoice]: ...

Implementations: - StripeProvider — real Stripe SDK integration (cloud mode) - OfflineProvider — no-op for air-gapped mode

Selected via BILLING_PROVIDER setting ("stripe" | "offline").

6.2 Purchase Flow (Cloud)

1. Admin → POST /billing/purchase { module_id }
2. Validate: user is org admin, module exists, not already owned
3. billing.service → stripe.charge(customer_id, module.price_cents)
4. On success → licensing.create_entitlement(org_id, module_id)
5. Return { transaction, entitlement }
6. User can now install the module

6.3 Platform License Flow (Cloud)

1. Admin → PUT /billing/account { billing_cycle: "yearly" }
2. stripe.create_subscription(customer_id, tier_plan_id, "yearly")
3. Webhook: invoice.paid → licensing.renew_org_license(org_id)
4. Webhook: invoice.payment_failed → licensing.mark_grace_period(org_id)

6.4 ECDSA License Signing

Key pair: ECDSA P-256. Private key in env/secrets. Public key embedded in app.

Org license JWT: 

{
  "iss": "substrate-license-authority",
  "sub": "org_uuid",
  "type": "platform",
  "tier": "organization",
  "max_users": 50,
  "max_projects": 5,
  "iat": 1711584000,
  "exp": 1743120000
}

Module entitlement JWT: 

{
  "iss": "substrate-license-authority",
  "sub": "org_uuid",
  "type": "module_entitlement",
  "module_id": "connector-jira",
  "module_version": "1.*",
  "iat": 1711584000,
  "exp": null
}

exp: null for one-time purchases (perpetual). Platform license expires yearly.

6.5 Verification

async def verify_entitlement(self, org_id, module_id) -> LicenseVerification:
    # 1. Check Redis cache
    # 2. Check DB for entitlement record
    # 3. Verify JWT signature (ECDSA public key)
    # 4. Check expiry
    # 5. Cache result in Redis (TTL 1h)
    # 6. Return valid/invalid with reason

6.6 Free Tier

On org creation, if no license exists, auto-create free tier: - max_users: 10, max_projects: 1 - No signed token needed - Never expires - Free modules (GitHub, GDPR) skip entitlement checks — marketplace.install_module() checks module.pricing == 'free' and bypasses licensing.verify_entitlement() for free modules

6.7 Air-Gapped License Import

1. Admin receives license files from Substrate sales
2. POST /licensing/license/import → upload org platform license JWT
3. POST /licensing/entitlements/import → upload module entitlement JWTs
4. Backend verifies ECDSA signatures against embedded public key
5. Stores in DB, caches in Redis
6. No outbound network needed

7. Plugin Loading Engine

7.1 Base Classes

# backend/app/core/plugins/base.py

class DataConnectorBase(ABC):
    meta: PluginMeta
    async def install(self, config: dict) -> None: ...
    async def configure(self, config: dict) -> None: ...
    async def sync(self, installation_id: str) -> SyncResult: ...
    async def uninstall(self, installation_id: str) -> None: ...
    async def health_check(self, config: dict) -> HealthStatus: ...

class PolicyPackBase(ABC):
    meta: PluginMeta
    async def activate(self, config: dict) -> None: ...
    async def evaluate(self, context: EvalContext) -> EvalResult: ...
    async def deactivate(self) -> None: ...
    def get_policies(self) -> list[dict]: ...

7.2 Plugin Registry

Singleton. Plugins register at boot (built-ins) or on install (uploaded bundles).

class PluginRegistry:
    _connectors: dict[str, type[DataConnectorBase]]
    _policy_packs: dict[str, type[PolicyPackBase]]
    _instances: dict[str, Any]  # lazy instantiation with shared deps

    def register(cls, plugin_class): ...
    def get_connector(cls, module_id): ...
    def get_policy_pack(cls, module_id): ...
    def get_or_create_instance(cls, module_id, deps): ...

7.3 Shared Dependencies

class PluginDeps:
    """Injected into plugins — they never import drivers directly."""
    pg: AsyncPGPool
    neo4j: Neo4jDriver
    redis: RedisClient
    opa: OPAClient

7.4 Boot Sequence

entrypoint.sh
  → register_builtins (verify + insert .substrate bundles into DB)
  → uvicorn starts → create_app() lifespan:
      1. Build PluginDeps
      2. Register built-in plugin classes in PluginRegistry
      3. Query DB for active installations
      4. For each active policy pack: re-load Rego policies into OPA

7.5 Runtime Install Flow

POST /marketplace/installations { module_id }
  → verify entitlement → check registry → INSERT installation (status: installed)

PATCH /installations/{id}/configure { config }
  → validate config vs config_schema → encrypt secrets → UPDATE (status: configured)

PATCH /installations/{id}/status { status: "active" }
  → re-verify entitlement → instantiate plugin → configure/activate → UPDATE (status: active)

POST /plugins/connectors/{id}/sync
  → check active → acquire Redis lock → plugin.sync() → store results → release lock

POST /plugins/policies/{id}/evaluate
  → check active → plugin.evaluate() via OPA → store results

7.6 Air-Gapped Upload

POST /marketplace/upload (.substrate file)
  → extract → verify signature → verify hashes → verify license (if paid)
  → INSERT marketplace_modules → dynamic import → register in PluginRegistry

7.7 File Layout

backend/app/
├── core/plugins/
│   ├── base.py          # Abstract bases + models
│   ├── registry.py      # PluginRegistry singleton
│   ├── deps.py          # PluginDeps container
│   └── loader.py        # Bundle extraction, verification, dynamic import
├── connectors/
│   └── github/
│       ├── __init__.py
│       └── connector.py # GitHubConnector(DataConnectorBase)
├── policy_packs/
│   └── gdpr/
│       ├── __init__.py
│       ├── pack.py      # GDPRPolicyPack(PolicyPackBase)
│       └── policies/    # 4 Rego files
└── modules/
    ├── marketplace/     # Catalog, install, configure, upload
    ├── billing/         # Payment, invoices, subscriptions
    ├── licensing/       # ECDSA, entitlements, tier enforcement
    └── plugins/         # Sync/evaluate triggers, registry API

8. Frontend Integration

8.1 Decomposed Structure

Split SettingsPage.tsx (~850 lines) into focused page components:

ui/src/pages/settings/
├── SettingsLayout.tsx
├── OrgSettingsPage.tsx
├── TeamSettingsPage.tsx
├── ProfileSettingsPage.tsx
├── ApiTokensPage.tsx
├── MarketplacePage.tsx          # Shop + Installed + Requests tabs
├── BillingPage.tsx              # Payment, invoices, usage
├── LlmConnectionsPage.tsx
├── PlatformDataPage.tsx
├── PreferencesPage.tsx
└── index.ts

New API client files (thin wrappers using generated types):

ui/src/api/
├── marketplaceApi.ts
├── billingApi.ts
├── licensingApi.ts
├── pluginsApi.ts
└── orgApi.ts

8.2 Marketplace Page — Three Tabs

Shop: Browse catalog, filter by type/pricing, install free / purchase paid / request (non-admin). Air-gapped upload button.

Installed: List installations with status dots (grey=installed, yellow=configured, green=active). Configure, activate, sync/evaluate, disable actions.

Requests: Admin sees pending requests to approve/deny. Users see their own request status.

8.3 Dynamic Config Form

ConfigSchemaForm.tsx — renders form fields from a module's config_schema (JSON Schema). Handles string, number, boolean, array, password, URI types. Validates before submit. This means new modules render config UI automatically without frontend code changes.

8.4 Purchase Flow (UI)

Admin clicks [Purchase] → billingApi.purchase(moduleId)
  → Success → "Installed! Configure now?" modal
  → Failure → error with retry

Non-admin clicks [Request] → modal with reason field
  → marketplaceApi.requestModule(moduleId, reason)
  → "Request sent to your org admin"

8.5 UserContext Changes

Add org fields to SubstrateUser:

{
  org_id: string;
  org_slug: string;
  org_role: 'admin' | 'member' | 'billing';
  projects: Array<{
    id: string;
    slug: string;
    teams: Array<{ id, slug, role, user_group }>;
  }>;
}

org_role === 'admin' gates all purchase/billing/upload actions in UI.

8.6 New Routes

/settings/marketplace       → MarketplacePage
/settings/billing           → BillingPage
/settings/connectors/:id    → ConnectorConfigPage
/settings/policy-packs/:id  → PolicyPackPage

9. Infrastructure

9.1 OPA Service

Location: infra/opa/

  • OPA 1.4.2, exposed on port 8181
  • Policies are NOT pre-loaded from files
  • Pushed to OPA via REST API when a policy pack is activated
  • Stateless — on restart, backend re-pushes active policies during lifespan startup
  • Added to root docker-compose.yml as backend dependency

Backend client: backend/app/core/opa_client.py - load_policy(policy_id, rego_content) — PUT to OPA - remove_policy(policy_id) — DELETE from OPA - evaluate(policy_path, input_data) — POST to OPA - health() — GET /health

Setting: OPA_URL = "http://opa:8181"

9.2 Keycloak Changes

substrate-realm.json updates: 1. New protocol mappers: org_id, org_slug, org_role, projects (JSON) 2. Restructured groups: flat 8 groups → hierarchical /orgs/{org}/projects/{proj}/teams/{team}/{group} 3. Updated demo user attributes with org fields 4. GitHub IdP: replace hardcoded secrets with ${GITHUB_OAUTH_CLIENT_ID} env var references

GitHub SPI Mapper: infra/keycloak/spi/github-mapper/ - Java Maven project (Keycloak BOM 26.x) - GitHubDataMapper extends AbstractIdentityProviderMapper - On GitHub OAuth login: fetch repos, pages, projects v2, actions, packages - Store as Keycloak user attributes (summary mode) - Configurable per-mapper: toggle each data fetch, set max repos, summary vs full mode - Built JAR deployed to Keycloak /opt/keycloak/providers/

9.3 Updated Docker Compose

10 total services: postgres, neo4j, redis, nats, keycloak, pgadmin, redis-commander, opa (new), ui, backend.

Backend depends_on: postgres, neo4j, redis, nats, keycloak, opa (all healthy).

10. OpenAPI Wiring

10.1 Single Source of Truth

api/openapi.yml is the canonical spec. Delete openapi.spec.yaml (identical copy). Update generate-types.sh to remove line 47 (cp command).

10.2 Generation Pipeline

FastAPI create_app().openapi() → api/openapi.yml
  → openapi-typescript → ui/src/types/openapi.generated.ts
  → datamodel-codegen  → backend/app/types/openapi_generated.py

10.3 Rules

  • Every backend endpoint MUST exist in OpenAPI spec
  • Frontend MUST use generated types from openapi.generated.ts
  • Backend response schemas MUST match OpenAPI definitions
  • All ~42 new endpoints added to spec before implementation

11. Built-in Modules

11.1 GitHub Data Connector

Class: GitHubConnector(DataConnectorBase) at backend/app/connectors/github/connector.py

Capabilities: - Repos: GET /user/repos → Neo4j :Repository nodes - Pages: detect from repo.has_pages → Neo4j :GitHubPages nodes - Projects v2: POST /graphql → Neo4j :GitHubProject nodes - Actions: GET /repos/{owner}/{repo}/actions/workflows → Neo4j :Workflow nodes - Packages: GET /user/packages → Neo4j :Package nodes

Sync pipeline: 

GitHub API → fetch data (with pagination + rate limit handling)
  → Create/update Neo4j nodes + edges
  → Store raw data in PostgreSQL
  → Update sync state in PostgreSQL
  → Invalidate Redis caches

11.2 GDPR Policy Pack

Class: GDPRPolicyPack(PolicyPackBase) at backend/app/policy_packs/gdpr/pack.py

4 Rego Policies:

ID Policy Checks
GDPR-001 Data Classification Service has classification + data owner
GDPR-002 Data Retention Retention policy set, <= 7 years, deletion mechanism exists
GDPR-003 Consent Mechanism If processing personal data: consent mechanism + lawful basis
GDPR-004 Cross-Border Transfer EU region OR adequacy decision OR standard contractual clauses

Activation: loads all 4 Rego files into OPA via REST API. Evaluation: calls OPA for each policy path, aggregates results. Deactivation: removes policies from OPA.

12. Dual Access Paths — Frontend vs API Key

12.1 Two Authentication Methods

Every endpoint accepts either method. The backend resolves a UserInfo object regardless of how the caller authenticated.

Method Header Source Used by
JWT (OIDC) Authorization: Bearer <jwt> Keycloak access token Frontend (browser), SSO flows
API Key X-API-Key: st_<token> Personal API token from /auth/tokens CLI tools, CI/CD, scripts, external integrations

12.2 OpenAPI Security Schemes

components:
  securitySchemes:
    BearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT
      description: Keycloak OIDC access token (frontend sessions)
    ApiKeyAuth:
      type: apiKey
      in: header
      name: X-API-Key
      description: Personal API token (automation, CLI, external clients)

Every endpoint declares both: 

security:
  - BearerAuth: []
  - ApiKeyAuth: []

12.3 Backend Auth Resolution

# backend/app/core/security.py

async def get_current_user(
    bearer: Optional[str] = Depends(optional_bearer),
    api_key: Optional[str] = Header(None, alias="X-API-Key"),
) -> UserInfo:
    if bearer:
        return await _resolve_from_jwt(bearer)
    elif api_key:
        return await _resolve_from_api_key(api_key)
    else:
        raise HTTPException(401, "No authentication provided")

JWT path (_resolve_from_jwt): 1. Decode JWT against Keycloak JWKS (cached in Redis, TTL 1h) 2. Validate issuer, audience, expiry 3. Extract claims → build UserInfo (including org_id, org_role, projects, teams)

API key path (_resolve_from_api_key): 1. Hash the provided key with SHA256 2. Look up hash in user_api_tokens table (must not be revoked, not expired) 3. Join to user_profiles → load stored profile 4. Check key's scopes include the requested action 5. Build UserInfo from stored profile (same shape as JWT path)

After resolution: Both paths produce the same UserInfo object. All downstream code (authorization, org context, role checks) works identically regardless of auth method.

12.4 API Key Scopes

When creating an API token, the user specifies scopes:

POST /auth/tokens
{
  "name": "ci-pipeline",
  "scopes": ["marketplace:read", "plugins:sync", "connectors:read"],
  "expires_in_days": 365
}

Available scopes:

Scope Allows
* Full access (same as the user's JWT)
marketplace:read Browse catalog, list installations
marketplace:write Install, configure, upload bundles
billing:read View invoices, usage
billing:write Purchase modules, configure billing
plugins:read View sync history, evaluation history
plugins:sync Trigger connector syncs
plugins:evaluate Trigger policy evaluations
connectors:read View connector config and health
orgs:read View org/project/team structure
orgs:write Manage org/project/team members

Scope checking middleware: 

def require_scope(scope: str):
    def checker(user: UserInfo = Depends(get_current_user)):
        if user.auth_method == "api_key" and "*" not in user.scopes:
            if scope not in user.scopes:
                raise HTTPException(403, f"API key missing scope: {scope}")
        return user
    return checker

12.5 Endpoint Grouping in OpenAPI

Endpoints are tagged with x-substrate-client-group to distinguish usage patterns:

Group Description Typical caller
frontend-session UI-driven actions (browse, configure, settings) Browser with JWT
automation Programmatic actions (sync, evaluate, CRUD) API key from scripts/CI
webhook Inbound webhooks (Stripe, GitHub) External service with signature
internal Service-to-service (license verify) Backend internal calls

Both JWT and API key work on all frontend-session and automation endpoints. The distinction is documentation — it tells consumers which endpoints are designed for which use case.

13. Authentication Flows — GitHub OAuth & Keycloak Native

13.1 Two Login Paths

┌──────────────────┐     ┌─────────────────────────────────────┐
│  Login Screen    │     │            Keycloak                  │
│                  │     │                                      │
│  [Login with     │────→│  Native login (username + password)  │
│   Keycloak]      │     │  → Validates credentials             │
│                  │     │  → Issues JWT with substrate-claims   │
│  [Login with     │────→│  → Redirects to GitHub OAuth         │
│   GitHub]        │     │  → GitHub returns access_token        │
│                  │     │  → GitHub SPI mapper fires:            │
│                  │     │    - Fetches repos, pages, projects   │
│                  │     │    - Stores as user attributes         │
│                  │     │  → Issues JWT with substrate-claims   │
│                  │     │    + github_username, github_id        │
└──────────────────┘     └───────────────┬─────────────────────┘
                                         │
                                         ▼
                          ┌──────────────────────────┐
                          │     Frontend receives     │
                          │     JWT access_token      │
                          │                          │
                          │  AuthBridge extracts it  │
                          │  → injects into api()    │
                          │  → UserContext derives    │
                          │    role, perspective,     │
                          │    org, projects, teams   │
                          └──────────────────────────┘

13.2 First-Time GitHub User

When a user logs in via GitHub for the first time:

  1. Keycloak creates a federated user identity
  2. GitHub SPI mapper fetches GitHub data → stores as user attributes
  3. User has NO org membership yet
  4. Backend /auth/me detects: user exists but no org
  5. Frontend shows onboarding flow:
  6. Create a new org (becomes org admin, free tier auto-created)
  7. OR accept an org invitation (if one exists for their email/GitHub username)
  8. Once in an org, the admin assigns them to projects/teams

13.3 First-Time Keycloak Native User

Same as above, minus the GitHub data enrichment. Admin can pre-provision users in Keycloak with org/team attributes, or users self-register and go through onboarding.

13.4 JWT Claims — Full Example

After login (either method), the JWT contains:

{
  "iss": "https://keycloak.example.com/realms/substrate",
  "sub": "uuid-user-123",
  "aud": "substrate-backend",
  "exp": 1711587600,
  "iat": 1711584000,

  "preferred_username": "alice",
  "email": "alice@acme.com",
  "given_name": "Alice",
  "family_name": "Chen",
  "name": "Alice Chen",

  "role": "Developer",
  "perspective": "engineering",
  "team": "payments-team",

  "org_id": "uuid-org-acme",
  "org_slug": "acme-corp",
  "org_role": "member",
  "projects": [
    {
      "id": "uuid-project-1",
      "slug": "ecommerce-platform",
      "teams": [
        {
          "id": "uuid-team-payments",
          "slug": "payments-team",
          "role": "Developer",
          "user_group": "backend-engineers"
        },
        {
          "id": "uuid-team-platform",
          "slug": "platform-team",
          "role": "SRE",
          "user_group": "sre"
        }
      ]
    }
  ],

  "identity_provider": "github",
  "github_username": "alicechen",
  "github_id": "12345",

  "groups": ["/orgs/acme-corp/projects/ecommerce-platform/teams/payments-team/backend-engineers"],
  "realm_access": { "roles": ["Developer", "default-roles-substrate"] }
}

13.5 API Key — Equivalent Context

When authenticating via API key, the backend loads the same fields from user_profiles (synced from JWT on last login). The UserInfo object is identical:

UserInfo(
    sub="uuid-user-123",
    email="alice@acme.com",
    role="Developer",
    org_id="uuid-org-acme",
    org_role="member",
    projects=[...],
    auth_method="api_key",       # ← only difference
    scopes=["plugins:sync", "connectors:read"],  # ← from API token record
)

14. Authorization Matrix

14.1 Org-Level Authorization

Action org_role: admin org_role: billing org_role: member
View org settings Y Y Y
Edit org settings Y N N
Create project Y N N
Delete project Y N N
Invite user to org Y N N
Remove user from org Y N N
View billing/invoices Y Y N
Configure billing Y Y N
Purchase module Y N N
Upload .substrate bundle Y N N
Import license file Y N N
View marketplace catalog Y Y Y
Install module Y N N
Request module Y Y Y
Approve/deny requests Y N N
Uninstall module Y N N

14.2 Project/Team-Level Authorization

Action Team role: Admin Team role: any other Not in team
View project/team Y Y N (unless org admin)
Create team in project Y N N
Add member to team Y N N
Remove member from team Y N N
Change member's role Y N N
Create user group Y N N
Configure installed connector Y Y N
Trigger connector sync Y Y N
Trigger policy evaluation Y Y N
View sync/eval history Y Y N

Org admin override: Org admins can perform any project/team action regardless of team membership.

14.3 Marketplace-Specific Authorization

Action Entitlement Auth Result
Install free module None needed Any org member Installed
Install paid module Must exist Org admin only Installed
Configure module Module installed Any team member Configured
Activate module Module configured Org admin Active
Trigger sync Module active Any team member Sync runs
Trigger evaluate Module active Any team member Evaluation runs
Disable module Module active Org admin Disabled
Uninstall module Any status Org admin Uninstalled

14.4 API Key Scope Enforcement

API key scopes are checked in addition to role/org_role authorization. Both must pass:

Request: POST /plugins/connectors/{id}/sync
  → Auth: API key with scopes=["plugins:sync"]
  → Scope check: "plugins:sync" ∈ scopes? ✓
  → Role check: user is team member on this connector's project? ✓
  → Result: allowed

Request: POST /billing/purchase
  → Auth: API key with scopes=["billing:write"]
  → Scope check: "billing:write" ∈ scopes? ✓
  → Role check: user.org_role == "admin"? Must be ✓
  → Result: allowed only if user is org admin AND key has billing:write scope

14.5 Backend Authorization Middleware

# backend/app/core/authorization.py

def require_org_admin(user: UserInfo = Depends(get_current_user)):
    if user.org_role != "admin":
        raise HTTPException(403, "Org admin required")
    return user

def require_org_billing(user: UserInfo = Depends(get_current_user)):
    if user.org_role not in ("admin", "billing"):
        raise HTTPException(403, "Org admin or billing role required")
    return user

def require_team_member(team_id: str):
    def checker(user: UserInfo = Depends(get_current_user)):
        if user.org_role == "admin":
            return user  # org admin override
        if not any(
            t["id"] == team_id
            for p in user.projects
            for t in p["teams"]
        ):
            raise HTTPException(403, "Not a member of this team")
        return user
    return checker

def require_team_admin(team_id: str):
    def checker(user: UserInfo = Depends(get_current_user)):
        if user.org_role == "admin":
            return user  # org admin override
        for p in user.projects:
            for t in p["teams"]:
                if t["id"] == team_id and t["role"] == "Admin":
                    return user
        raise HTTPException(403, "Team admin required")
    return checker

15. Security Considerations

  • Sensitive config fields (tokens, keys) encrypted at rest in module_installations.config
  • ECDSA private key stored in env var / mounted secret, never in DB
  • GitHub IdP credentials moved from hardcoded JSON to environment variables
  • Dual auth: JWT (OIDC) for frontend sessions, API key (X-API-Key) for automation
  • API key scopes enforce least-privilege access for programmatic clients
  • Stripe webhook signature verification
  • Bundle signature verification prevents tampered module uploads
  • License verification prevents unauthorized module usage
  • OPA policies evaluated deterministically (no LLM in enforcement path)
  • All file uploads validated: size limits, type checking, hash verification
  • Authorization checks at every endpoint: org_role + team role + API key scope

16. Configuration Management

16.1 Problem

The current SettingsPage.tsx (~850 lines) has 10 settings domains all using local state or hardcoded seed data. None persist to backend. Platform data configs (PostgreSQL, Redis, Neo4j, NATS, pgvector) are env vars only — not configurable at runtime. LLM connections, notification preferences, feature flags, and retention policies exist only in the frontend.

16.2 Industry Standard: Override Hierarchy

Use Dynaconf (Python) — a battle-tested library that supports config from YAML, env vars, CLI args, Redis, and REST API overrides with a well-defined precedence order.

Override sequence (lowest → highest priority):

1. Defaults        → hardcoded in code (Pydantic BaseSettings defaults)
2. YAML file       → backend/config/settings.yaml (shipped with app)
3. Env vars        → SUBSTRATE_* prefix (docker-compose, .env files)
4. CLI args        → --setting=value (uvicorn startup)
5. DB overrides    → org_settings table (per-org runtime overrides via REST API)
6. Redis cache     → hot config cache (TTL-based, invalidated on DB write)

Rule: Levels 1–4 are deployment-time (set by ops). Level 5 is runtime (set by org admins via UI). Level 6 is a cache of level 5. The UI only writes to level 5. The backend merges all levels at request time.

16.3 Backend Configuration Module

backend/app/modules/config/
├── router.py          # REST API for reading/writing settings
├── service.py         # Merge hierarchy, validate, persist
├── repository.py      # org_settings table CRUD
├── schemas.py         # Pydantic models for each config domain
└── defaults.py        # Default values for all config domains

16.4 New DB Table

-- V11__org_settings.sql
CREATE TABLE org_settings (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    domain VARCHAR(50) NOT NULL,
    settings JSONB NOT NULL DEFAULT '{}',
    updated_by VARCHAR(255),
    updated_at TIMESTAMPTZ DEFAULT now(),
    UNIQUE(org_id, domain)
);

-- Each domain is a separate row per org:
-- domain: 'org_profile', 'llm_connections', 'platform_postgres', 'platform_redis',
--         'platform_neo4j', 'platform_nats', 'platform_vector', 'notifications',
--         'features', 'retention'

16.5 Config Domains & Schemas

Domain 1: Org Profile (org_profile)

class OrgProfileConfig(BaseModel):
    org_name: str
    legal_name: str | None = None
    primary_domain: str | None = None
    timezone: str = "UTC"
    region: str | None = None
    billing_contact: str | None = None
    support_contact: str | None = None

Domain 2: LLM Connections (llm_connections)

class LlmEndpoint(BaseModel):
    id: str
    name: str
    kind: Literal['dense', 'sparse', 'reranker', 'embedding', 'coding', 'stable-diffusion']
    api_base_url: str
    api_key: str                    # encrypted at rest
    general_config: dict = {}
    multi_lora_config: dict = {}
    moe_config: dict = {}
    reusable: bool = True

class LlmPurposeRoute(BaseModel):
    id: str
    purpose: str
    endpoint_id: str

class LlmConnectionsConfig(BaseModel):
    endpoints: list[LlmEndpoint] = []
    purpose_routes: list[LlmPurposeRoute] = []

Domain 3–7: Platform Data (platform_postgres, platform_redis, platform_neo4j, platform_nats, platform_vector)

class PlatformConnectionConfig(BaseModel):
    """Generic connection config — each platform type uses relevant fields."""
    host: str
    port: int
    database: str | None = None
    username: str | None = None
    password: str | None = None     # encrypted at rest
    ssl_enabled: bool = False
    ssl_cert_path: str | None = None
    connection_pool_size: int = 10
    timeout_seconds: int = 30
    extra: dict = {}                # platform-specific (e.g., neo4j bolt mode, nats jetstream)

Read-only fields derived from live connection (not editable): 

class PlatformConnectionStatus(BaseModel):
    connected: bool
    version: str | None = None
    uptime: str | None = None
    memory_usage: str | None = None
    active_connections: int | None = None
    last_checked: datetime

Domain 8: Notifications (notifications)

class NotificationPreference(BaseModel):
    key: str                        # e.g., "pr_violation_blocked"
    label: str
    description: str
    enabled: bool = True

class NotificationsConfig(BaseModel):
    preferences: list[NotificationPreference] = []

Domain 9: Features (features)

class FeatureFlag(BaseModel):
    key: str                        # e.g., "institutional_memory"
    label: str
    description: str
    enabled: bool = True

class FeaturesConfig(BaseModel):
    flags: list[FeatureFlag] = []

Domain 10: Retention (retention)

class RetentionRule(BaseModel):
    id: str
    source: str                     # e.g., "Observed Graph"
    data_type: str                  # e.g., "Drift Scores"
    start_at: datetime
    end_at: datetime
    description: str

class RetentionConfig(BaseModel):
    rules: list[RetentionRule] = []

16.6 Config Merge Logic

# backend/app/modules/config/service.py

class ConfigService:
    async def get_merged_config(self, org_id: str, domain: str) -> dict:
        """Merge all config layers for a given domain."""

        # Level 1: Defaults from code
        defaults = get_domain_defaults(domain)

        # Level 2: YAML file
        yaml_config = load_yaml_config(domain)

        # Level 3: Env vars (SUBSTRATE_{DOMAIN}_{KEY})
        env_config = load_env_config(domain)

        # Level 4: CLI args (already in pydantic settings)
        # (merged into env_config by pydantic-settings)

        # Level 5: DB overrides (per-org)
        db_config = await self.repo.get_org_settings(org_id, domain)

        # Merge: later levels override earlier
        merged = {**defaults, **yaml_config, **env_config, **(db_config or {})}

        return merged

    async def update_config(self, org_id: str, domain: str, patch: dict, user_id: str) -> dict:
        """Write runtime config override to DB (level 5)."""

        # Validate against domain schema
        schema_class = get_domain_schema(domain)
        current = await self.get_merged_config(org_id, domain)
        updated = {**current, **patch}
        schema_class(**updated)  # validates

        # Persist to DB
        await self.repo.upsert_org_settings(org_id, domain, patch, user_id)

        # Invalidate Redis cache
        await self.redis.delete(f"config:{org_id}:{domain}")

        return updated

16.7 Secret Handling

Fields marked as sensitive (api_key, password) are: 1. Encrypted at rest in org_settings.settings using Fernet symmetric encryption 2. Redacted on read — API returns "•••••••" for secret fields unless ?reveal=true and user is org admin 3. Write-only update — sending "•••••••" back means "keep current value" 4. Key management — encryption key from SUBSTRATE_CONFIG_ENCRYPTION_KEY env var

16.8 API Endpoints

# Config Management
GET    /api/v1/config/domains
       → list all config domains with metadata
       → response: { domains: [{ id, label, description, editable }] }

GET    /api/v1/config/{domain}
       → get merged config for domain (current org)
       → secrets redacted unless ?reveal=true (org admin only)
       → response: { data: {...}, meta: { source_layers: [...] } }

PATCH  /api/v1/config/{domain}
       → update runtime config (writes to DB level 5)
       → org admin only
       → body: partial config JSON
       → response: { data: {...} }  (merged result)

DELETE /api/v1/config/{domain}
       → reset to defaults (remove DB override)
       → org admin only
       → response: { data: {...} }  (defaults only)

# Platform connection health (read-only)
GET    /api/v1/config/platform/{service}/status
       → live connection status for postgres|redis|neo4j|nats|vector
       → response: PlatformConnectionStatus

16.9 YAML Config File

Shipped with the app, editable by ops in self-hosted deployments:

# backend/config/settings.yaml

org_profile:
  timezone: "UTC"

llm_connections:
  endpoints: []
  purpose_routes: []

platform_postgres:
  host: "postgres"
  port: 5432
  database: "substrate"
  connection_pool_size: 10

platform_redis:
  host: "redis"
  port: 6379
  ssl_enabled: false

platform_neo4j:
  host: "neo4j"
  port: 7687
  extra:
    bolt_tls: false

platform_nats:
  host: "nats"
  port: 4222
  extra:
    jetstream_enabled: true

platform_vector:
  host: "postgres"
  port: 5432
  database: "substrate"
  extra:
    vector_dimensions: 1536

notifications:
  preferences:
    - key: "pr_violation_blocked"
      label: "PR violation blocked"
      description: "Notify when your PR is blocked by governance."
      enabled: true
    - key: "verification_queue_assigned"
      label: "Verification queue assigned"
      description: "Alert when items are assigned to you."
      enabled: true
    - key: "sprint_retro_insight"
      label: "Sprint retro insight ready"
      description: "Notify when sprint structural insight is generated."
      enabled: true
    - key: "key_person_risk"
      label: "Key-person risk detected"
      description: "Alert when a departing member owns critical services."
      enabled: false
    - key: "drift_threshold_breach"
      label: "Drift threshold breach"
      description: "Alert when a domain tension threshold is exceeded."
      enabled: true
    - key: "adr_gap_detected"
      label: "ADR gap detected"
      description: "Notify when a service with no ADR is discovered."
      enabled: false

features:
  flags:
    - key: "institutional_memory"
      label: "Institutional Memory"
      description: "WHY edges, DecisionNodes, and historical context tracking."
      enabled: true
    - key: "ssh_runtime_connector"
      label: "SSH Runtime Connector"
      description: "Agentless host verification against observed runtime state."
      enabled: true
    - key: "pre_change_simulation"
      label: "Pre-Change Simulation"
      description: "Sandboxed what-if analysis before code changes land."
      enabled: true
    - key: "intent_mismatch_detection"
      label: "Intent Mismatch Detection"
      description: "Compare project intent with PR code vectors."
      enabled: true
    - key: "fix_pr_generation"
      label: "Fix PR Generation"
      description: "Auto-generate remediation patches for deterministic violations."
      enabled: false
    - key: "global_graphrag"
      label: "Global GraphRAG"
      description: "Leiden-based map-reduce retrieval across the architecture graph."
      enabled: true

retention:
  rules: []

16.10 Env Var Override Convention

Any YAML config value can be overridden via env var with SUBSTRATE_ prefix and __ as nesting separator:

# Override postgres host
SUBSTRATE_PLATFORM_POSTGRES__HOST=my-db-host.internal

# Override feature flag
SUBSTRATE_FEATURES__FLAGS__0__ENABLED=false

# Override LLM endpoint URL
SUBSTRATE_LLM_CONNECTIONS__ENDPOINTS__0__API_BASE_URL=http://custom-llm:8080/v1

This is the standard Dynaconf / pydantic-settings convention.

16.11 Frontend Config API Client

// ui/src/api/configApi.ts
export const configApi = {
  listDomains: () =>
    api<ConfigDomainsResponse>('/config/domains'),

  getConfig: (domain: string, reveal?: boolean) =>
    api<ConfigResponse>(`/config/${domain}${reveal ? '?reveal=true' : ''}`),

  updateConfig: (domain: string, patch: Record<string, unknown>) =>
    api<ConfigResponse>(`/config/${domain}`, {
      method: 'PATCH', body: JSON.stringify(patch)
    }),

  resetConfig: (domain: string) =>
    api<ConfigResponse>(`/config/${domain}`, { method: 'DELETE' }),

  getPlatformStatus: (service: string) =>
    api<PlatformConnectionStatus>(`/config/platform/${service}/status`),
};

16.12 Frontend Settings Wiring

Each decomposed settings page calls configApi:

Page Domain Editable Notes
OrgSettingsPage org_profile org admin Name, domain, timezone, contacts
TeamSettingsPage — org admin Uses org/team APIs (Section 5.5), not config
ProfileSettingsPage — read-only OIDC claims from UserContext
ApiTokensPage — any user Uses auth/tokens APIs (existing)
MarketplacePage — org admin Uses marketplace APIs (Section 5.1)
BillingPage — org admin Uses billing APIs (Section 5.2)
LlmConnectionsPage llm_connections org admin Endpoints + purpose routes
PlatformDataPage platform_* org admin 5 sub-sections, each with config + live status
NotificationsPage notifications any user Toggle preferences
FeaturesPage features org admin Feature flag toggles
RetentionPage retention org admin Per-data-type retention windows

17. Files Modified / Created

Modified

  • api/openapi.yml — add ~50 new endpoints (marketplace + billing + licensing + plugins + orgs + config)
  • api/generate-types.sh — remove spec.yaml copy
  • backend/app/main.py — register 5 new module routers (+ config) + plugin lifespan
  • backend/app/settings.py — add OPA, billing, licensing, config encryption settings
  • backend/app/core/security.py — add API key auth + org context extraction
  • backend/entrypoint.sh — add register_builtins step
  • infra/keycloak/substrate-realm.json — groups, mappers, demo users, IdP env vars
  • infra/keycloak/import-and-start.sh — env var pass-through
  • docker-compose.yml — add OPA include and backend deps
  • ui/src/app/UserContext.tsx — add org fields
  • ui/src/app/userProfile.ts — extract org claims
  • ui/src/app/settings.ts — add marketplace, billing sub-pages
  • ui/src/router.tsx — add new routes

Created

  • backend/db/postgres/V5–V11 — 7 new Flyway migrations (V11: org_settings)
  • backend/db/neo4j/marketplace_schema.cypher — new constraints
  • backend/app/modules/marketplace/ — router, service, repository, schemas
  • backend/app/modules/billing/ — router, service, repository, providers/
  • backend/app/modules/licensing/ — router, service, repository, keys
  • backend/app/modules/plugins/ — router, service
  • backend/app/core/plugins/ — base, registry, deps, loader
  • backend/app/modules/config/ — router, service, repository, schemas, defaults
  • backend/app/core/opa_client.py — OPA HTTP client
  • backend/app/connectors/github/ — GitHubConnector
  • backend/app/policy_packs/gdpr/ — GDPRPolicyPack + 4 Rego files
  • backend/app/cli/register_builtins.py — boot-time bundle registration
  • backend/bundles/ — 2 .substrate bundle files
  • infra/opa/ — Dockerfile, docker-compose.yml, config
  • infra/keycloak/spi/github-mapper/ — Java Maven project
  • ui/src/pages/settings/ — 10 decomposed settings pages
  • ui/src/api/marketplaceApi.ts — marketplace API client
  • ui/src/api/billingApi.ts — billing API client
  • ui/src/api/licensingApi.ts — licensing API client
  • ui/src/api/pluginsApi.ts — plugins API client
  • backend/config/settings.yaml — default YAML config file
  • ui/src/api/orgApi.ts — org/IAM API client
  • ui/src/api/configApi.ts — config management API client
  • ui/src/components/ConfigSchemaForm.tsx — dynamic JSON Schema form renderer

Deleted

  • api/openapi.spec.yaml — redundant copy