tune Config & Inspector expand_more

ModelOp Enterprise Governance

Curated Insight

Taxonomy Context

ModelOp

Bridging Standard Metrics & Enterprise Needs

ModelOp Center provides a comprehensive suite of out-of-the-box (OOTB) monitors. However, enterprise requirements often demand unique calculations. This guide introduces the capabilities of custom Python-based monitors.

code The Developer Value

Write standard Python code using Data Science libraries and have it automatically integrated into an enterprise-grade governance platform. No complex API integrations required.

verified_user The Governance Value

Ensures that no matter how complex the model metric, the "evidence" is always captured in a standardized, auditable format for risk and compliance.

Monitors by Model Modality

Custom monitors can be built to supplement any of these OOTB categories based on your domain.

Data Science Metrics Catalog

Hover or click on any metric pill below to view its algorithmic definition and dual-persona insights. These definitions guide standard and custom monitor creation.

psychology

1. Generative AI / NLP Validations

Secures conversational agents and generative models against hallucinations, toxic output, and data leakage.

balance

2. Ethical Fairness & Bias

Evaluates model behavior disparities against protected classes to ensure regulatory compliance.

show_chart

3. Regression & Credit Risk

Assesses continuous prediction errors and rank-ordering capabilities, crucial for financial/credit models.

join_inner

4. Classification Performance

Evaluates discrete prediction models. Requires schemas mapping score_column and label_column.

query_stats

5. Data & Concept Drift

Detects shifts in input distributions over time by comparing baseline data against a sample slice.

Enterprise Monitor Selection Pathway

Selecting the right AI monitor requires mapping the technical model type directly to enterprise business outcomes. Use this dynamic tree to navigate from your raw data state to the specific ModelOp monitors required.

manage_search Node Inspector

(Ctrl+Click for Multi)

Model Selection

Select your model types above, then click nodes in the graph to map the required governance monitors to their specific enterprise use cases.
System Component Identifier
modelop.model.registration
sync Generating Pathway Map...

filter_alt Pathway Filters

Toggle model types below to dynamically generate the recommended monitor pathway branches.

Industry Context Presets

info Disclaimer: This is an enablement tool intended for guidance and is not to be used for final or automated business decision making.

Key Concepts: Execution Architectures

Interact with the architecture diagrams below. Toggle between standard monitoring and agentic LLM patterns. Both utilize uniform structures aligned through the Job Execution core.

manage_search Node Inspector

Ctrl+Click multi

Execution Process

Select a node or edge in the diagram to view its details.

  1. Process initiated (via UI/API)
  2. Metrics job created (via MLC)
  3. Job sent to Runtime
  4. Runtime loads datasets & code
  5. Runtime executes Python source
  6. Output yielded as JSON
  7. Model Test Result attached
System Component Identifier
modelop.core
sync Generating Architecture Map...

Artifact Explorer & Generator

A custom monitor is defined by specific files in a Git repository. Explore the required structure and generate contextual Data Science code boilerplates.

verified Required Files

custom_metrics.py terminal

Primary Model Source Code

metadata.json data_object

Monitor Classification Meta

required_assets.json list_alt

Input Data Definitions

custom_metrics.py

The algorithmic brain. Use the pills below to generate boilerplate logic for different Data Science use cases.

Onboarding Roadmap

Follow this interactive guide to promote your custom monitor from a local IDE script into a production-ready ModelOp asset.

1

cloud_upload Connect Git Repository

expand_more

Import your custom code via the ModelOp UI. Navigate to Monitors > Add Monitor and select "Git".

  • Provide the Repository URL and target Branch.
  • Assign an Access Group to control viewing permissions.
  • The system automatically scans for the metrics() entry point.
2

camera Freeze a Snapshot

chevron_right

Snapshots create an immutable version of your monitor code linked to a specific commit.

This guarantees production stability, ensuring that subsequent commits to the Git branch don't silently alter or break actively scheduled tests.

3

link Map Data & Execute

chevron_right

Attach the monitor snapshot to your Business Model's "Monitoring" tab.

  • The UI will prompt you to map specific data assets as defined in your required_assets.json.
  • Click Play to spawn the Job and yield your metrics.

Out-of-the-Box (OOTB) Monitors Catalog

ModelOp Center provides an extensive library of pre-built monitors covering classification, regression, fairness, drift, stability, and volumetric analysis. Click on each group to explore available monitors and their configurations.

trending_up
Classification & Performance
AUC, Accuracy, Precision, Recall, F1, Confusion Matrix
expand_more
show_chart
Regression & Diagnostics
MAE, RMSE, R², Gini, C-Stat
expand_more
balance
Fairness & Bias
Group Metrics, Disparities, Aequitas Analysis
expand_more
query_stats
Drift Detection
KS Test, JS Divergence, KL Divergence
expand_more
shield_check
Stability Analysis
PSI, CSI, Population Metrics
expand_more
inventory_2
Volumetric & Validation
Record Counts, Schema Checks, Data Quality
expand_more

info Selecting the Right Monitor

Use the Selection Pathway tab to map your model type (Classification, Regression, or GenAI/LLM) to the appropriate OOTB monitor. Combine multiple monitors for comprehensive governance.

Job Scheduling & On-Demand Execution

Execute monitors via the moc_python_sdk HttpClient API or orchestrate complex workflows using MLC/BPMN templates. Supports both scheduled and on-demand metric generation.

api Creating Jobs with moc_python_sdk

The HttpClient provides a straightforward REST interface for job creation:

from moc_python_sdk import HttpClient

# Initialize secure client
client = HttpClient(
    endpoint="https://modelop-instance.com",
    username="api_user",
    password="api_token",
    verify_ssl=True
)

# Define job payload
job_payload = {
    "modelTestId": "job_auc_monitor_prod_v1",
    "monitorSnapshotId": "snapshot_auc_monitor_v2",
    "dataAssetMappings": {
        "baseline_data": "asset_train_dataset_v3",
        "sample_data": "asset_prod_slice_20260416"
    },
    "executionContext": {
        "priority": "HIGH",
        "timeout_seconds": 1800,
        "email_on_complete": "ml-team@company.com"
    }
}

# Create the job
response = client.create_job(payload=job_payload)
job_id = response['id']
job_status = response['status']  # 'CREATED', 'RUNNING', 'COMPLETED'
Required Fields: modelTestId, monitorSnapshotId, dataAssetMappings
Optional: executionContext (priority, timeout, notifications)
Environment: Set MODELOP_ENDPOINT, MODELOP_OAUTH_SCOPE, MODELOP_VERIFY_SSL

flow_chart MLC/BPMN Subprocess Patterns

Complex monitoring workflows can be orchestrated using Camunda BPMN 2.0 subprocesses:

<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL">
  <bpmn:process id="monitor_orchestration">
    <bpmn:startEvent id="start" />
    
    <bpmn:callActivity id="spawn_job" 
                         calledElement="create_monitor_job">
      <bpmn:ioMapping>
        <bpmn:inputParameter name="monitorId">${monitorSnapshot}</bpmn:inputParameter>
        <bpmn:outputParameter name="jobId">${job_response.id}</bpmn:outputParameter>
      </bpmn:ioMapping>
    </bpmn:callActivity>
    
    <bpmn:boundaryEvent id="timeout_handler" 
                          attachedToRef="spawn_job" 
                          timerEventDefinition>
      <bpmn:timerEventDefinition>
        <bpmn:timeDuration>PT30M</bpmn:timeDuration>
      </bpmn:timerEventDefinition>
    </bpmn:boundaryEvent>
    
    <bpmn:endEvent id="end" />
  </bpmn:process>
</bpmn:definitions>
Service Tasks: Java Delegates invoke HttpClient.create_job()
Boundary Events: Handle timeouts, errors, and signal triggers
Reference: See mlc-building-blocks repository

monitoring Monitoring Job Status

Query job status and retrieve results once execution completes:

# Poll job status
import time

def wait_for_job(client, job_id, max_wait_seconds=1800):
    """Wait for job completion with exponential backoff."""
    start_time = time.time()
    attempt = 0
    
    while time.time() - start_time < max_wait_seconds:
        job_info = client.get_job_status(job_id)
        status = job_info['status']
        
        if status == 'COMPLETED':
            return job_info['results']
        elif status == 'FAILED':
            raise Exception(f"Job {job_id} failed: {job_info['error']}")
        
        # Exponential backoff: 5s, 10s, 20s, etc.
        wait_time = min(5 * (2 ** attempt), 60)
        time.sleep(wait_time)
        attempt += 1
    
    raise TimeoutError(f"Job {job_id} did not complete within {max_wait_seconds}s")

# Retrieve results
results = wait_for_job(client, job_id)
print(f"AUC Score: {results['auc_score']}")
print(f"Timestamp: {results['execution_timestamp']}")

Custom Integrations & Sensor Setup

Connect external data sources via datasource endpoints. Examples include Azure Blob Storage, Snowflake, and REST APIs. Follow the ETL pattern to bridge external systems with ModelOp's job execution engine.

cloud_download Datasource Endpoint Architecture

Endpoints define ETL bridges that extract data from external systems, transform it, and load it into ModelOp:

Extract: Query external system (S3, RDBMS, API)
Transform: Map fields, filter rows, schema validation
Load: Store as Asset in ModelOp's data catalog
Register: Link Asset to Job's dataAssetMappings

Endpoint definitions are stored in the datasource_endpoints/ directory and are reusable across multiple jobs.

extension Example: Azure Blob Connector

Production-ready Azure integration pattern:

name: "azure_prod_data_endpoint"
type: "AZURE_BLOB_STORAGE"
config:
  account_name: "mycompanydata"
  account_key: "${AZURE_ACCOUNT_KEY}"  # Store as secret
  container: "ml-datasets"
  path: "prod/monitoring/sample_*.parquet"

# Extract Configuration
extract:
  query: |
    SELECT score, label, feature_1, feature_2, feature_3
    FROM prod_batch
    WHERE date >= DATEADD(day, -7, CAST(GETDATE() AS DATE))
  format: "parquet"
  partition_by: "date"

# Transform Configuration
transform:
  - drop_columns: ["internal_id", "debug_flag"]
  - rename_fields:
      "model_score": "score_column"
      "actual_value": "label_column"
  - type_conversion:
      score_column: "float"
      label_column: "integer"
  - filter: "score_column IS NOT NULL AND label_column IS NOT NULL"

# Load Configuration
load:
  asset_name: "asset_azure_prod_7day_v1"
  asset_role: "SAMPLE_DATA"
  storage_location: "s3://modelop-assets/azure_imports/"
Authentication: Use environment secrets, not hardcoded keys
Scheduling: Pair with Airflow/Prefect for daily updates
Source: See datasource_endpoints/STABLE/azure_example/

sensors Custom Sensor & Red-Teaming Setup

Integrate custom sensors for adversarial testing and PII detection:

# PII Detection Sensor
import re
from modelop.schema import infer

def init(init_param):
    global PII_PATTERNS
    # Common PII regex patterns
    PII_PATTERNS = {
        'ssn': r'\d{3}-\d{2}-\d{4}',
        'email': r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}',
        'credit_card': r'\b(?:\d{4}[-\s]?){3}\d{4}\b',
        'phone': r'\b\d{3}[-.\s]?\d{3}[-.\s]?\d{4}\b'
    }

def metrics(data: pd.DataFrame):
    """Scans all string columns for PII patterns."""
    pii_findings = {}
    
    for col in data.select_dtypes(include=['object']).columns:
        for pii_type, pattern in PII_PATTERNS.items():
            matches = data[col].astype(str).str.contains(pattern, regex=True).sum()
            if matches > 0:
                pii_findings[f"{col}_{pii_type}_count"] = matches
                pii_findings[f"{col}_{pii_type}_flag"] = "ALERT"
    
    yield {
        "total_pii_instances": sum(pii_findings.values()),
        "details": pii_findings,
        "severity": "CRITICAL" if pii_findings else "PASS"
    }

SDK Integration & Result Querying

The moc_python_sdk provides comprehensive REST API bindings for all platform operations including schema generation, result querying, and notification management.

settings HttpClient Configuration

Initialize the SDK client with secure and unsecured modes:

from moc_python_sdk import HttpClient
import os

# SECURE MODE (Production)
client_secure = HttpClient(
    endpoint=os.getenv("MODELOP_ENDPOINT"),
    username=os.getenv("MODELOP_USERNAME"),
    password=os.getenv("MODELOP_PASSWORD"),
    verify_ssl=os.getenv("MODELOP_VERIFY_SSL", "true") == "true"
)

# UNSECURED MODE (Development)
client_dev = HttpClient(
    endpoint="http://localhost:8090",
    verify_ssl=False
)

# Configuration via Environment Variables
# MODELOP_ENDPOINT: Base URL
# MODELOP_OAUTH_SCOPE: OAuth scope (if applicable)
# MODELOP_VERIFY_SSL: SSL certificate verification
# MODELOP_TIMEOUT_SECONDS: Request timeout

search Querying Model Test Results

Retrieve and analyze monitor execution results:

# Query results by Model ID
results = client.get_model_test_results(
    modelId="model_v2",
    limit=100,
    offset=0,
    sort_by="execution_timestamp",
    sort_order="DESC"
)

# Parse results
for result in results:
    print(f"Model: {result['model_id']}")
    print(f"Monitor: {result['monitor_id']}")
    print(f"Status: {result['status']}")  # PASS, FAIL, ERROR
    print(f"Metrics: {result['metrics_json']}")
    print(f"Timestamp: {result['execution_timestamp']}")
    print("---")

# Query by stored model
stored_model_results = client.get_model_test_results_by_stored_model_id(
    storedModelId="sm_12345",
    days_back=7
)

notifications Notification Management

Subscribe to job completion and failure notifications:

# Retrieve notifications
notifications = client.get_notifications_by_stored_model_id(
    storedModelId="sm_12345",
    notification_type="JOB_COMPLETED"
)

# Process alerts
alerts = client.get_notifications_by_stored_model_id(
    storedModelId="sm_12345",
    notification_type="THRESHOLD_EXCEEDED"
)

for alert in alerts:
    print(f"Alert: {alert['message']}")
    print(f"Severity: {alert['severity']}")  # INFO, WARNING, CRITICAL
    print(f"Metric: {alert['metric_name']}")
    print(f"Threshold: {alert['threshold']}")
    print(f"Actual Value: {alert['actual_value']}")

schema Schema Generation & Inference

Auto-generate Avro schemas from data assets:

from moc_python_sdk import schema_infer

# Infer from DataFrame
df_schema = schema_infer.infer_schema_from_dataframe(
    df=df_sample,
    primary_keys=["id"],
    required_fields=["score", "label"]
)

# Generate Avro specification
avro_spec = schema_infer.generate_avro_spec(
    schema_dict=df_schema,
    namespace="com.company.models",
    doc="Production monitoring schema"
)

print(avro_spec)

MCP Services & Extensions

Extend ModelOp's capabilities using the Model Context Protocol (MCP). MCP enables secure tool integration, custom services, and seamless agent orchestration.

extension Model Context Protocol (MCP) Overview

MCP provides a standardized interface for tools and services to integrate with ModelOp's agent framework:

Protocol: JSON-RPC 2.0 over stdio, HTTP, or WebSocket
Server Role: Exposes tools and resources
Client Role: Consumes tools via agents
Authentication: OAuth2, API keys, mTLS

MCP separates concerns: Agents remain stateless while Services manage state and integrate with external systems.

code MCP SDKs & Support

Official MCP SDKs are available for multiple languages:

Python SDK
pip install mcp-sdk
Full feature parity with Node.js
TypeScript/JavaScript
npm install @modelop/mcp-sdk
Preferred for browser-based tools
Java SDK
<groupId>com.modelop</groupId>
Enterprise integration patterns
Kotlin SDK
gradle: com.modelop:mcp-kotlin
Modern JVM development

build Building a Custom MCP Service

Template for a custom MCP server that provides tools for model monitoring:

from mcp import Server, Tool, Resource
from mcp.server.stdio import StdioServerTransport

server = Server("model-monitoring-service")

@server.tool()
def create_monitor_job(model_id: str, monitor_type: str, data_asset: str):
    """Create a new monitoring job via MCP."""
    from moc_python_sdk import HttpClient
    
    client = HttpClient(endpoint=os.getenv("MODELOP_ENDPOINT"))
    job_response = client.create_job({
        "modelTestId": f"job_{model_id}_{monitor_type}",
        "monitorSnapshotId": f"snapshot_{monitor_type}_latest",
        "dataAssetMappings": {"sample_data": data_asset}
    })
    
    return {
        "job_id": job_response['id'],
        "status": job_response['status'],
        "created_at": job_response['created_timestamp']
    }

@server.tool()
def query_recent_results(model_id: str, days_back: int = 7):
    """Query monitoring results from the past N days."""
    client = HttpClient(endpoint=os.getenv("MODELOP_ENDPOINT"))
    results = client.get_model_test_results(modelId=model_id, limit=100)
    
    # Filter by date
    from datetime import datetime, timedelta
    cutoff_date = datetime.now() - timedelta(days=days_back)
    
    filtered = [r for r in results 
                if datetime.fromisoformat(r['execution_timestamp']) > cutoff_date]
    
    return {"count": len(filtered), "results": filtered}

@server.resource()
def get_monitor_status(model_id: str):
    """Expose monitor status as a resource for agent inspection."""
    client = HttpClient(endpoint=os.getenv("MODELOP_ENDPOINT"))
    status = client.get_model_status(model_id)
    return status

# Serve the MCP protocol
async def main():
    transport = StdioServerTransport()
    await server.run(transport)

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())
Tools: Executable functions agents can call
Resources: Read-only data sources agents can query
Error Handling: Proper exception propagation via JSON-RPC

psychology Integrating Services with Agents

Agents automatically discover MCP tools and route requests:

# Agent configuration discovers MCP services
# services.json or environment configuration:
{
  "mcp_servers": [
    {
      "name": "model-monitoring-service",
      "command": "python",
      "args": ["/path/to/monitoring_mcp_server.py"],
      "env": {
        "MODELOP_ENDPOINT": "https://modelop.company.com",
        "MODELOP_USERNAME": "${AGENT_USERNAME}",
        "MODELOP_PASSWORD": "${AGENT_PASSWORD}"
      }
    }
  ]
}

# Agent invokes tools discovered from MCP
# Example: "Create a monitoring job for model_v2 with drift detection"
# Agent:
#   1. Discovers "create_monitor_job" tool from model-monitoring-service
#   2. Calls with parameters: model_id="model_v2", monitor_type="drift"
#   3. Receives job_id, status from tool response
#   4. Chains additional tools (e.g., query_recent_results)

cloud_upload Deploying MCP Services

MCP services can be deployed in various environments:

Docker Container
Containerized service with MCP protocol support via stdio or TCP
Kubernetes Service
Deploy as Deployment with ingress for HTTP MCP endpoints
AWS Lambda
Serverless MCP handlers for event-driven monitoring