Bridging Standard Metrics & Enterprise Needs
ModelOp Center provides a comprehensive suite of out-of-the-box (OOTB) monitors. However, enterprise requirements often demand unique calculations. This guide introduces the capabilities of custom Python-based monitors.
code The Developer Value
Write standard Python code using Data Science libraries and have it automatically integrated into an enterprise-grade governance platform. No complex API integrations required.
verified_user The Governance Value
Ensures that no matter how complex the model metric, the "evidence" is always captured in a standardized, auditable format for risk and compliance.
Monitors by Model Modality
Custom monitors can be built to supplement any of these OOTB categories based on your domain.
Data Science Metrics Catalog
Hover or click on any metric pill below to view its algorithmic definition and dual-persona insights. These definitions guide standard and custom monitor creation.
1. Generative AI / NLP Validations
Secures conversational agents and generative models against hallucinations, toxic output, and data leakage.
2. Ethical Fairness & Bias
Evaluates model behavior disparities against protected classes to ensure regulatory compliance.
3. Regression & Credit Risk
Assesses continuous prediction errors and rank-ordering capabilities, crucial for financial/credit models.
4. Classification Performance
Evaluates discrete prediction models. Requires schemas mapping score_column and label_column.
5. Data & Concept Drift
Detects shifts in input distributions over time by comparing baseline data against a sample slice.
Enterprise Monitor Selection Pathway
Selecting the right AI monitor requires mapping the technical model type directly to enterprise business outcomes. Use this dynamic tree to navigate from your raw data state to the specific ModelOp monitors required.
manage_search Node Inspector
(Ctrl+Click for Multi)Model Selection
filter_alt Pathway Filters
Toggle model types below to dynamically generate the recommended monitor pathway branches.
Industry Context Presets
info Disclaimer: This is an enablement tool intended for guidance and is not to be used for final or automated business decision making.
Key Concepts: Execution Architectures
Interact with the architecture diagrams below. Toggle between standard monitoring and agentic LLM patterns. Both utilize uniform structures aligned through the Job Execution core.
manage_search Node Inspector
Ctrl+Click multiExecution Process
Select a node or edge in the diagram to view its details.
- Process initiated (via UI/API)
- Metrics job created (via MLC)
- Job sent to Runtime
- Runtime loads datasets & code
- Runtime executes Python source
- Output yielded as JSON
- Model Test Result attached
manage_search Node Inspector
Ctrl+Click multiAgentic Architecture
Select a node or edge in the diagram to view its details.
- Automated testing analysis
- Auto-Fill intake workflows
- Auto-Gen documentation templates
- Auto-Map risks from Policy
Artifact Explorer & Generator
A custom monitor is defined by specific files in a Git repository. Explore the required structure and generate contextual Data Science code boilerplates.
custom_metrics.py terminal
Primary Model Source Code
metadata.json data_object
Monitor Classification Meta
required_assets.json list_alt
Input Data Definitions
custom_metrics.py
The algorithmic brain. Use the pills below to generate boilerplate logic for different Data Science use cases.
Onboarding Roadmap
Follow this interactive guide to promote your custom monitor from a local IDE script into a production-ready ModelOp asset.
cloud_upload Connect Git Repository
expand_moreImport your custom code via the ModelOp UI. Navigate to Monitors > Add Monitor and select "Git".
- Provide the Repository URL and target Branch.
- Assign an Access Group to control viewing permissions.
- The system automatically scans for the
metrics()entry point.
camera Freeze a Snapshot
chevron_rightSnapshots create an immutable version of your monitor code linked to a specific commit.
This guarantees production stability, ensuring that subsequent commits to the Git branch don't silently alter or break actively scheduled tests.
link Map Data & Execute
chevron_rightAttach the monitor snapshot to your Business Model's "Monitoring" tab.
- The UI will prompt you to map specific data assets as defined in your
required_assets.json. - Click Play to spawn the Job and yield your metrics.
Out-of-the-Box (OOTB) Monitors Catalog
ModelOp Center provides an extensive library of pre-built monitors covering classification, regression, fairness, drift, stability, and volumetric analysis. Click on each group to explore available monitors and their configurations.
info Selecting the Right Monitor
Use the Selection Pathway tab to map your model type (Classification, Regression, or GenAI/LLM) to the appropriate OOTB monitor. Combine multiple monitors for comprehensive governance.
Job Scheduling & On-Demand Execution
Execute monitors via the moc_python_sdk HttpClient API or orchestrate complex workflows using MLC/BPMN templates. Supports both scheduled and on-demand metric generation.
api Creating Jobs with moc_python_sdk
The HttpClient provides a straightforward REST interface for job creation:
from moc_python_sdk import HttpClient
# Initialize secure client
client = HttpClient(
endpoint="https://modelop-instance.com",
username="api_user",
password="api_token",
verify_ssl=True
)
# Define job payload
job_payload = {
"modelTestId": "job_auc_monitor_prod_v1",
"monitorSnapshotId": "snapshot_auc_monitor_v2",
"dataAssetMappings": {
"baseline_data": "asset_train_dataset_v3",
"sample_data": "asset_prod_slice_20260416"
},
"executionContext": {
"priority": "HIGH",
"timeout_seconds": 1800,
"email_on_complete": "ml-team@company.com"
}
}
# Create the job
response = client.create_job(payload=job_payload)
job_id = response['id']
job_status = response['status'] # 'CREATED', 'RUNNING', 'COMPLETED'
flow_chart MLC/BPMN Subprocess Patterns
Complex monitoring workflows can be orchestrated using Camunda BPMN 2.0 subprocesses:
<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL">
<bpmn:process id="monitor_orchestration">
<bpmn:startEvent id="start" />
<bpmn:callActivity id="spawn_job"
calledElement="create_monitor_job">
<bpmn:ioMapping>
<bpmn:inputParameter name="monitorId">${monitorSnapshot}</bpmn:inputParameter>
<bpmn:outputParameter name="jobId">${job_response.id}</bpmn:outputParameter>
</bpmn:ioMapping>
</bpmn:callActivity>
<bpmn:boundaryEvent id="timeout_handler"
attachedToRef="spawn_job"
timerEventDefinition>
<bpmn:timerEventDefinition>
<bpmn:timeDuration>PT30M</bpmn:timeDuration>
</bpmn:timerEventDefinition>
</bpmn:boundaryEvent>
<bpmn:endEvent id="end" />
</bpmn:process>
</bpmn:definitions>
monitoring Monitoring Job Status
Query job status and retrieve results once execution completes:
# Poll job status
import time
def wait_for_job(client, job_id, max_wait_seconds=1800):
"""Wait for job completion with exponential backoff."""
start_time = time.time()
attempt = 0
while time.time() - start_time < max_wait_seconds:
job_info = client.get_job_status(job_id)
status = job_info['status']
if status == 'COMPLETED':
return job_info['results']
elif status == 'FAILED':
raise Exception(f"Job {job_id} failed: {job_info['error']}")
# Exponential backoff: 5s, 10s, 20s, etc.
wait_time = min(5 * (2 ** attempt), 60)
time.sleep(wait_time)
attempt += 1
raise TimeoutError(f"Job {job_id} did not complete within {max_wait_seconds}s")
# Retrieve results
results = wait_for_job(client, job_id)
print(f"AUC Score: {results['auc_score']}")
print(f"Timestamp: {results['execution_timestamp']}")
Custom Integrations & Sensor Setup
Connect external data sources via datasource endpoints. Examples include Azure Blob Storage, Snowflake, and REST APIs. Follow the ETL pattern to bridge external systems with ModelOp's job execution engine.
cloud_download Datasource Endpoint Architecture
Endpoints define ETL bridges that extract data from external systems, transform it, and load it into ModelOp:
Endpoint definitions are stored in the datasource_endpoints/ directory and are reusable across multiple jobs.
extension Example: Azure Blob Connector
Production-ready Azure integration pattern:
name: "azure_prod_data_endpoint"
type: "AZURE_BLOB_STORAGE"
config:
account_name: "mycompanydata"
account_key: "${AZURE_ACCOUNT_KEY}" # Store as secret
container: "ml-datasets"
path: "prod/monitoring/sample_*.parquet"
# Extract Configuration
extract:
query: |
SELECT score, label, feature_1, feature_2, feature_3
FROM prod_batch
WHERE date >= DATEADD(day, -7, CAST(GETDATE() AS DATE))
format: "parquet"
partition_by: "date"
# Transform Configuration
transform:
- drop_columns: ["internal_id", "debug_flag"]
- rename_fields:
"model_score": "score_column"
"actual_value": "label_column"
- type_conversion:
score_column: "float"
label_column: "integer"
- filter: "score_column IS NOT NULL AND label_column IS NOT NULL"
# Load Configuration
load:
asset_name: "asset_azure_prod_7day_v1"
asset_role: "SAMPLE_DATA"
storage_location: "s3://modelop-assets/azure_imports/"
datasource_endpoints/STABLE/azure_example/sensors Custom Sensor & Red-Teaming Setup
Integrate custom sensors for adversarial testing and PII detection:
# PII Detection Sensor
import re
from modelop.schema import infer
def init(init_param):
global PII_PATTERNS
# Common PII regex patterns
PII_PATTERNS = {
'ssn': r'\d{3}-\d{2}-\d{4}',
'email': r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}',
'credit_card': r'\b(?:\d{4}[-\s]?){3}\d{4}\b',
'phone': r'\b\d{3}[-.\s]?\d{3}[-.\s]?\d{4}\b'
}
def metrics(data: pd.DataFrame):
"""Scans all string columns for PII patterns."""
pii_findings = {}
for col in data.select_dtypes(include=['object']).columns:
for pii_type, pattern in PII_PATTERNS.items():
matches = data[col].astype(str).str.contains(pattern, regex=True).sum()
if matches > 0:
pii_findings[f"{col}_{pii_type}_count"] = matches
pii_findings[f"{col}_{pii_type}_flag"] = "ALERT"
yield {
"total_pii_instances": sum(pii_findings.values()),
"details": pii_findings,
"severity": "CRITICAL" if pii_findings else "PASS"
}
SDK Integration & Result Querying
The moc_python_sdk provides comprehensive REST API bindings for all platform operations including schema generation, result querying, and notification management.
settings HttpClient Configuration
Initialize the SDK client with secure and unsecured modes:
from moc_python_sdk import HttpClient
import os
# SECURE MODE (Production)
client_secure = HttpClient(
endpoint=os.getenv("MODELOP_ENDPOINT"),
username=os.getenv("MODELOP_USERNAME"),
password=os.getenv("MODELOP_PASSWORD"),
verify_ssl=os.getenv("MODELOP_VERIFY_SSL", "true") == "true"
)
# UNSECURED MODE (Development)
client_dev = HttpClient(
endpoint="http://localhost:8090",
verify_ssl=False
)
# Configuration via Environment Variables
# MODELOP_ENDPOINT: Base URL
# MODELOP_OAUTH_SCOPE: OAuth scope (if applicable)
# MODELOP_VERIFY_SSL: SSL certificate verification
# MODELOP_TIMEOUT_SECONDS: Request timeout
search Querying Model Test Results
Retrieve and analyze monitor execution results:
# Query results by Model ID
results = client.get_model_test_results(
modelId="model_v2",
limit=100,
offset=0,
sort_by="execution_timestamp",
sort_order="DESC"
)
# Parse results
for result in results:
print(f"Model: {result['model_id']}")
print(f"Monitor: {result['monitor_id']}")
print(f"Status: {result['status']}") # PASS, FAIL, ERROR
print(f"Metrics: {result['metrics_json']}")
print(f"Timestamp: {result['execution_timestamp']}")
print("---")
# Query by stored model
stored_model_results = client.get_model_test_results_by_stored_model_id(
storedModelId="sm_12345",
days_back=7
)
notifications Notification Management
Subscribe to job completion and failure notifications:
# Retrieve notifications
notifications = client.get_notifications_by_stored_model_id(
storedModelId="sm_12345",
notification_type="JOB_COMPLETED"
)
# Process alerts
alerts = client.get_notifications_by_stored_model_id(
storedModelId="sm_12345",
notification_type="THRESHOLD_EXCEEDED"
)
for alert in alerts:
print(f"Alert: {alert['message']}")
print(f"Severity: {alert['severity']}") # INFO, WARNING, CRITICAL
print(f"Metric: {alert['metric_name']}")
print(f"Threshold: {alert['threshold']}")
print(f"Actual Value: {alert['actual_value']}")
schema Schema Generation & Inference
Auto-generate Avro schemas from data assets:
from moc_python_sdk import schema_infer
# Infer from DataFrame
df_schema = schema_infer.infer_schema_from_dataframe(
df=df_sample,
primary_keys=["id"],
required_fields=["score", "label"]
)
# Generate Avro specification
avro_spec = schema_infer.generate_avro_spec(
schema_dict=df_schema,
namespace="com.company.models",
doc="Production monitoring schema"
)
print(avro_spec)
MCP Services & Extensions
Extend ModelOp's capabilities using the Model Context Protocol (MCP). MCP enables secure tool integration, custom services, and seamless agent orchestration.
extension Model Context Protocol (MCP) Overview
MCP provides a standardized interface for tools and services to integrate with ModelOp's agent framework:
MCP separates concerns: Agents remain stateless while Services manage state and integrate with external systems.
code MCP SDKs & Support
Official MCP SDKs are available for multiple languages:
pip install mcp-sdknpm install @modelop/mcp-sdk<groupId>com.modelop</groupId>gradle: com.modelop:mcp-kotlinbuild Building a Custom MCP Service
Template for a custom MCP server that provides tools for model monitoring:
from mcp import Server, Tool, Resource
from mcp.server.stdio import StdioServerTransport
server = Server("model-monitoring-service")
@server.tool()
def create_monitor_job(model_id: str, monitor_type: str, data_asset: str):
"""Create a new monitoring job via MCP."""
from moc_python_sdk import HttpClient
client = HttpClient(endpoint=os.getenv("MODELOP_ENDPOINT"))
job_response = client.create_job({
"modelTestId": f"job_{model_id}_{monitor_type}",
"monitorSnapshotId": f"snapshot_{monitor_type}_latest",
"dataAssetMappings": {"sample_data": data_asset}
})
return {
"job_id": job_response['id'],
"status": job_response['status'],
"created_at": job_response['created_timestamp']
}
@server.tool()
def query_recent_results(model_id: str, days_back: int = 7):
"""Query monitoring results from the past N days."""
client = HttpClient(endpoint=os.getenv("MODELOP_ENDPOINT"))
results = client.get_model_test_results(modelId=model_id, limit=100)
# Filter by date
from datetime import datetime, timedelta
cutoff_date = datetime.now() - timedelta(days=days_back)
filtered = [r for r in results
if datetime.fromisoformat(r['execution_timestamp']) > cutoff_date]
return {"count": len(filtered), "results": filtered}
@server.resource()
def get_monitor_status(model_id: str):
"""Expose monitor status as a resource for agent inspection."""
client = HttpClient(endpoint=os.getenv("MODELOP_ENDPOINT"))
status = client.get_model_status(model_id)
return status
# Serve the MCP protocol
async def main():
transport = StdioServerTransport()
await server.run(transport)
if __name__ == "__main__":
import asyncio
asyncio.run(main())
psychology Integrating Services with Agents
Agents automatically discover MCP tools and route requests:
# Agent configuration discovers MCP services
# services.json or environment configuration:
{
"mcp_servers": [
{
"name": "model-monitoring-service",
"command": "python",
"args": ["/path/to/monitoring_mcp_server.py"],
"env": {
"MODELOP_ENDPOINT": "https://modelop.company.com",
"MODELOP_USERNAME": "${AGENT_USERNAME}",
"MODELOP_PASSWORD": "${AGENT_PASSWORD}"
}
}
]
}
# Agent invokes tools discovered from MCP
# Example: "Create a monitoring job for model_v2 with drift detection"
# Agent:
# 1. Discovers "create_monitor_job" tool from model-monitoring-service
# 2. Calls with parameters: model_id="model_v2", monitor_type="drift"
# 3. Receives job_id, status from tool response
# 4. Chains additional tools (e.g., query_recent_results)
cloud_upload Deploying MCP Services
MCP services can be deployed in various environments: