#atom

Subtitle:

Measuring and understanding AI system utilization and performance

Core Idea:

AI Usage Analytics involves collecting, processing, and analyzing data about how AI components are being used within applications to gain insights about user behavior, system performance, and overall value delivery.

Key Principles:

  1. Comprehensive Data Collection:
    • Track all relevant metrics including request volumes, response characteristics, user interactions, and system performance.
  2. User-Centric Analysis:
    • Tie AI usage to specific users or user segments to understand adoption patterns and value creation.
  3. Continuous Improvement Loop:
    • Use analytics insights to drive iterative enhancements to AI implementations.

Why It Matters:

How to Implement:

  1. Event Instrumentation:
    • Add tracking code at all AI interaction points within the application.
  2. Analytics Pipeline:
    • Establish data collection, processing, and storage infrastructure.
  3. Visualization & Reporting:
    • Create dashboards that present key metrics and trends in an actionable format.

Example:

import { PostHog } from 'posthog-node';

// Initialize analytics client
const analytics = new PostHog(process.env.POSTHOG_API_KEY, {
host: 'https://app.posthog.com'
});

class SupportAssistantAnalytics {
async trackInteraction(interactionData) {
const {
userId,
sessionId,
prompt,
response,
model,
responseTime,
inputTokens,
outputTokens,
userRating,
wasHelpful,
category
} = interactionData;

// Track the basic interaction
analytics.capture({
distinctId: userId,
event: 'ai_assistant_interaction',
properties: {
sessionId,
category,
model,
responseTimeMs: responseTime,
inputTokenCount: inputTokens,
outputTokenCount: outputTokens,
estimatedCost: this.calculateCost(model, inputTokens, outputTokens)
}
});

// If user provided feedback, track it separately
if (userRating !== undefined) {
analytics.capture({
distinctId: userId,
event: 'ai_assistant_feedback',
properties: {
sessionId,
rating: userRating,
wasHelpful,
category
}
});
}

// Check for potential issues in response
if (responseTime > 5000) {
analytics.capture({
distinctId: userId,
event: 'ai_slow_response',
properties: {
sessionId,
responseTimeMs: responseTime,
model
}
});
}
}

calculateCost(model, inputTokens, outputTokens) {
// Calculate cost based on model and token counts
const pricing = {
'gpt-4': { input: 0.03, output: 0.06 },
'gpt-3.5-turbo': { input: 0.0015, output: 0.002 },
'claude-3-opus': { input: 0.015, output: 0.075 },
'gemini-1.5-pro': { input: 0.00025, output: 0.0005 }
};

const modelPrice = pricing[model] || pricing['gpt-3.5-turbo']; // default fallback

return (
(inputTokens / 1000) * modelPrice.input +
(outputTokens / 1000) * modelPrice.output
).toFixed(4);
}

// Analysis helpers
async getUserAdoption(timeframe = '30d') {
// Query PostHog for user adoption metrics
const result = await analytics.query( SELECT count(distinct person_id) as active_users, count(*) as total_interactions, avg(properties.responseTimeMs) as avg_response_time FROM events WHERE event = 'ai_assistant_interaction' AND timestamp > now() - interval '${timeframe}' GROUP BY date_trunc('day', timestamp) ORDER BY date_trunc('day', timestamp) );

return result;
}

async getTopCategories(timeframe = '30d') {
// Query PostHog for most used assistant categories
const result = await analytics.query( SELECT properties.category, count(*) as usage_count, avgfloat) as avg_cost FROM events WHERE event = 'ai_assistant_interaction' AND timestamp > now() - interval '${timeframe}' GROUP BY properties.category ORDER BY usage_count DESC LIMIT 10 ;

return result;
}
}
```

Connections:

References:

  1. Primary Source:
    • Documentation from analytics platforms like PostHog, Amplitude, or Mixpanel
  2. Additional Resources:
    • Best practices for event tracking design
    • Visualization techniques for AI metrics

Tags:

#AIAnalytics #usageTracking #productMetrics #userInsights #dataAnalysis #AIPerformance #analyticsImplementation

Connections:

Sources: