Cklig Statistics Flow v2.0

Complete System Architecture & Technical Documentation

Server-Side Event Aggregation with Optimized Idle Monitor

System Overview

The Cklig Statistics Flow v2.0 introduces server-side event aggregation with in-memory stack processing for improved performance and resource utilization. The system consists of four main components working together to collect, process, and manage user interactions on client websites.

Latest Enhancement: Optimized Idle Monitor System

  • 1-minute interval checks - Frequent monitoring ensures no data is lost
  • Smart stack reduction - Processed stacks are immediately removed, reducing next iteration workload
  • Non-empty validation - Only stacks with events > 0 are transported to Cklig API
  • Memory optimization - Progressive stack count reduction improves performance

Client Website

Vue.js Widget Script
  • Real-time event tracking
  • Widget display management
  • User interaction capture
  • Domain verification
  • Local fallback storage

Statistics Server

Node.js + MongoDB
  • In-memory stack objects
  • Server-side event counting
  • Optimized 1-min idle monitoring
  • MongoDB persistence
  • WebSocket batch transport
  • Smart stack lifecycle management

Cklig API

Laravel + GraphQL
  • Widget authentication
  • Token management
  • Batch multi-insert operations
  • AccountStatisticsFlag control
  • Business logic processing

Cklig Planet

Laravel Package
  • Usage consumption tracking
  • Service limit enforcement
  • Token lifecycle control
  • Statistics flag management
  • Cascade deletion triggers

System Architecture Flow

1. Track Events
2. Count & Stack
(In Memory)
3. Interval Check
(1-min cycle)
4. Batch WebSocket
Push
5. Multi-Insert
& Count
6. Usage Check
7. Token/Flag
Management

Key Innovation in v2.0: Server-side event aggregation with optimized idle monitoring. 1-minute interval checks with intelligent stack reduction ensure data delivery while minimizing system load. Each processed stack is immediately removed, progressively reducing the number of objects to check in subsequent cycles.

Data Flow Process

Phase 1: Initial Authentication
  • Widget Script Load: Client website loads widget script on page load
  • Widget Authentication: Script requests verified widgets from cklig-api using account_id and domain verification
  • JWT Token Generation: API generates JWT tokens for verified widgets (7-day expiration)
  • Statistics Token Request: Script requests UUID token for statistics server communication
  • Token Distribution: API creates UUID token, stores in GraphQL, and sends to both client website and statistics server
  • Flag Initialization: AccountStatisticsFlag created with enable_statistics: true for account+host combination
Phase 2: Server-Side Event Processing
  • Event Collection: Widget script tracks user interactions and sends to statistics server
  • Stack Management: Statistics server updates in-memory stack object
  • Stack key: Statistics token (account_id + host unique combination)
  • Events added to stack.events object (sessions/pages/clicks)
  • Counters incremented in stack.counters object
  • Updated_at timestamp refreshed on each event
  • MongoDB storage continues in parallel for persistence
  • Interval Monitoring: Server checks two conditions every minute
  • Event-based: sum(counters) >= configured interval threshold
  • Time-based: now() - stack.updated_at > 30-minute idle_threshold
Phase 3: Optimized Batch Processing
  • 1-Minute Monitor Cycle: Background process runs every 60 seconds
  • Stack Iteration: Process checks all active in-memory stacks
  • Condition Evaluation: Check if either trigger condition is met
  • Event threshold reached (e.g., 50 events), OR
  • Idle time exceeded (30 minutes since last update)
  • Non-Empty Validation: Only process stacks with events.length > 0
  • WebSocket Transport: Complete stack sent to Cklig API as single batch
  • Immediate Stack Removal: Processed stacks deleted from memory
  • Progressive Optimization: Fewer stacks to check in next cycle
Phase 4: Usage Management & Control
  • Multi-Insert Operations: User journey events bulk inserted into OnSiteUserJourneyEvent
  • New AccountStatisticsCounter row inserted with batch counts
  • Usage Logging: logUsage called with total batch count
  • Limit Check: Planet verifies account usage limits against batch total
  • Flag Update: If exceeded, AccountStatisticsFlag.enable_statistics set to false
  • Cascade Delete: Cklig API deletes AccountStatisticsToken
  • Delete request sent to Statistics Server
  • Statistics Server removes token and associated stack from memory
  • Service Control: Widget continues local tracking but server processing stops

Entity Relationship Diagrams

Core Authentication Entities

Account
  • id (Primary Key)
  • accountType
  • active (boolean)
  • verified (boolean)
  • createdAt
  • updatedAt
Widget
  • id (Primary Key)
  • account (Foreign Key)
  • domain (Foreign Key)
  • type (ButtonForm/Popup)
  • active (boolean)
  • createdAt
  • updatedAt
Domain
  • id (Primary Key)
  • account (Foreign Key)
  • name (example.com)
  • createdAt
  • updatedAt

Token Management Entities

WidgetToken
  • id (Primary Key)
  • widget (Foreign Key)
  • token (JWT String)
  • host (domain)
  • expiresAt
  • createdAt
AccountStatisticsToken
  • id (Primary Key)
  • account (Foreign Key)
  • token (UUID String)
  • host (domain)
  • expiresAt
  • createdAt
AccountStatisticsFlag
  • account (Composite PK)
  • host (Composite PK)
  • enableHostStatistics (boolean)
  • createdAt
  • updatedAt

Statistics & Journey Tracking Entities

OnSiteUserJourney
  • id (Primary Key)
  • sessionId (UUID)
  • account (Foreign Key)
  • userIp (encrypted)
  • baseDomain
  • createdAt
  • updatedAt
OnSiteUserJourneyEvent
  • id (Primary Key)
  • onSiteUserJourney (Foreign Key)
  • type (session/page/click)
  • step (integer)
  • event (JSON)
  • pageUrl (String)
  • timestamp
  • createdAt
AccountStatisticsCounter
  • id (Primary Key)
  • account (Foreign Key)
  • host
  • sessionEventCounter
  • pageEventCounter
  • clickEventCounter
  • createdAt
  • updatedAt

MongoDB Collections (Statistics Server)

sessions
  • _id (ObjectId)
  • account_id
  • user_id (UUID)
  • session_id
  • ip_address
  • browser_info
  • screen_info
  • base_domain
  • created_at
page_events
  • _id (ObjectId)
  • account_id
  • user_id
  • session_id
  • event_type (PageLoad/PageUnload)
  • page_url
  • page_details
  • time_on_page
  • referrer
  • timestamp
click_events
  • _id (ObjectId)
  • account_id
  • user_id
  • session_id
  • event_type (Click/Focus/Blur)
  • target_element
  • position_data
  • form_data (optional)
  • timestamp

In-Memory Stack Structure

// Statistics Server In-Memory Stack Object
{
  "stack_key": "uuid_statistics_token",
  "account_id": "account_123",
  "host": "example.com",
  "created_at": "2025-09-24T10:00:00Z",
  "updated_at": "2025-09-24T10:15:30Z",
  "interval_threshold": 50,
  "idle_threshold_minutes": 30,

  "events": {
    "sessions": [
      {
        "type": "sessions-added",
        "event": {
          "cklig-track-account": "account_123",
          "cklig-track-user-id": "user_456",
          "cklig-track-user-session": "session_789",
          "cklig-track-browser": {
            "name": "Chrome",
            "version": "117.0.0.0"
          }
        },
        "timestamp": "2025-09-24T10:00:15Z"
      }
    ],
    "pages": [
      {
        "type": "pages-added",
        "event": {
          "cklig-track-event-type": "PageLoad",
          "cklig-track-event-page-url": "https://example.com/products",
          "cklig-track-event-timeonpage": 45000
        },
        "timestamp": "2025-09-24T10:01:30Z"
      }
    ],
    "clicks": [
      {
        "type": "events-added",
        "event": {
          "cklig-track-event-type": "Click",
          "cklig-track-event-target": {
            "tagName": "BUTTON",
            "id": "add-to-cart",
            "className": "btn btn-primary"
          }
        },
        "timestamp": "2025-09-24T10:02:45Z"
      }
    ]
  },

  "counters": {
    "sessions": 1,
    "pages": 2,
    "clicks": 2,
    "total": 5
  },

  "metadata": {
    "memory_usage_bytes": 2048,
    "last_flush_at": "2025-09-24T09:45:00Z",
    "flush_trigger": "pending"
  }
}

Process Diagrams

Optimized Idle Monitor Process

Enhanced Idle Monitor Logic (v2.0 Update)

  • 60-second interval: Background process runs every minute for frequent monitoring
  • Idle threshold check: Evaluates if now() - stack.updated_at > 30 minutes
  • Non-empty validation: Only processes stacks where events.length > 0
  • Immediate removal: Processed stacks deleted from memory immediately
  • Progressive optimization: Each cycle has fewer stacks to check
  • Memory efficiency: Continuous reduction of active stack count
1-Minute Idle Monitor Cycle
  • Background Process Activation: Timer triggers every 60 seconds
  • Stack Inventory Check: Iterate through all active in-memory stack objects
  • Idle Time Calculation: Compare current time with stack.updated_at timestamp
  • Threshold Evaluation: Check if idle time > 30 minutes
  • Content Validation: Verify stack has events.length > 0
  • Transport Decision: Process only non-empty idle stacks
Smart Stack Processing
  • Batch Serialization: Convert stack object to WebSocket message format
  • API Transport: Send complete stack to Cklig API via WebSocket
  • Acknowledgment Wait: Confirm successful receipt by API
  • Immediate Memory Cleanup: Delete processed stack from memory
  • Stack Count Reduction: Next cycle has fewer objects to process
  • Performance Optimization: System efficiency improves over time
Memory & Performance Benefits
  • Reduced Iteration Overhead: Fewer stacks checked each minute
  • Memory Usage Optimization: Continuous stack cleanup
  • Faster Processing Times: Less computational load per cycle
  • Improved System Responsiveness: Lower memory footprint
  • Scalability Enhancement: Better performance under load
  • Resource Conservation: Optimal memory utilization

Detailed Sequence Flow Diagrams

Authentication Sequence Flow

1

Widget Script Initialization

Client website loads widget script on page load with account_id parameter. Script initializes Vue.js component and prepares for authentication.

2

Widget Authentication Request

GET /site/account/{account_id}/widgets with Referer header for domain verification. API validates the requesting domain against registered domains.

3

JWT Token Generation & Validation

API verifies domain authorization, generates JWT tokens for verified widgets with 7-day expiration, including widget_id and account_id in claims.

4

Statistics Token Request

GET /site/widget/{widget_id}/account_statistics using Bearer JWT authentication to request UUID statistics token for server communication.

5

UUID Token Creation & Distribution

API generates UUID token, stores in AccountStatisticsToken table, creates AccountStatisticsFlag, and distributes to both client and statistics server via POST /api/token.

6

Statistics Server Stack Initialization

Statistics server receives token, creates in-memory stack object with account_id and host as key, initializes counters and event arrays.

Event Processing Sequence Flow

1

User Interaction Capture

Widget script captures user interactions (page loads, clicks, form interactions) and formats event data with session context, timestamps, and element details.

2

Event Transmission to Statistics Server

Events sent to statistics server endpoints (/api/sessions/add, /api/pages/add, /api/events/add) using UUID token authentication.

3

Dual Storage Strategy

Statistics server stores events in MongoDB for persistence AND adds to in-memory stack object, incrementing appropriate counters (sessions/pages/clicks).

4

Interval Monitoring & Trigger Evaluation

Server continuously monitors two conditions: event count threshold (e.g., 50 events) OR idle timeout (e.g., 30 minutes since last update).

5

Batch Transport Trigger

When either condition is met, complete stack object is serialized and sent to Cklig API via WebSocket as single batch message containing all accumulated events.

6

GraphQL Batch Processing

Cklig API processes batch with multi-insert operations into OnSiteUserJourneyEvent table and creates new AccountStatisticsCounter row with batch totals.

Enhanced Idle Monitor Sequence Flow

⏰

1-Minute Timer Activation

Background job runs every 60 seconds to check for idle stacks. Frequent monitoring ensures no data is lost even for low-activity accounts.

🔄

Stack Iteration & Evaluation

Process iterates through all active in-memory stacks, evaluating each for idle condition: now() - stack.updated_at > 30 minutes

✅

Idle Stack Detection with Validation

Stacks exceeding 30-minute idle threshold are identified for processing, but only if events.length > 0 to avoid empty batch transport.

🚀

Selective Batch Transport

Only non-empty idle stacks are processed through the batch transport mechanism, ensuring efficient use of WebSocket resources.

🧹

Immediate Stack Removal

Processed stacks are immediately deleted from memory, reducing the number of objects to check in the next 1-minute cycle.

📊

Progressive Performance Optimization

Each cycle becomes more efficient as fewer stacks need evaluation, leading to improved system performance over time.

Usage Control Sequence Flow

1

Batch Usage Calculation

After batch processing, system calculates total usage from batch (sum of session + page + click events) for Planet consumption logging.

2

Planet Integration & Limit Verification

Planet package receives logUsage("Statistics", batch_total) call, verifies against account limits and consumption history.

3

Limit Exceeded Response

If limits exceeded, Planet triggers limit enforcement: AccountStatisticsFlag.enable_statistics set to false for account+host combination.

4

Cascade Token Deletion

System initiates cascade deletion: AccountStatisticsToken deleted from database, DELETE /api/token/{token} sent to Statistics Server.

5

Stack Memory Cleanup

Statistics server removes token from validation and deletes associated in-memory stack object, freeing memory resources.

6

Graceful Service Degradation

Widget continues functioning with local storage only, maintaining user experience while server-side processing is suspended for the account.

WebSocket Message Formats

V2.0 Batch Message Structure

Key Innovation: In V2.0, the statistics server sends events as batches instead of individual messages, dramatically reducing WebSocket traffic.

// V2.0 Batch Message Format - Array of Mixed Event Types
[
  {
    "type": "sessions-added",
    "event": {
      "cklig-track-account": "account_123",
      "cklig-track-user-id": "uuid_user_456",
      "cklig-track-user-session": "session_789",
      "cklig-track-user-ip": "192.168.1.100",
      "cklig-track-browser": {
        "name": "Chrome",
        "version": "117.0.0.0",
        "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "platform": "Windows",
        "language": "en-US"
      },
      "cklig-track-base-domain": "example.com",
      "cklig-track-screen": {
        "width": 1920,
        "height": 1080,
        "colorDepth": 24,
        "orientation": "landscape"
      }
    },
    "timestamp": "2025-09-24T10:15:30.123Z"
  },
  {
    "type": "events-added",
    "event": {
      "cklig-track-user-id": "uuid_user_456",
      "cklig-track-event-type": "Click",
      "cklig-track-event-target": {
        "tagName": "BUTTON",
        "id": "add-to-cart",
        "className": "btn btn-primary",
        "position": {
          "clientX": 450,
          "clientY": 320
        }
      }
    },
    "timestamp": "2025-09-24T10:17:12.789Z"
  }
  // ... additional events in batch (up to 50 events total)
]

V2.0 Batch Processing Benefits:

  • Events sent as array instead of individual WebSocket messages
  • Mixed event types (sessions, pages, clicks) in single batch
  • Events maintain chronological order within batch
  • Batch triggered by event count (50) OR idle timeout (30 min)
  • ~100:1 reduction in WebSocket message frequency
  • Enhanced idle monitoring with 1-minute cycles ensures data delivery

API Reference

Cklig API Endpoints

Method Endpoint Purpose Authentication
GET /site/account/{id}/widgets Get verified widgets for account Domain verification (Referer header)
GET /site/widget/{widget_id}/account_statistics Request statistics UUID token Bearer JWT (widget token)

Statistics Server - Token Endpoints

Method Endpoint Purpose Authentication
POST /api/token Create statistics UUID token Internal API
PUT /api/token/{token} Extend statistics token expiration Internal API
DELETE /api/token/{token} Delete statistics token Internal API

Statistics Server - Event Endpoints

Method Endpoint Purpose Authentication
POST /api/sessions/add Add user session data Bearer UUID
POST /api/pages/add Add page event data Bearer UUID
POST /api/events/add Add interaction events Bearer UUID
POST /api/referrers/add Add referrer data Bearer UUID

Security & Validation Features

Multi-Layer Security Architecture

Domain Verification
  • Referer Header Validation: Matches registered domains
  • CORS Configuration: Strict origin policies
  • Widget Token Binding: Tokens tied to specific domains
  • Cross-Domain Protection: Prevents unauthorized usage
  • Host Validation: Statistics tokens linked to verified hosts
Token Security
  • JWT Signing: RS256 algorithm with key rotation
  • Claims Validation: widget_id, account_id, domain verification
  • Expiration Management: 7-day sliding window
  • UUID Generation: Cryptographically secure random tokens
  • Token Revocation: Immediate invalidation capability
Data Protection
  • IP Address Hashing: PII protection in storage
  • Event Sanitization: Input validation and filtering
  • HTTPS Enforcement: All communications encrypted
  • Database Encryption: At-rest data protection
  • Access Logging: Comprehensive audit trails

Monitoring & Health Checks

System Health Monitoring

// GET /api/health/stacks - Complete Health Response
{
  "status": "healthy",
  "timestamp": "2025-09-24T10:30:00Z",
  "version": "2.0.1",
  "uptime_seconds": 345600,

  "statistics_server": {
    "active_stacks": 245,
    "total_events_in_memory": 8750,
    "memory_usage_mb": 187.5,
    "memory_limit_mb": 500,
    "memory_utilization_percent": 37.4,
    "oldest_stack_age_minutes": 12,
    "mongodb_connection_status": "connected"
  },

  "idle_monitor_statistics": {
    "monitor_interval_seconds": 60,
    "last_monitor_run": "2025-09-24T10:29:00Z",
    "stacks_processed_last_cycle": 3,
    "idle_triggered_flushes_last_hour": 23,
    "event_triggered_flushes_last_hour": 156,
    "empty_stacks_skipped_last_hour": 7,
    "average_cycle_processing_time_ms": 15.3
  },

  "flush_statistics": {
    "failed_flushes_last_hour": 2,
    "flush_success_rate_percent": 98.9,
    "average_batch_size": 47.3,
    "websocket_connection_status": "connected"
  },

  "health_score": 98.7
}

Key Features & Performance

Performance Improvements

Idle Monitor System Enhancements

  • Reduced monitoring interval: From 30-60 minutes to 1 minute for better responsiveness
  • Smart stack reduction: Immediate removal of processed stacks reduces iteration overhead
  • Non-empty validation: Only stacks with events are transported, avoiding unnecessary API calls
  • Progressive optimization: Each cycle becomes more efficient as fewer stacks require checking
  • Memory efficiency: Continuous stack cleanup prevents memory bloat
Message Reduction
  • ~100:1 ratio (Batch vs Individual messages)
  • Dramatically reduces WebSocket traffic
  • Lower network overhead and latency
  • Enhanced with optimized idle monitoring
Database Efficiency
  • ~90% fewer database insert operations
  • Bulk multi-insert operations
  • Reduced database load and improved performance
  • Optimized batch processing with smart triggers
Data Delivery Guarantee
  • Dual trigger mechanism ensures delivery
  • Event threshold OR idle timeout protection
  • 1-minute monitoring cycle ensures responsiveness
  • Maximum 30-minute delay for low-activity accounts

Cklig Statistics Flow v2.0

Complete technical documentation with server-side event aggregation and optimized idle monitoring

Generated for PhonePlus Statistics Flow Project

Latest Update: Enhanced Idle Monitor System

  • 60-second monitoring cycles replace previous 30-60 minute intervals
  • Smart stack reduction through immediate removal of processed objects
  • Non-empty validation ensures only meaningful data is transported
  • Progressive optimization improves system performance over time
  • Memory efficiency through continuous cleanup and optimization

Key Features: Entity Relationships • Process Flows • Sequence Diagrams • Optimized Idle Monitoring • Batch Processing • Multi-Insert Operations • Usage Control • Token Lifecycle Management • Real-time Health Monitoring • Progressive Memory Optimization