Streaming & Real-Time
PlannedEdge-to-Cloud Real-Time Risk Intelligence Platform
Streaming feature computation and scoring from edge to cloud.
This system is planned. The case study describes the intended design; metrics are targets or planned, not measured results.
Problem
Batch risk scoring reacts too slowly. Moving to streaming introduces hard problems: backpressure, exactly-enough processing, and replay after failures.
Why this matters
Streaming feature computation and low-latency scoring underpin real-time ML for fraud, abuse, and safety. These edge-to-cloud reliability problems are exactly what platform teams own at scale.
Constraints
- Bounded end-to-end scoring latency under load.
- Backpressure and overload handled without data loss.
- Pipelines are replayable for recovery and audit.
- Edge compute is thin; heavy logic stays in the cloud.
Architecture
Edge
- Collectorsthin
- Event publish
Stream
- BrokerKafka
- Feature windows
Scoring
- Real-time modelRust/Go
- Backpressure control
Act
- Alerting
- Replay / recovery
Data flow
Edge events publish to the broker, the processor computes windowed features and scores them, alerts fire with correlation IDs, and the log can be replayed for recovery.
Control plane vs data plane
Control: Pipeline configuration, windowing rules, alert thresholds, and replay control.
Data: The event path: ingest, windowed feature computation, scoring, and alert emission.
Core capabilities
- Streaming feature computation with windowing.
- Low-latency scoring with a real-time model path.
- Alerting with clear severity and correlation IDs.
- Replay/recovery for failure and audit scenarios.
Staff-level tradeoffs
Thin edge, heavy cloud.
Keeps edge deployment simple and cheap while centralizing the hard scoring logic.
Replayable log as the source of truth.
Enables recovery and audit, and makes the pipeline testable with recorded streams.
Tech stack
Systems
- Rust
Backend
- Go
ML / Data
- Real-time ML
- Kafka
- event schemas
Infrastructure
- Kubernetes
- object storage for replay
Metrics
- End-to-end scoring latency Target
- Bounded under load
- Throughput Target
- Events/sec at target latency
- Data loss under backpressure Planned
- None (replayable log)
- Recovery Planned
- Replay from log
Metrics are labeled measured, target, or planned. Nothing here is an achieved result unless it is marked measured.
Failure modes
Backpressure / overload
Bounded queues shed or buffer to the replayable log rather than dropping data.
Consumer lag grows
Processors autoscale and lag is alerted before it breaches latency SLOs.
Node failure
Recover by replaying from the log rather than losing in-flight events.
What's next
- Prototype the stream processor with a recorded event set.
- Add windowed features, then the scoring path and alerting.