Author(s): Leapfrog Technology

Originally published on Towards AI.

WebSockets are deceptively simple. Every connected user maintains a persistent connection to the server, and each connection continuously occupies server resources such as memory, CPU cycles, network buffers, and application state. Unlike traditional HTTP requests, WebSocket connections are long-lived and stateful, meaning resource consumption grows almost linearly with the number of users. You can, however, derive a simple mathematical model to introduce the problem.

The fundamental problem

Every active WebSocket connection consumes server resources. For each connection, a Node.js process maintains:

• A TCP socket
• Read/write buffers
• Event listeners
• User session metadata
• Heartbeat/ping state

So:
Total Memory=N×Mc
where

• N = number of concurrent users
• Mc​ = memory consumed per connection

Resource usage assumptions for 1000 concurrent users establishing WebSocket connections

The following estimates assume:

• 1,000 simultaneously connected clients
• Clients are mostly idle (heartbeat/ping messages only)
• Small text messages (less than 1 KB)
• No large binary payloads
• One Node.js process
• Linux server
• TLS termination handled separately (or already accounted for)
• Figures represent application memory only, not total system memory

(These are realistic ballpark figures (actual values vary by implementation).

Resource Usage for Raw WebSocket (ws node.js package) implementation

The ws library is a lightweight implementation that exposes the WebSocket protocol with minimal abstraction.

Typical per-connection memory consumption:

Memory:

1000 × 30KB = 30,0000KB
≈ 29 MB

Resource Usage for Socket.IO package implementation

Socket.IO builds on top of WebSockets (via Engine.IO) and adds higher-level functionality such as

• Engine.IO layer
• Rooms
• Acknowledgements
• Automatic reconnection
• Packet encoding
• Additional metadata

Typical per-connection memory consumption:

Memory:

1000 × 600KB = 60,000KB
≈ 58 MB

Visual comparison

This is approximately linear growth:

Memory ∝ Number of Connections

But memory isn’t the real problem

Memory is only one dimension. Suppose an EC2 instance has:

• 2 GB RAM
• 1 vCPU

If your application itself consumes:

• 500 MB for Node.js application code
• 200 MB for caches
• 300 MB OS overhead

Available for WebSockets:

2GB − 1GB = 1GB

Using raw WebSockets:

1000MB ÷ 30KB ≈ 34,000 theoretical connections

Using Socket.IO:

1000MB ÷ 60KB ≈ 17,000 theoretical connections

In reality, CPU, network bandwidth, and event-loop latency become bottlenecks much earlier (often around 5k — 20k connections per Node process depending on message frequency).

The scalability challenge

The issue isn’t serving 1,000 users. The issue is serving 100,000 users.

If one server holds 10,000 persistent connections:

100,000 ÷ 10,000 = 10 servers

Now you must manage:

• Load balancing
• Sticky sessions
• Cross-instance pub/sub
• Connection failover
• Redis adapters
• Auto scaling

At that point, WebSockets become a distributed systems problem rather than a networking problem.

Scaling WebSockets with Node.js, Nginx, and Redis Pub/Sub

To move beyond a single-server bottleneck, WebSocket connections are typically distributed across multiple Node.js instances running in a cluster behind a reverse proxy like Nginx.

Figure: High-Level Architecture supporting scalable WebSocket with Redis Backplane.

At a high level, the architecture looks like this:

• Nginx handles incoming WebSocket upgrades and load balances connections across Node.js upstream servers.
• Each Node.js instance manages only a subset of active connections.
• A Redis Pub/Sub layer synchronizes messages across all instances.

This solves a key limitation: WebSocket connections are stateful, so without coordination, one server cannot notify clients connected to another server.

We can model the scaling behavior as:

N = n_1 + n_2 + n_3 + … + n_k

Where:

• N = total concurrent connections
• n_k = connections handled by each Node.js instance

If each instance supports ~10,000 connections:

100,000 users ≈ 10 Node instances

However, connection distribution alone is not enough. Message propagation requires cross-instance communication:

Client A → Node A → Redis Pub/Sub → Node B, Node C → Clients

So Redis acts as the “event backbone”, ensuring messages reach all subscribed instances regardless of which server holds the original socket.

In practice, the system scales in two dimensions:

• Vertical (per node): limited by memory per connection
• Horizontal (cluster): limited by coordination overhead (Redis + network hops)

This architecture allows WebSockets to scale from thousands to hundreds of thousands of concurrent connections, but introduces a new tradeoff: distributed complexity replaces single-server simplicity.

The operational cost of self-managed WebSocket scaling

Once WebSockets are distributed across multiple Node.js instances with a load balancer and a Redis Pub/Sub backbone, the system becomes functionally scalable but operationally heavy.

At this stage, scaling is no longer just about handling connections; it requires continuous management of infrastructure behavior in real time.

Engineers must now monitor and tune:

• Connection distribution across nodes (to avoid uneven load)
• Memory per instance (to prevent socket exhaustion)
• Redis Pub/Sub throughput and latency
• Message fan-out patterns across services
• Network hops introduced by inter-node communication
• Failure recovery and reconnection storms during outages

In essence, the system shifts from a simple connection model to a continuously evolving distributed system where every component becomes a potential bottleneck.

The AWS approach: Offloading the complexity

Managed AWS WebSocket solutions, such as API Gateway WebSockets, fundamentally change this model by removing the need to manage persistent connection infrastructure directly.

Instead of maintaining Node.js servers for connection handling, AWS takes responsibility for:

• Maintaining persistent WebSocket connections at scale
• Handling connection lifecycle (establish, reconnect, disconnect)
• Scaling underlying infrastructure automatically
• Routing messages to connected clients via managed APIs
• Integrating with backend services through AWS Lambda, SQS, or EventBridge

In this model, engineers no longer manage sockets as infrastructure primitives. Instead, they work with higher-level events:

Client Event →AWS API Gateway → Compute Layer (Lambda / Service) → Event Routing Back to Clients

What this removes from engineering ownership

Moving to AWS-managed WebSockets shifts responsibility away from engineers in several critical areas:

• No server provisioning or horizontal scaling for WebSocket nodes
• No load balancer tuning or sticky session management
• No Redis Pub/Sub coordination layer
• No manual connection tracking or socket state distribution
• No handling of connection spikes or reconnection storms
• No direct memory management per connection

Simple 1-to-1 Chat using AWS API Gateway WebSockets

This is a minimal architecture for building a real-time 1-to-1 chat system using AWS-managed WebSockets. The goal is to avoid managing servers, connection state, or scaling logic manually.

Create a WebSocket API in API Gateway

In AWS API Gateway, create a WebSocket API. You will be asked to define a Route Selection Expression, typically: request.body.action

This means every incoming message is routed based on the action field in the JSON payload.

Figure: Create a WebSocket API (websocket-1–1-chat) in API Gateway with a route selection expression

Figure: Define custom route key (sendMessage) for integration

Once created, API Gateway provides:

• WebSocket Endpoint

wss://{api-id}.execute-api.{region}.amazonaws.com/{stage}

• Connection URL (HTTP POST)

https://h7axxxxxxx.execute-api.us-west-2.amazonaws.com/production/@connections

Figure: WebSocket Endpoint and @connection URL endpoint of websocket-1–1-chat API

What they mean:

• WebSocket Endpoint → used by clients to connect and stay connected in real time
• Management / POST endpoint → used by backend (Lambda) to send messages to a specific connected client

Understanding Route Keys

API Gateway uses route keys to decide how messages are handled:

Built-in routes:

• $connect
  Triggered when a client first connects
  → Used for authentication and initialization
• $disconnect
  Triggered when a client disconnects
  → Used for cleanup
• $default
  Triggered when no matching route is found
  → Catch-all for messages

Custom route example

If a client sends:

{ "action": "sendMessage", "toUserId": "user_2", "message": "Hello"}

Then API Gateway routes it to:

sendMessage

You must explicitly create this route in API Gateway.

Wire Routes to Lambda Functions

Each route is connected to a Lambda:

This is done in API Gateway → Routes → Attach Integration.

Figure: Attach integrations for $connect, $disconnect, $default and sendMessage route keys

Store Connection IDs in DynamoDB

Every connected client gets a unique: connectionId

During $connect, store this in DynamoDB:

Table: Connections

userIDconnectionsuser_1abc123user_2xyz789

This allows you to map:

userId → connectionId

Without this, you cannot target specific users.

Sending a Message (1-to-1 Flow)

Figure: Flow diagram for API Gateway and Lambda interactions based on route keys

When user_1 sends a message to user_2:

Step 1: User 1 sends a message

{ "action": "sendMessage", "toUserId": "user_2", "message": "Hey!"}

Step 2: Lambda receives the request

The messageHandler Lambda:

• Reads sender (user_1)
• Reads recipient (user_2)
• Looks up user_2 in DynamoDB
• Retrieves connectionId

Step 3: Forward message using API Gateway Management API

Lambda calls:

POST /@connections/{connectionId}

Payload:

{ "from": "user_1", "message": "Hey!"}

This sends the message directly to user_2’s active WebSocket connection.

$Disconnect Cleanup

When a user disconnects:

• $disconnect route triggers
• Remove their connectionId from DynamoDB

This prevents stale connections and ensures accurate routing.

Final architecture flow

User 1 → WebSocket API Gateway → Lambda (sendMessage) → DynamoDB lookup (user_2 connectionId) → API Gateway Management API → User 2 receives message

Figure: Rough sequence diagram representing user_1 sending a message to user_2 involving API Gateway and Lambda Integrations

AWS advantage with a tradeoff

With AWS API Gateway WebSockets, there are:

• No servers to manage
• No Redis Pub/Sub required
• No connection scaling logic
• No load balancers or sticky sessions

You only define:

• Routes ($connect, $disconnect, custom actions)
• Lambda logic
• Connection storage (DynamoDB)

AWS handles persistent connections, scaling, and delivery infrastructure automatically, allowing you to focus purely on application logic rather than socket management.

The tradeoff

While self-managed architectures offer full control and lower per-request cost at scale, they require deep operational expertise in distributed systems.

AWS-managed WebSockets reduce this burden by abstracting connection management entirely, but in exchange, they introduce constraints in protocol control, cost predictability at extreme scale, and platform dependency.

In practice, the decision is not about capability, but about where you want complexity to live:

• Self-managed WebSockets: maximum control, maximum operational responsibility
• AWS-managed WebSockets: reduced control, significantly reduced infrastructure burden

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI