event sourcing - LMAX Architecture - Growth of data -

consider following scenario lmax architecture description martin fowler:

i'll use simple non-lmax example illustrate. imagine making order jelly beans credit card. <...>

in lmax architecture, split operation two. first operation capture order information , finish outputting event (credit card validation requested) credit card company. business logic processor carry on processing events other customers until received credit-card-validated event in input event stream. on processing event carry out confirmation tasks order.

so order held in-memory until result of payment processing received.

now let assume instead of credit card processing step, have step takes more time, example: need perform inventory check, has physically verify have particular flavor of jelly bean has been ordered. might take hour.

if case, not lead growth of data held in-memory because potentially lot of orders awaiting inventory status updated event?

possibly in such scenario, need remove order memory , include part of output event, external system (inventory) responsible generating input event includes order detail.

the problem see approach, cannot include inventory part of business logic processor.

thoughts how address this?

working orders in financial exchange can stay around days, or months, part of working set. example, waiting futures contract expire. customer accounts similar. "working set" mean deals/orders/sales etc. active. once deal complete part of historical data.

memory systems large, i.e. hundreds of gbs in single server, working set of business fits memory. available memory size increasing @ rate faster large business growing.

the scenario describe not issue. can become issue when need hold historical data traditional database or file-based system more suitable.

a simple exercise calculate memory required active entities, or working set, , compare available in modern servers. possible keep 100s of millions of active entities around in memory.