Mastering Event Sourcing and CQRS with Apache Kafka and .NET Core
In modern distributed systems, maintaining consistency, auditability, and scalable read/write separation is a constant challenge. Event Sourcing combined with CQRS (Command Query Responsibility Segregation) offers a powerful architectural pattern to address these challenges — and using **Apache Kafka** as the event backbone plus **.NET Core** on the implementation side gives you a robust, scalable, and performant solution. In this article, we’ll walk from fundamentals to advanced techniques, with code, trade-offs, and real-world patterns.
🚀 Why Event Sourcing + CQRS?
Let’s start with context. Traditional CRUD systems store the current state of entities (e.g. Customer, Order), often losing history or requiring a separate audit trail. Event Sourcing instead captures every change as an immutable event. The current state is then **derived** by replaying these events.
In a CQRS architecture, you split the responsibilities:
- Commands / Write side: Accept user intention (e.g. “PlaceOrder”), validate and persist as events.
- Queries / Read side: Provide optimized views / projections to serve queries.
This separation enables independent scaling, optimized data models for reads, and full auditability of all changes. Event Sourcing ensures you never lose historical data and allows you to rebuild state at any point in time. However, with this power comes complexity: you must manage eventual consistency, concurrency, event versioning, snapshots, and messaging reliability.
🔗 Why Apache Kafka fits as the Event Backbone
Apache Kafka is essentially a distributed, durable, ordered commit log. It offers retention, partitioning, fault tolerance, and high throughput, making it a compelling option to implement an event store in many real-world situations.
Using Kafka as the event store (or part of it) gives you:
- Immutable, time-ordered events with retention and replay capability.
- Easy subscription by multiple consumers (for building read models, analytics, etc.).
- Scalable partitioning so events can be processed in parallel (per key or aggregate).
- Integration with stream processing (Kafka Streams, ksqlDB) for materialized views or transformations.
That said, Kafka isn't a perfect drop-in replacement for a full event store. Issues around retention (how long events remain), transactional guarantees (across aggregates), snapshotting, and queryability must be handled carefully. Many systems use Kafka in tandem with a more expressive store (like EventStoreDB, relational DB, or specialized event stores) to address these trade-offs.
🏗 Architecture Overview: Components & Flow
Here’s a high-level architecture flow for Event Sourcing + CQRS using Kafka and .NET Core:
- A client issues a command (e.g. “CreateOrder”).
- The command handler loads the current aggregate state (by replaying events, possibly using snapshots).
- The command logic emits one or more domain events (e.g. OrderCreated, ItemAdded).
- Events are appended to a Kafka topic (e.g. `order-events`).
- One or more **projection processors** or **event consumers** subscribe to that topic, transforming events into one or more **read models** (e.g. SQL, document DB, Elasticsearch).
- The query side of the system serves API requests by querying the read model (which is kept in sync). Because of eventual consistency, there may be slight lag between writes and reads.
- Optionally, you can replay the log to rebuild read models, or rebuild an aggregate from older events (e.g. for debugging or migrations).
A simplified diagram:
Client → Command API → Event Broker (Kafka) → Projection / Consumers → Read Model → Query API → Client
💻 Code Example: Basic .NET Core Command Handler + Kafka
// Simplified .NET Core command handler producing a Kafka event
public class CreateOrderCommand
{
public Guid OrderId { get; set; }
public string CustomerId { get; set; }
public List Lines { get; set; }
}
public class OrderCreatedEvent
{
public Guid OrderId { get; set; }
public string CustomerId { get; set; }
public List Lines { get; set; }
public DateTime OccurredAt { get; set; }
}
public class OrderCommandHandler
{
private readonly IConsumerFactory _consumerFactory;
private readonly IProducer _producer;
public OrderCommandHandler(IProducer producer)
{
_producer = producer;
}
public async Task Handle(CreateOrderCommand cmd)
{
// Basic validation omitted
var @event = new OrderCreatedEvent {
OrderId = cmd.OrderId,
CustomerId = cmd.CustomerId,
Lines = cmd.Lines,
OccurredAt = DateTime.UtcNow
};
// Write event to Kafka topic
var message = new Message
{
Key = cmd.OrderId.ToString(),
Value = @event
};
var result = await _producer.ProduceAsync("order-events", message);
// Optional: You may want to wait for acknowledgment, handle errors, etc.
}
}
🧠 Handling Projections: Event Consumers & Read Models
Projection handlers subscribe to events and build read-optimized views. Below is a sketch of a projection consumer:
using Confluent.Kafka;
public class OrderProjectionConsumer
{
private readonly IConsumer _consumer;
private readonly MyReadDbContext _db;
public void Start()
{
_consumer.Subscribe("order-events");
while (true)
{
var cr = _consumer.Consume();
var evt = cr.Message.Value;
// Upsert into read model table
var existing = _db.Orders.Find(evt.OrderId);
if (existing == null)
{
_db.Orders.Add(new OrderRead
{
OrderId = evt.OrderId,
CustomerId = evt.CustomerId,
CreatedAt = evt.OccurredAt
});
}
_db.SaveChanges();
}
}
}
Because events arrive asynchronously and possibly out of order (depending on partitions), the projection logic must be idempotent, resilient to duplicates, and tolerant to reordering or late arrivals.
🔍 Advanced Concepts & Best Practices
Once the basic flow is working, real systems demand more sophistication. Here are some key patterns and trade-offs:
- Snapshotting: To avoid replaying thousands of events to reconstruct state, periodically snapshot the aggregate state and only replay events after the snapshot point.
- Event Versioning & Schema Evolution: Events evolve over time. Use version fields, backward/forward compatibility strategies, or transformation pipelines.
- Concurrency / Optimistic Locking: When handling commands concurrently, you may detect conflicts (e.g. two commands against same aggregate). You can handle by version checks or retries (compare expected version).
- Idempotency & Deduplication: Ensure consumers/projects are idempotent (ignore duplicate events) or include dedup logic (e.g. record last processed offset).
- Exactly Once / Transaction Semantics: Kafka + external database writes need care. You may use Kafka transactional APIs or outbox patterns to coordinate atomic writes.
- Replaying & Migration: You should be able to replay your event log to rebuild read models or migrate event formats.
- Handling Retention / Archival: Kafka topics may drop older data by retention policies. If you rely on indefinite history, consider external archival or a hybrid store.
- Consistency Guarantees: The read side is eventually consistent; you may need to expose versioning, stale reads, or retry logic upstream.
- Monitoring & Alerts: Track consumer lags, dead letter handling, and event backlog.
📦 Real-World Examples & Libraries
Several open source projects and community patterns help accelerate your implementation:
- ASP.NET Core Clean Architecture + CQRS + Event Sourcing sample — demonstrates DDD, SOLID, and event sourcing in practice.
- EventSourcing.NetCore samples — useful examples and patterns for .NET event sourcing.
- Many blog series such as “Hands-on DDD featuring Event Storming, CQRS, Event Sourcing in .NET” show how to build query APIs using Kafka consumers.
⚡ Key Takeaways
- Event Sourcing + CQRS gives you full history, auditability, separation of responsibilities, and scalable read/write paths.
- Apache Kafka is a strong candidate for the event store backbone, but must be used with care (retention, archival, transaction semantics).
- Projections asynchronously transform events into read models — they must be idempotent, fault-tolerant, and eventually consistent.
- Advanced features like snapshotting, versioning, concurrency control, and replay capabilities are essential for production usage.
- Use open source reference implementations and patterns to avoid reinventing boilerplate and edge-case logic.
❓ Frequently Asked Questions
- What is the difference between Event Sourcing and simple Event-Driven Architecture?
- Event-Driven Architecture emits events to decouple components, but state is still stored via CRUD. Event Sourcing uses the events *as the primary source of truth* and rebuilds state by replaying them.
- Can Kafka really replace a dedicated event store like EventStoreDB?
- Kafka can serve many needs of an event store (durable log, partitioning, replay). But it lacks certain features like specialized projections, complex querying, snapshot management, and ACID operations for aggregates. Many systems use Kafka plus an auxiliary store.
- How do I handle versioning when the event schema changes?
- Use version fields or schema evolution techniques (e.g. backward-compatible changes, transformation layers). Maintain compatibility by writing adapters or migration logic when reading old versions.
- Is eventual consistency a problem?
- Some clients may read stale data briefly. Mitigate by using versioning, retries, or exposing version metadata to clients. Often, the benefits outweigh the consistency delay.
- How do I replay the event log to rebuild read models?
- You can reset your read-model database, then consume events from Kafka from the earliest offset or from snapshots forward, reprocessing all projection logic to rebuild views.
💬 Found this article helpful? Please leave a comment below or share it with your network to help others learn!
About LK-TECH Academy — Practical tutorials & explainers on software engineering, AI, and infrastructure. Follow for concise, hands-on guides.
Posts
