Corentin GS's Blog

Beyond PGN: Designing an Ultra-Efficient Chess Storage Format

Wed, 24 Jun 2026 00:00:00 GMT

Every chess database begins with a simple question:

How do we store a game of chess?

For decades, the default answer has been PGN: Portable Game Notation. It is human-readable, easy to share, easy to edit, and universally supported. A PGN game feels natural because it looks close to how chess players already talk about moves:

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4

For humans, this is excellent.

For machines, it is surprisingly wasteful.

A chess engine, a database, or a high-performance analysis tool does not need the move to be written as "Nxd4" or "O-O". It does not need spaces, move numbers, dots, or SAN disambiguation. It only needs to know one thing:

Which move was played?

When I first started thinking about chess storage, I assumed PGN was already compact enough. Then I did the math, and PGN started looking less like an efficient storage format and more like a convenient text format that happens to be compressible.

If we want to build a modern chess database capable of storing millions of games, loading them instantly, syncing them efficiently, and navigating them without constantly reparsing text, we can go much further.

The First Realization: A Chessboard Is Only 64 Squares

A chessboard looks large to a beginner.

But to a computer, it is tiny.

There are only 64 squares. That means any square can be represented with just 6 bits, because:

2^6 = 64

So instead of storing a move as text like:

Nf3

or:

exd5

we can store it as two square coordinates:

from_square -> to_square

For example:

e2 -> e4

One square needs 6 bits.

Two squares need:

6 bits + 6 bits = 12 bits

That means most normal chess moves can be represented in only 12 bits.

This is the simplest binary approach:

[from: 6 bits][to: 6 bits]

It is compact, direct, and fast to decode.

The reader does not need to parse SAN. It does not need to understand whether "Nbd2" means the knight from b1 or f3. It does not need to interpret captures, checks, checkmates, or castling notation. It simply reads two square IDs and applies the move.

The chess rules are still needed to update the board correctly, but the move identity itself is no longer hidden inside text.

Castling and en passant fit naturally into this scheme. For castling, we encode the king's from-square to its destination: e1→g1 for kingside, e1→c1 for queenside. The rook move is implied by the rules. For en passant, we encode it as a normal pawn capture using the from-square and the destination square. The capture of the opponent's pawn is a board-state consequence.

The Promotion Problem

Chess always has exceptions.

Most moves can be represented with a starting square and a destination square. But promotions need one extra piece of information.

If a pawn reaches the final rank, it can promote to one of four pieces:

Queen
Rook
Bishop
Knight

Four possibilities require 2 bits:

00 = queen
01 = rook
10 = bishop
11 = knight

So a promotion move can be stored as:

[from: 6 bits][to: 6 bits][promotion: 2 bits]

That gives us:

12 bits + 2 bits = 14 bits

So with a simple coordinate-based binary encoding, we can represent almost every chess move in 12 bits, and promotion moves in 14 bits.

Compared to PGN text, that is already a major improvement.

Why This Is Already Better Than PGN

Let's take a common PGN move:

Nxd4

That is four characters. In a normal text encoding, each character usually costs 8 bits.

So this move alone costs roughly:

4 characters × 8 bits = 32 bits

And that does not include spaces, move numbers, comments, variations, line breaks, or metadata.

PGN is also more expensive to interpret than its size alone suggests. A PGN parser must deal with disambiguation like "Nbd2", "R1e2", or "exd5". To understand those moves, the parser must reconstruct the board state, find legal pieces that can reach the target square, and determine the correct one. PGN is semantic text that requires chess-aware parsing.

A coordinate move costs about:

12 bits

And it needs no chess-aware parsing to identify the move. The system reads two square IDs and applies the move directly.

A rough comparison looks like this:

PGN text move:       ~24–40 bits or more
Binary coordinates: ~12–14 bits

That is the first big lesson:

PGN was designed for readability and portability, with compactness as a secondary concern.

The Second Realization: Maybe We Don't Need Coordinates at All

We can do better.

At any given chess position, the number of legal moves is limited.

A player does not have 4,096 possible moves. They usually have something like 20, 30, 40, or maybe 50 legal moves. The theoretical maximum is 218 legal moves (requiring 8 bits), but such positions are vanishingly rare in practice.

So instead of storing:

from_square -> to_square

we could do something more clever.

Imagine that both the writer and the reader generate the exact same list of legal moves for the current position:

0: e2e4
1: d2d4
2: g1f3
3: c2c4
...

Then we do not need to store the move itself.

We only need to store its index.

If the move played was the third move in the list, we store:

That is the legal-move-index approach.

Instead of asking:

Which square did the piece move from, and where did it go?

we ask:

Out of all legal moves in this position, which one was chosen?

This can be much smaller.

If a position has fewer than 64 legal moves, the index fits in 6 bits.

If it has fewer than 128 legal moves, it fits in 7 bits.

Most chess positions fit comfortably in that range.

So now our rough comparison becomes:

PGN text:           ~24–40 bits per move
Coordinates:        ~12–14 bits per move
Legal move index:   ~6–7 bits per move

That is a dramatic reduction.

The Beautiful Edge Case: Forced Moves Cost Almost Nothing

This approach has an elegant consequence: forced moves cost nothing.

Sometimes a player has only one legal move.

For example, in a forced checkmate sequence or a position where every move except one is illegal, there may be exactly one valid choice.

If there is only one legal move, we do not need to store anything.

The decoder generates the legal move list, sees that only one move exists, and applies it automatically.

The move costs:

0 bits

That feels almost magical.

But it is not magic. It is simply using the rules of chess as shared context between the encoder and decoder.

When the rules determine the answer, the file does not need to repeat it.

This is the core principle behind many efficient encodings:

Do not store information that can be reconstructed deterministically.

The Catch: Compression Is Not Free

At this point, legal-move indexing sounds like the obvious winner.

Why store 12 bits when we can store 6?

Why use coordinates at all?

The answer is performance.

To decode a coordinate-based move, the reader does something simple:

Read from-square
Read to-square
Apply move

This is extremely fast.

To decode a legal-move index, the reader must do more work:

Generate all legal moves
Sort/order them deterministically
Read the index
Select the matching move
Apply move

That means every move requires legal move generation.

For a single game, this is fine.

For millions of games, it becomes expensive.

Compression is only half the problem. A chess database is also a performance problem.

If your format is beautifully small but slow to scan, slow to open, slow to index, or difficult to randomly access, it may be worse in practice than a larger format.

This gives us the fundamental trade-off:

Smaller files usually require more computation.
Faster decoding usually requires storing more explicit information.

There is no universal best answer. There is only the best answer for a specific system.

Coordinates vs Legal Move Indexes

The coordinate approach is simple, direct, and practical.

It gives us:

[from square][to square][optional promotion]

Its strengths are obvious:

Very fast to decode.
Easy to implement.
Easy to validate.
Good for random access.
Good for database indexing.
Does not require generating all legal moves just to identify the move.

Its weakness is that it is not maximally compact.

The legal-move-index approach is more compressed.

It gives us:

[index of move in legal move list]

Its strengths are:

Extremely compact.
Can exploit forced moves.
Can approach the theoretical information needed to describe a chess game.

Its weaknesses are serious:

Requires legal move generation at every ply.
Requires a perfectly deterministic move ordering.
Harder to implement safely.
More expensive to decode at massive scale.
More fragile if the rules, variants, or encoding assumptions change.

That narrows the question to: what does the system need to optimize for?

A serious chess database needs to:

Import millions of games.
Open individual games instantly.
Jump to arbitrary positions.
Build position indexes.
Search references.
Display games quickly.
Sync data efficiently.
Support annotations, comments, variations, and repertoires.
Export back to PGN when needed.

For that kind of system, raw compactness is not enough. A format must also be operationally useful.

A coordinate-based binary format is often the better engineering compromise. It is still far smaller than PGN, but it remains simple and fast.

For cold archival storage where decoding speed matters less, legal-move indexing is more attractive. The file would be tiny, and we could afford extra computation because the data would rarely be read.

The Practical Design Choice

For a modern chess database, I would start with coordinate-based binary moves.

Something like:

12 bits for normal moves
14 bits for promotions

This gives an excellent balance:

Much smaller than PGN
Much faster to parse than PGN
Simple enough to implement safely
Efficient for massive databases
Friendly to random access and indexing

Legal-move indexing can still be useful later, especially for a compressed distribution format or cold storage format.

But for the primary editable database representation, coordinates are a better foundation.

This leads to a layered architecture:

Internal database format: coordinate-based binary moves
Optional archive format: legal-move-index compression
Export format: deterministic PGN

That way, each format does what it is best at.

PGN remains the universal language for humans and existing tools.

The binary coordinate format becomes the fast internal representation.

The legal-move-index format becomes an optional compression layer for distribution or archival storage.

What About the Initial Position?

So far, we assumed that every game starts from the normal chess starting position.

But real databases also contain games that begin from custom positions:

Chess960 games.
Training positions.
Composed studies.
Partial game fragments.
Engine lines.
Positions imported from FEN.

For those cases, the format needs to store the initial board state.

The usual text format for this is FEN:

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1

FEN is useful and readable, but again, it is text.

A binary format could store a compact "Bit-FEN" instead.

At minimum, an initial position needs:

Piece placement.
Side to move.
Castling rights.
En passant square, if any.
Halfmove clock.
Fullmove number.

The exact encoding can vary, but the principle is the same:

Store chess state as structured binary data, not as text that must be reparsed.

For normal games, we can omit the initial board entirely and assume the standard starting position.

For custom games, we include a compact binary initial-state block.

That gives us efficiency without losing generality.

Conclusion: PGN Is the Beginning, Not the End

PGN gave chess software a common language.

But modern chess software can go beyond it.

A binary chess format can represent moves in 12 to 14 bits using coordinates. With legal-move indexing, it can sometimes approach 6 to 7 bits per move, or even 0 bits for forced moves.

But the smallest format is not always the best format.

For an interactive chess database, the best design is likely a pragmatic one:

Use binary coordinates for speed.
Use legal-move indexing where compression matters.
Use PGN for import and export.
Use a compact binary position format for custom starting states.

That gives us a layered system where each format plays to its strength:

Human compatibility.
Machine efficiency.
Fast random access.
Compact storage.
Deterministic export.
Room for future compression.

PGN remains useful.

But it does not have to be where the story ends.

Sometimes the real breakthrough begins when we stop asking:

How do we store the notation?

And start asking:

What is the smallest useful truth the machine actually needs?

Beyond Minimalism: A Japanese-Inspired UX for the Web

Fri, 19 Jun 2026 00:00:00 GMT

After looking at why Japanese websites often feel overloaded, and what Western UX can learn from them, the obvious question is simple:

Should Japanese websites become minimalist?

I do not think so.

Western minimalism solves real problems: focus, hierarchy, less noise, faster pages. All good. But it can also become an aesthetic reflex: remove the text, hide the details, simplify the comparison, make the action bigger, trust the brand mood to carry the rest.

That works until the user needs context. As the first article in this series argued, density can be information, navigation, promotion, comparison, reassurance, or visible service — and the second article showed what Western UX can borrow from that.

Japanese web design does not need Western minimalism. And Western UX does not need to absorb Japanese density wholesale. The more interesting future sits between them: interfaces that preserve context, reassurance, comparison, and visible service, but organize them with hierarchy, spacing, expandable details, and accessibility.

Clutter is a problem of structure, not of quantity.

Accumulated Density vs Designed Density

The first mistake is treating density as one thing.

Some density carries real value. Delivery conditions, cancellation terms, official notices, price comparisons, support paths, campaign deadlines, and compatibility details can all reduce uncertainty. They are part of the service.

Other density is only organizational sediment. A department adds a banner, a campaign demands a module, a legal warning becomes permanent, and an internal stakeholder protects a link. Over time, the page records company politics instead of user needs.

That is accumulated density.

Designed density starts from a different question: if the user really needs this information, where should it live, how should it be grouped, and when should it appear?

Accumulated density is what happens when a page records company politics. Designed density is what happens when a page records user needs. Here is the difference:

Accumulated density	Designed density
Everything competes	Clear priority
Banners everywhere	Modules by user intent
Links repeated randomly	Redundancy used deliberately
Trust cues scattered	Trust cues near decision points
Dense because nobody removes	Dense because users need context

The point is to make information legible enough that density can do its job — not to make dense pages look sparse.

Ma Is the Interval Between Things

Ma (間) in design is the meaningful interval between elements that lets each one make sense in relation to the others. It is relational spacing: the pause that gives a page room to think.

In Western design conversations, ma often gets flattened into a nicer word for whitespace. That misses the point.

Ma is relational space. It is the interval that lets one thing make sense beside another thing.

In interface design, ma can be the space between decision groups. It can be the pause before a high-commitment action. It can separate a promotion from a functional tool. It can give the user a moment to understand why a trust cue is near a button, why a comparison table follows a promise, or why support information appears before checkout.

Intervals make density sustainable. Enough information makes simplicity trustworthy. A page needs both — and this is exactly the case for trustful minimalism.

Many pages fail in opposite ways. One makes every element shout. Another leaves the user hunting for basic facts. Ma offers another path: enough separation to think, without pretending that users only need one sentence and a button.

Bento as an Interface Model

Ma gives density rhythm. Bento gives it structure.

Bento structure as an interface model: many modules, each with a clear job, held in place by visible rhythm and ma.

A bento box gives each item a place. Rice, fish, vegetables, pickles, and sauce can sit together because the container gives the variety order. Nothing needs to become the same thing to belong.

A Japanese-inspired bento interface might include these modules:

A primary action
A short value proposition
A trust strip
A comparison block
A campaign block
A FAQ preview
A support path
Compatibility or expert details
A plain-language entry point for first-time users

The page is a set of compact zones, each with a job: decide here, compare here, verify here, learn here, ask for help here.

This pattern fits complex products especially well. Insurance, travel, and government services rarely fit into a pure landing-page funnel. Users need to compare, hesitate, check conditions, and return later, and a bento structure respects that without turning the page into a wall.

Shogi, Go, and Relational Meaning

Board games offer a useful metaphor, as long as it stays a metaphor.

Chess is easy to read as focal-object design: distinct shapes, explicit identities, direct tactical roles. As pieces disappear, the board often becomes simpler.

Go works differently. The stones are visually identical, and their meaning comes from relationship: territory, influence, proximity, pressure, and future possibility. Shogi adds another kind of density because captured pieces can return to the board. The game state can recompose instead of only simplifying.

That proves nothing about interfaces, but it is a useful way to think.

A Western landing page isolates one action. A Japanese service page preserves a field of possible actions. A good hybrid keeps the field visible and makes the next move legible. The lesson from Shogi is relational meaning: the surrounding field changes the value of each move.

For UX, the button is not the whole story. A "Start" button beside a vague promise feels different from a "Start" button beside pricing, cancellation terms, support availability, setup time, and proof that the service is official.

The object is the same. The field changes the decision.

Layered Information UX

The practical pattern I would take from all this is layered information UX.

Layered information gives immediate reassurance, visible structure, and deeper detail on demand. It decides what the user needs now, what they may need next, and what should remain available without dominating the page.

That can look like:

Short trust cues near the main call to action
Expandable delivery, pricing, return, and compatibility details
Beginner and expert paths
Comparison cards before long technical tables
Sticky summaries for dense forms or product pages
Campaign modules visually separate from decision-critical information
Semantic HTML instead of image-based text, so density is accessible to screen readers, search, and translation
Keyboard- and zoom-friendly dense layouts, designed from the start rather than patched on

This is where Japanese density and Western minimalism can stop arguing.

Minimalism brings priority and focus. Japanese information-rich design adds what many modern interfaces underplay: context, reassurance, room to explore, and visible service. Layered UX combines them by asking the more precise question: what information should be visible at this moment of trust?

Not all information deserves equal weight. Some information deserves to be seen before the user commits.

What Trustful Minimalism Looks Like in Practice

Article 2 introduced the term trustful minimalism. It keeps the visual discipline of modern UX but restores the information users need to feel confident.

In practice, it looks like a few small choices repeated consistently:

A trust strip beside the main action, not buried in the footer
Comparison options visible before commitment
Reassurance close to the decision: price, cancellation, delivery, support, security, return policy
Expandable details for users who want to dig deeper
Separation between promotional modules and decision-critical modules
A bento structure for complex products rather than a single hero block

It also looks like restraint: remove the elements that do not help a user decide, trust, or compare. Keep the ones that do.

What Western Designers Can Borrow

Western designers should not copy Japanese visual density as a style. Copying the surface gives you banners, badges, mascots, and crowded grids without the cultural or service logic that made them useful. Article 2 walks through what Western UX can borrow in detail.

Worth borrowing:

Bento structure for complex products
Trust cues near calls to action
Visible comparison instead of pure persuasion
Exploration paths, not only funnels
Ma as rhythm, not decoration
Progressive disclosure that does not hide legitimacy
Character or mascot guidance only when it helps the task

Skip these:

Random banner competition
Image-based text
Every stakeholder on the homepage
Duplicate links without logic
Density without hierarchy
Decorative reassurance that does not answer a real user concern

The first article reframes this question around what the density is doing. The better question is more specific: what does the user need to understand, trust, compare, and choose?

If the information does not help with one of those jobs, remove it. If it does help, design a place for it.

Enough Context, Enough Room

In the first article, density was the object of suspicion: maybe those crowded pages were not only clutter, but context, navigation, promotion, comparison, reassurance, or visible service.

The second article turned that suspicion back toward Western UX. Simplicity is useful, but it can also erase the information users need to feel confident.

This final article is where I land: the future is structured, contextual, and trust-aware.

The future of UX belongs to interfaces that understand why information is there. Japanese web design reminds us that users need context, confidence, comparison, and visible service readiness — not just a button. Western UX reminds us that attention is limited and hierarchy matters.

The interface that earns trust gives people enough context to decide and enough room to change their mind.

References and Further Reading

Stanford Encyclopedia of Philosophy, Japanese Aesthetics (Ma, Wabi-sabi, related concepts)
Tuch, A. N., et al. (2009), Visual complexity of websites: Effects on users' experience, physiology, mood, and behavior, Human-Computer Interaction
Nordhoff, August, Oliveira, and Reinecke, A Case for Design Localization (CHI 2018)
Baughan et al., studies on visual complexity, website search, and cross-cultural attention
Nisbett and Masuda, Culture and Point of View and Attending Holistically Versus Analytically
W3C, Requirements for Japanese Text Layout
Japan House Los Angeles, A Perspective on the Japanese Concept of Ma
For the series: Why Japanese Websites Look Overloaded: Density, Tokyo, and Trust and What Western UX Can Learn from Japanese Web Design

Context and Cancellation in Go: A Practical Guide

Wed, 17 Jun 2026 00:00:00 GMT

Every goroutine you spawn is a promise that something will eventually stop. If you don't control when that happens, you leak intention: your program keeps doing work nobody asked for, long after the caller moved on.

Go's context package is the standard answer. But like most standard answers, it's easy to use poorly and hard to use well. This builds on the channel patterns from the worker pool article and the pipeline article: once goroutines compose, cancellation becomes part of the design, not a cleanup detail.

The Problem: Orphaned Work

Imagine a handler that queries three databases in parallel:

func handleRequest(w http.ResponseWriter, r *http.Request) {
    var wg sync.WaitGroup
    wg.Add(3)

    go func() { defer wg.Done(); queryDB1() }()
    go func() { defer wg.Done(); queryDB2() }()
    go func() { defer wg.Done(); queryDB3() }()

    wg.Wait()
    // ... respond
}

What happens when the client disconnects after 50ms? The goroutines keep running. The databases keep working. Resources burn for a request that will never be answered.

This is the orphaned work problem. And it's everywhere in concurrent code.

What Context Gives You

context.Context is a request-scoped value that carries:

Cancellation signals — tell goroutines to stop
Deadlines — automatic cancellation after a timeout
Key-value pairs — request-scoped metadata (trace IDs, user info)

The key insight: context flows down. You create it at the top of a request, pass it to every function and goroutine, and everything respects the same lifecycle.

The Three Flavors

1. `context.Background()` — The Root

Use this at the top level: main, test setup, initialization. It never cancels. Everything else derives from it.

ctx := context.Background()

2. `context.WithCancel()` — Manual Control

When you need to say "stop now":

ctx, cancel := context.WithCancel(context.Background())
defer cancel() // always call this

// pass ctx to goroutines
// call cancel() when you're done

The cancel function is idempotent. Call it once, twice, a hundred times—same result. This is important for defer patterns.

3. `context.WithTimeout()` / `WithDeadline()` — Time-Bound

When you need to say "stop after 2 seconds":

ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()

The context automatically cancels when the deadline hits. You still call cancel() in defer to release resources early if you finish sooner.

Listening for Cancellation

A context doesn't stop goroutines by force. It asks politely. You have to listen:

func queryWithContext(ctx context.Context, db *sql.DB) error {
    rows, err := db.QueryContext(ctx, "SELECT ...")
    if err != nil {
        return err
    }
    defer rows.Close()

    for rows.Next() {
        select {
        case <-ctx.Done():
            return ctx.Err() // context cancelled
        default:
            // process row
        }
    }
    return rows.Err()
}

The ctx.Done() channel closes when cancellation happens. Checking it in a select gives your goroutine a clean exit path.

The Pattern: Per-Request Context

In HTTP handlers, the request already carries a context:

func handler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context() // use this!

    // add timeout
    ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
    defer cancel()

    result, err := service.Process(ctx, r.Body)
    if err != nil {
        if errors.Is(err, context.DeadlineExceeded) {
            http.Error(w, "timeout", http.StatusGatewayTimeout)
            return
        }
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    json.NewEncoder(w).Encode(result)
}

Notice how we derive from r.Context() instead of context.Background(). This way, if the client disconnects, our timeout context inherits that cancellation.

Propagation: The Golden Rule

Every function that does I/O or spawns goroutines should take context.Context as its first parameter.

This is the convention. Follow it even when you don't think you need it. The caller knows their constraints better than you do.

Bad:

func FetchUser(userID string) (*User, error)

Good:

func FetchUser(ctx context.Context, userID string) (*User, error)

Goroutines and Context

When you spawn a goroutine, pass it the context:

For fan-out work, golang.org/x/sync/errgroup gives you cancellation and error propagation without hand-rolling channel bookkeeping:

go get golang.org/x/sync/errgroup

func fetchAll(ctx context.Context, urls []string) ([]Result, error) {
    g, ctx := errgroup.WithContext(ctx)

    results := make([]Result, len(urls))

    for i, url := range urls {
        idx, u := i, url
        g.Go(func() error {
            res, err := fetch(ctx, u)
            if err != nil {
                return err
            }
            results[idx] = res
            return nil
        })
    }

    if err := g.Wait(); err != nil {
        return nil, err
    }

    return results, nil
}

Key points:

Derive the child context with errgroup.WithContext so we can stop siblings on error
Pass ctx to fetch() so it can respect cancellation
g.Wait() waits for every goroutine and returns the first error

Common Mistakes

1. Storing Context in Structs

type Service struct {
    ctx context.Context // DON'T
}

Context is for function parameters, not struct fields. It makes the lifetime unclear.

2. Ignoring `ctx.Err()`

select {
case <-ctx.Done():
    return nil // what happened?
}

Better:

case <-ctx.Done():
    return fmt.Errorf("fetch interrupted: %w", ctx.Err())

3. Not Checking Cancellation in Tight Loops

for i := 0; i < 1e9; i++ {
    // heavy computation
    // never checks ctx.Done()
}

Check periodically:

for i := 0; i < 1e9; i++ {
    if i % 1000 == 0 {
        select {
        case <-ctx.Done():
            return ctx.Err()
        default:
        }
    }
    // heavy computation
}

4. Creating Background Contexts Deep in the Stack

func helper() {
    ctx := context.Background() // wrong
    db.QueryContext(ctx, ...)
}

Always accept context from the caller:

func helper(ctx context.Context) {
    db.QueryContext(ctx, ...)
}

Values: Use Sparingly

Context values carry request-scoped metadata:

type contextKey string

const traceIDKey contextKey = "traceID"

ctx = context.WithValue(ctx, traceIDKey, traceID)

Access them with a type check:

traceID, ok := ctx.Value(traceIDKey).(string)
if !ok {
    return errors.New("missing trace ID")
}

Rules for values:

Use for request-scoped data (trace IDs, auth tokens), not application data
Use typed keys to avoid collisions: type contextKey string
Don't use as a replacement for function parameters
Document what keys your function expects

Graceful Shutdown

The ultimate cancellation pattern: shutting down a server cleanly.

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    srv := &http.Server{
        Addr:    ":8080",
        Handler: http.HandlerFunc(handler),
    }

    go func() {
        <-ctx.Done()
        shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
        defer cancel()
        srv.Shutdown(shutdownCtx)
    }()

    if err := srv.ListenAndServe(); err != http.ErrServerClosed {
        log.Fatal(err)
    }
}

signal.NotifyContext creates a context that cancels on Ctrl+C. We use that to trigger a graceful shutdown with its own timeout.

Summary

Always derive from the caller's context — never create Background() deep in the stack
Pass context as the first parameter — make it obvious and consistent
Check ctx.Done() in loops and selects — give your goroutines an exit
Call cancel() in defer — clean up resources, prevent leaks
Use WithTimeout for external calls — protect against slow dependencies
Use values sparingly — they're convenient but opaque

Context is about scope: every request gets a boundary, every goroutine lives within it, and when the boundary closes, everything inside knows to stop.

That's how you build systems that don't leak: memory, connections, or intentions.

How to Merge PGN Files in F#: Streaming, Performance, and Discriminated Unions

Fri, 05 Jun 2026 00:00:00 GMT

I needed to merge hundreds of Lichess PGN files into a single database for opening preparation. The existing tools either required Python dependencies, discarded game metadata, or merged games into variation trees rather than concatenating them cleanly. I wanted a standalone CLI that takes a folder of .pgn files and produces one merge.pgn — fast, memory-efficient, and correct.

The result is PGN Merger, a .NET CLI tool built in F# that merges chess PGN files with streaming I/O and ~64 KB memory usage.

Why F# for a CLI Tool

F# is a functional-first language on .NET. Three features make it a strong fit for file-processing CLIs.

Discriminated unions model success and failure at the type level:

type FileResult =
    | Success of string * int64
    | Failure of string * string

The compiler forces you to handle both cases. No nullable returns, no forgotten error checks.

Pattern matching makes argument parsing readable and exhaustive:

match args.[i] with
| "--output" when i + 1 < args.Length ->
    loop (i + 2) { opts with OutputPath = args.[i + 1] }
| "--recursive" ->
    loop (i + 1) { opts with Recursive = true }
| "--help" | "-h" | "-?" ->
    None
| arg when arg.StartsWith("--") ->
    printfn "Error: Unknown option '%s'" arg
    None

First-class .NET interop means System.IO, System.Diagnostics.Stopwatch, and high-performance streams are available directly. No FFI overhead. You write F#, you get native .NET performance.

Streaming I/O: Merging PGN Files Without Loading Them into Memory

The naive merge approach loads entire files into memory before writing. For a folder with 10 GB of PGN files, that's 10 GB in RAM before a single byte hits disk.

PGN Merger never loads an entire file into memory. It streams through each file with 64 KB buffers:

use inputStream = new FileStream(pgnFile, FileMode.Open, FileAccess.Read, FileShare.Read, 65536)
use reader = new StreamReader(inputStream, Encoding.UTF8, true, 65536)

The copy loop operates on a pre-allocated char array, allocated once per file:

let streamFileToWriter (reader: StreamReader) (writer: StreamWriter) bufferSize =
    let buffer = Array.zeroCreate<char> bufferSize
    let mutable charsRead = 0
    while (charsRead <- reader.Read(buffer, 0, buffer.Length); charsRead > 0) do
        writer.Write(buffer, 0, charsRead)

No strings created during streaming. Memory footprint is ~64 KB per file regardless of input size. You could merge a 50 GB PGN archive on a machine with 512 MB of RAM.

Both input and output streams use 64 KB buffers, which means writes are batched, the OS receives larger sequential requests, and the read-ahead cache can prefetch the next block. The access pattern is purely sequential: front-to-back reads, append-only writes. On a modern NVMe SSD, throughput is hundreds of MB/s, bottlenecked by the storage device, not the code.

Safety comes from F#'s use bindings (deterministic Dispose() even on exceptions) and exhaustive exception handling per file. The tool returns non-zero exit codes on failure, making it script-friendly.

Discriminated Unions vs Exceptions for Error Handling

Most CLI tools use try-catch blocks and return -1 on any error. PGN Merger models every outcome explicitly:

type FileResult =
    | Success of string * int64       // filename, bytes written
    | Failure of string * string       // filename, error message

Each file processed produces a FileResult. The merge loop collects these and reports partial success (exit code 2) when some files fail but others succeed. This is critical when you're merging 10,000 files — one corrupted file shouldn't kill the entire batch.

The compiler enforces exhaustiveness. If you add a new case to FileResult, every match expression breaks at compile time until you handle it. No silent failures.

Install and Try PGN Merger

# Install from NuGet (requires .NET 10.0 SDK or later)
dotnet tool install --global PgnMerger

# Merge all PGN files in a folder
pgn-merger ./your_pgn_folder

# With verbose output and custom filename
pgn-merger ./your_pgn_folder --verbose --output database.pgn

Browse the NuGet package page for version history, or check the project page for a feature overview, comparison table, and FAQ.

More chess tooling: I also built corentings/chess, a Go library for chess move generation, PGN encoding, and UCI interop. For tournament stories, read about my experience at the Karen Asrian Memorial in Armenia.

Related posts on performance and concurrency: If you enjoyed the streaming I/O patterns here, you might like my posts on worker pools in Go and parallel merge sort with goroutines.

What Western UX Can Learn from Japanese Web Design

Fri, 05 Jun 2026 00:00:00 GMT

What Western UX Can Learn from Japanese Web Design

Western web design has spent years removing things: navigation, sidebars, copy, options, hesitation. What remains is familiar: a hero image, a short promise, one button, three benefit cards, and maybe a testimonial.

This can work. It can also become an ideology.

Not every user is ready to click. Not every product can be understood through a slogan. Not every culture reads visual simplicity as trust. Japanese websites challenge one of contemporary UX's strongest assumptions: less information does not automatically mean more clarity.

Western designers do not need to copy Japanese visual density. A Yahoo! Japan-style homepage will not improve a French SaaS landing page or an American portfolio site. The useful lesson is narrower and stronger:

What useful information have we removed in the name of simplicity?

Japanese web density is useful when it gives context, comparison, and reassurance around the decision.

Context Is Part of Trust

Minimalist pages often assume the user needs a promise and a next step. Dense Japanese commercial pages often assume the user also needs the situation around that promise.

What is the price?
What are the conditions?
What are the alternatives?
Is this campaign still active?
What happens after clicking?
Where is support?
Is the company real?
Can I compare before deciding?

In high-risk or high-comparison categories such as finance, healthcare, travel, electronics, education, insurance, government services, subscriptions, and B2B procurement, a beautiful sparse page can feel elegant but insufficient. It can make the user wonder what is being hidden.

A sparse page says:

Trust us. Click.

A reassurance-rich page says:

Here is the information you need to trust us.

Neither model is universally better. The trap is mistaking visual confidence for user confidence. A page can look calm while leaving too many practical questions unanswered.

Visible care builds trust that polish alone cannot.

The Problem Is Unmanaged Density

Dense design has costs. Japanese websites can overload the eye, bury priority, create accessibility problems, rely on image-based text, break on mobile, and reflect stakeholder politics more than user needs.

Research on visual complexity and website search shows that complexity can slow people down. Even users familiar with dense pages are affected by weak hierarchy, poor grouping, and excessive competition between elements.

The real divide is between accumulated density and designed density.

Accumulated density happens when every department adds a banner, every campaign demands homepage space, every legal concern becomes a warning block, and every stakeholder protects a link.

Designed density accepts that users need information, then organizes it into clear layers. It uses hierarchy, grouping, rhythm, disclosure, and strong labeling. It gives the user context without making every element shout.

Structured Density: Bento, Ma, and Service Counters

One Japanese-inspired pattern that travels well is the bento layout. A bento box can contain many things, but each thing has a place. Variety does not become chaos because the structure is visible.

In interface design, a bento approach groups compact modules around user needs:

main value proposition
trust signals
pricing or conditions
comparison
user proof
support
related actions

The page becomes a set of zones rather than a wall of competing elements.

This connects to another useful Japanese concept: ma, often translated as interval, pause, or meaningful space. Ma is the gap that lets elements make sense together.

Ma is not empty space for decoration. It is the interval that helps dense information become readable.

In UI, ma can separate decision groups, calm a high-commitment action, distinguish promotional content from functional content, or create rhythm between dense modules.

A dense page without ma becomes exhausting. A minimal page without context becomes vague. A strong interface needs both: information and interval.

From Funnel to Service Counter

A Western landing page often behaves like a funnel. Its job is to move the user toward one action.

A Japanese commercial homepage often behaves more like a service counter. Its job is to answer multiple kinds of users at once: new visitors, returning customers, campaign shoppers, comparison seekers, support users, loyalty members, and people who need reassurance before acting.

From a conversion-funnel perspective, this looks inefficient. But not every page should be a pure funnel. Some pages need to orient. Some need to compare. Some need to reassure. Some need to support exploration before conversion.

The interface should not force evaluating users into a checkout mindset too early.

What Western Designers Can Borrow

Japanese web design is not a style to copy directly. It is a set of questions worth importing.

What does the user need to know before they can trust this action?
Are we hiding information users need for comparison?
Do we support exploration as well as conversion?
Which details reduce anxiety, and which details merely compete for attention?
Can density adapt to first-time users, returning users, experts, mobile users, and cautious buyers?

Put anxiety-reducing details close to the moment of decision: secure payment, cancellation terms, delivery estimate, support availability, official status, setup time, compatibility, or return policy.

Do not borrow banner competition, image-based text, weak hierarchy, purposeless duplicate links, decorative mascots that do not help the task, or the habit of making every stakeholder equally visible.

A useful redesign asks one question:

What is the user's uncertainty, and what interface structure reduces it?

Toward Trustful Minimalism

A better term than maximalism or pure minimalism is trustful minimalism.

Trustful minimalism keeps the visual discipline of modern UX but restores the information users need to feel confident. It still defines a primary action. It still uses hierarchy and space. But it does not rely on brand mood alone.

For a Western product page, landing page, or service page, the framework is simple:

Define the primary action.
Identify the top anxieties blocking that action: price, setup time, cancellation, delivery, compatibility, support, security, legitimacy, return policy.
Place short reassurance cues near the action.
Group deeper details into expandable sections, comparison modules, or bento-style cards.
Separate promotional content from decision-critical content.
Test perceived trust, completeness, and confidence, not only conversion speed.

A redesign can make a page look cleaner while making users less certain. If the only metric is visual preference, the team may remove exactly the elements that made the page trustworthy.

Good design means enough: enough context to understand, enough hierarchy to choose, enough reassurance to trust, enough space to think, and enough restraint to avoid overload.

The useful lesson from Japanese information-rich design is not more decoration. It is a better design question: what context does the user need, and what can we remove without weakening trust?

References and further reading

Richard E. Nisbett and Takahiko Masuda, "Culture and Point of View"
Takahiko Masuda and Richard E. Nisbett, "Attending Holistically Versus Analytically"
Nordhoff, August, Oliveira, and Reinecke, "A Case for Design Localization"
Baughan et al., studies on visual complexity, website search, and cross-cultural attention
Dianne Cyr and Haizley Trevor-Smith, "Localization of Web Design"
Elizabeth Würtz, "Intercultural Communication on Web Sites"
W3C, "Requirements for Japanese Text Layout"
Japan House Los Angeles, "A Perspective on the Japanese Concept of Ma"
Stanford Encyclopedia of Philosophy, "Japanese Aesthetics"

Why Japanese Websites Look Overloaded: Density, Tokyo, and Trust

Sun, 31 May 2026 00:00:00 GMT

Why Japanese Websites Look Overloaded: Density, Tokyo, and Trust

Open Yahoo! Japan after browsing a Western startup website and the contrast is immediate. The Western page gives you a hero image, a sentence, and a button. The Japanese page gives you news, weather, shopping, finance, auctions, login, campaigns, rankings, emergency notices, and many routes elsewhere.

To a Western designer, this often looks like clutter. The better UX question asks: what job is this density doing?

Many Japanese domestic websites use high-density interfaces to show context, reassurance, comparison options, promotions, navigation, and trust cues at once. The result can be messy, but it is not always meaningless.

A Japanese portal homepage showing high-density UI: news, weather, campaigns, rankings, and navigation modules are all visible at once.

Density Is Not One Thing

Calling everything "clutter" hides the differences that matter.

Type	What it does	Always bad?
Information density	Shows facts, specs, labels, notices, comparisons	No
Navigational density	Gives many routes, menus, categories, breadcrumbs	Sometimes
Promotional density	Shows coupons, rankings, campaigns, limited-time offers	Depends
Trust density	Shows security, support, returns, company info, official notices	Often useful
Technical bloat	Adds heavy JS, slow assets, tracking, excessive DOM	Yes

Only technical bloat is inherently bad. The rest can serve a function, even if poorly executed. A page with visible delivery conditions is not the same problem as a page slowed down by unoptimized scripts.

Separating density types lets us diagnose what each part of the page is doing, rather than collapsing everything into "clutter."

Websites as Service Counters

Western landing pages often behave like posters: one dominant message, one focal object, one call to action. Many Japanese portals, service pages, and retail sites behave more like maps, directories, counters, or flyers. They assume users arrive with questions, comparisons, and multiple possible next steps.

A Japanese homepage often answers several questions upfront:

What is new?
What campaign is active?
Where do I log in?
Where is support?
What are the rankings?
What are the conditions?
Can I compare?
Is this official?

In this model, the page is part storefront, part catalogue, part customer-service desk, part seasonal promotion board. The density supports multiple user intents at once.

A Japanese homepage annotated as a service counter: each cluster answers a different user question before it needs to be asked.

This does not make every dense page good. A service counter can still be chaotic. The point is that the density may be trying to solve a different design problem from a minimalist landing page.

Tokyo Helps Explain the Pattern

Japanese web density makes more sense beside the dense physical and media environments many users navigate every day.

Walk through Shinjuku Station at rush hour and information is part of the architecture: platform numbers, exit codes, train lines, arrows, warnings, shop signs, campaign posters, ticket machines, and convenience-store shelves. The first impression is density. The second is that people know how to move through it.

Japanese urban density is often vertical, layered, and sign-rich. A domestic portal page can feel less strange when read through this spatial logic.

Tokyo did not cause Japanese web design. But Tokyo, Akihabara, department-store floor guides, konbini shelves, ticket machines, manga pages, chirashi flyers, and public notice boards all train similar habits: scan, filter, compare, follow labels, and enter from more than one point.

Dense is not random. Japanese stations compress routes, exits, platforms, warnings, and services into navigable systems.

The useful analogy is that both city and website can operate as navigable information environments.

What Research Suggests

The science is suggestive, not conclusive. It does not prove that Japanese users prefer dense websites. But it does show that attention is not culturally neutral.

Masuda and Nisbett's work on analytic versus holistic attention found that Western participants tended to focus more on focal objects, while Japanese participants reported more contextual and relational information. Their 2001 underwater-scene study was not about websites. It suggests one design hypothesis: surrounding interface cues may carry more meaning for users trained in context-rich environments.

Miyamoto, Nisbett, and Masuda later compared Japanese and American physical environments and found that Japanese scenes tended to contain more elements, ambiguity, and overlapping relationships. The built environment is not just scenery. It can train perception.

Cross-cultural web-design research also finds measurable visual differences. Nordhoff, August, Oliveira, and Reinecke analyzed 80,901 website designs across 44 countries and found that Japan, China, and South Korea clustered toward higher visual complexity and text density.

But familiarity is not efficiency. Baughan and colleagues studied 84 U.S. American and 65 Japanese participants searching website screenshots of varying complexity. Japanese participants took longer overall, and complexity hurt search efficiency. Dense interfaces may be locally familiar and commercially meaningful, but they still need hierarchy.

Density Still Needs Hierarchy

Japanese text composition also changes what compact information can look like. The W3C's Requirements for Japanese Text Layout documents mixed writing systems, vertical and horizontal composition, line-breaking rules, ruby annotations, emphasis marks, and spacing conventions.

Japanese script does not cause clutter. But kanji, kana, Latin characters, numerals, and compact labels create different possibilities for dense display than English or French interfaces.

The useful lesson is structured density: context-rich design that does not sacrifice usability. Useful density supports comparison, reassurance, and navigation. Costly density hides priority, forces exhaustive search, and makes every element compete.

Some Japanese websites are cluttered. Visual hierarchy can collapse. Promotional banners can multiply until they obscure the thing they promote. Technical bloat can make a page slow and inaccessible.

What looks like clutter from one design ideology can be context, reassurance, comparison, and transparency from another. A Western landing page says: "here is the action." A Japanese high-density page often says: "here is the situation."

The UX test is simple: what happens if you remove an element? Does the user lose information they needed, confidence they relied on, or a route they expected? If yes, it was not only clutter. It was structure.

Instead of asking why Japanese websites are overloaded, ask what the overload is doing.

Sources and Further Reading

Baughan, A., Oliveira, N., August, T., Yamashita, N., & Reinecke, K. (2021). "Do Cross-Cultural Differences in Visual Attention Patterns Affect Search Efficiency on Websites?" Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. PDF
Cyr, D., & Trevor-Smith, H. "Localization of Web Design: An Empirical Comparison of German, Japanese, and United States Web Site Characteristics." Journal of the American Society for Information Science and Technology, 55(13), 1199-1208, 2004.
Masuda, T., & Nisbett, R. E. (2001). "Attending Holistically Versus Analytically: Comparing the Context Sensitivity of Japanese and Americans." Journal of Personality and Social Psychology, 81(5), 922-934.
Miyamoto, Y., Nisbett, R. E., & Masuda, T. "Culture and the Physical Environment: Holistic Versus Analytic Perceptual Affordances." Psychological Science, 17(2), 113-119, 2006.
Nisbett, R. E., & Masuda, T. (2003). "Culture and Point of View." Proceedings of the National Academy of Sciences, 100(19), 11163-11170.
Nordhoff, M., August, T., Oliveira, N. A., & Reinecke, K. (2018). "A Case for Design Localization: Diversity of Website Aesthetics in 44 Countries." Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.
W3C. (2020). "Requirements for Japanese Text Layout." JLREQ
Würtz, E. "Intercultural Communication on Web Sites: A Cross-Cultural Analysis of Web Sites from High-Context Cultures and Low-Context Cultures." Journal of Computer-Mediated Communication, 11(1), 2005.

TDD Isn't About Bugs — It's Your Permission to Refactor

Sun, 24 May 2026 00:00:00 GMT

TDD Isn't About Bugs — It's Your Permission to Refactor

The Moment It Clicked

Three years ago, I had a crisis of confidence with test-driven development.

I'd built a user registration system with 80% test coverage, all green, CI passing. Then I decided to refactor the email validation logic: just move it to a separate module. A tiny change.

Thirty minutes later, everything was broken.

My tests tested how the code worked, not what it did. The moment the structure changed, they failed. I spent more time fixing tests than doing the refactoring itself.

I'll show you what I was doing wrong. It might look familiar.

The Trap: Testing Structure

Here's the TypeScript code I had, a User class with inline validation:

class User {
	email: string;
	age: number;

	constructor(email: string, age: number) {
		if (!email.includes("@")) {
			throw new Error("Invalid email");
		}
		if (age < 18) {
			throw new Error("Must be at least 18 years old");
		}
		this.email = email;
		this.age = age;
	}
}

And the test:

describe("User", () => {
	it("should create a user", () => {
		const user = new User("alice@example.com", 25);
		expect(user.email).toBe("alice@example.com");
		expect(user.age).toBe(25);
	});
});

What am I testing? Structure. I'm reaching into the object and checking its properties directly.

Now watch what happens when I hide the internals:

class User {
	private email: string;
	private age: number;

	constructor(email: string, age: number) {
		if (!email.includes("@")) {
			throw new Error("Invalid email");
		}
		if (age < 18) {
			throw new Error("Must be at least 18 years old");
		}
		this.email = email;
		this.age = age;
	}

	get email(): string {
		return this.email;
	}

	get age(): number {
		return this.age;
	}
}

The test breaks. The behavior hasn't changed (the same user is created with the same rules), but I changed the structure.

I'm now rewriting tests for a refactoring that shouldn't require any test changes.

The Fix: Test Behavior

The solution: test what the code does, not how it's organized.

But first, I need to fix another problem. Throwing exceptions makes testing awkward: you need expect(() => fn()).toThrow(), which hides the error behind a callback. The fix is treating errors as first-class values instead of hidden control flow.

Here's the pattern that made this click, the Result type:

type Result<T, E> =
	| { readonly ok: true; readonly value: T }
	| { readonly ok: false; readonly error: E };

function ok<T>(value: T): Result<T, never> {
	return { ok: true, value };
}

function err<E>(error: E): Result<never, E> {
	return { ok: false, error };
}

Instead of throwing, functions return a Result that's either a success or a failure. Both paths are explicit. No try/catch. Errors are data you can inspect, test, and chain. (In production code, you'd use a library like neverthrow which adds map, flatMap, and match for composing results.)

With Result in place, I can rewrite the User class with a static factory method:

class UserCreationError {
	constructor(
		public readonly message: string,
		public readonly field: string
	) {}
}

class User {
	private constructor(
		private readonly _email: string,
		private readonly _age: number
	) {}

	static create(email: string, age: number): Result<User, UserCreationError> {
		if (!email.includes("@")) {
			return err(new UserCreationError("Invalid email format", "email"));
		}
		if (age < 18) {
			return err(new UserCreationError("Must be at least 18 years old", "age"));
		}
		return ok(new User(email, age));
	}

	get email(): string {
		return this._email;
	}

	get age(): number {
		return this._age;
	}
}

Now the tests verify behavior, the observable contract:

describe("User Registration", () => {
	it("should create a user with valid email and age", () => {
		const result = User.create("alice@example.com", 25);

		expect(result.ok).toBe(true);
		if (result.ok) {
			expect(result.value.email).toBe("alice@example.com");
			expect(result.value.age).toBe(25);
		}
	});

	it("should reject invalid emails", () => {
		const result = User.create("not-an-email", 25);

		expect(result.ok).toBe(false);
		if (!result.ok) {
			expect(result.error.field).toBe("email");
			expect(result.error.message).toMatch(/email/i);
		}
	});

	it("should reject users under 18", () => {
		const result = User.create("alice@example.com", 17);

		expect(result.ok).toBe(false);
		if (!result.ok) {
			expect(result.error.message).toMatch(/18/);
		}
	});
});

When I refactor the internals now (make _email private, change the data structure, extract a class), these tests keep passing. They test the contract: valid inputs succeed, invalid inputs fail with the right error. The internal structure is irrelevant.

This is the red-green-refactor cycle that defines test-driven development: write a failing test (red), make it pass with the simplest code (green), then improve the design while keeping tests green (refactor). The tests give you permission to refactor aggressively because they catch real problems, not structural changes.

Going Deeper: Value Objects

Once I started testing behavior, I noticed something. My validation logic was scattered. The same email check appeared in registerUser, changeEmail, and resetPassword. Three places, same rule, same potential for bugs.

The fix is a Value Object: a small, self-validating type that represents a single concept. If you have an Email instance, it's valid by construction:

class InvalidEmailError {
	readonly message = "Invalid email format";
}

class Email {
	private constructor(private readonly _value: string) {}

	static create(raw: string): Result<Email, InvalidEmailError> {
		const value = raw.trim().toLowerCase();
		if (!value.includes("@")) {
			return err(new InvalidEmailError());
		}
		return ok(new Email(value));
	}

	get value(): string {
		return this._value;
	}

	equals(other: Email): boolean {
		return this._value === other._value;
	}
}

Notice the normalization (trim().toLowerCase()) lives inside the Value Object. Every Email instance is guaranteed lowercase and trimmed. The validation rule exists in one place. If the rule changes, you change one class.

Similarly for Age:

class InvalidAgeError {
	readonly message = "Must be at least 18 years old";
}

class Age {
	private constructor(private readonly _value: number) {}

	static create(value: number): Result<Age, InvalidAgeError> {
		if (!Number.isInteger(value) || value < 18) {
			return err(new InvalidAgeError());
		}
		return ok(new Age(value));
	}

	get value(): number {
		return this._value;
	}
}

The User class now composes these Value Objects instead of accepting raw primitives:

class User {
	private constructor(
		private readonly _email: Email,
		private readonly _age: Age
	) {}

	static create(email: string, age: number): Result<User, UserCreationError> {
		const emailResult = Email.create(email);
		if (!emailResult.ok)
			return err(new UserCreationError(emailResult.error.message, "email"));

		const ageResult = Age.create(age);
		if (!ageResult.ok)
			return err(new UserCreationError(ageResult.error.message, "age"));

		return ok(new User(emailResult.value, ageResult.value));
	}

	get email(): string {
		return this._email.value;
	}

	get age(): number {
		return this._age.value;
	}
}

The same tests from before still pass. They test User.create() returning success or failure, which hasn't changed. The internal composition with Value Objects is invisible to the tests.

The Payment System

Last year, I refactored a production payment processing module: about 2,400 lines of conditional logic scattered across processPayment, validatePayment, and handleRefund. Validation rules were duplicated in all three functions.

The old tests looked like this:

it("processes payment", () => {
	const processor = new PaymentProcessor(mockGateway, mockDB);
	processor.validateAmount(100);
	processor.validateCurrency("USD");
	const result = processor.process({ amount: 100, currency: "USD" });
	expect(result.status).toBe("success");
	expect(mockGateway.charge).toHaveBeenCalledTimes(1);
});

Those tests called internal methods directly. When I extracted validation into a Payment Value Object, every test broke because validateAmount and validateCurrency no longer existed as separate methods.

I rewrote the tests to focus on the contract:

it("accepts valid payments", () => {
	const result = Payment.create({ amount: 100, currency: "USD" });
	expect(result.ok).toBe(true);
});

it("rejects negative amounts", () => {
	const result = Payment.create({ amount: -50, currency: "USD" });
	expect(result.ok).toBe(false);
});

Then I refactored aggressively. Extracted Currency as its own Value Object. Moved validation into the Payment factory. Deleted 800 lines of duplicated logic. Tests never broke once, because they tested what happened, not how.

The whole refactoring took a day. With the old tests, it would have taken a week: three days of refactoring, two days of fixing tests.

The same idea shows up in concurrency code too. When you refactor a pipeline or a worker pool, the contract stays the same while the internals change. The tests that survive are the ones that verify behavior, not goroutine counts.

The "TDD Is Slow" Myth

I used to think test-driven development slowed me down. Studies tell a more nuanced story: TDD has a modest upfront cost of 10–30% more development time, but reduces defects by 40–90% (Nagappan et al., Microsoft/IBM, 2008). George & Williams (2003) found TDD pairs took 16% more time but passed 18% more black-box tests.

The upfront cost is real. The payoff is also real. And it compounds: every safe refactoring makes the next one easier.

When I couldn't refactor safely, I coded defensively. I avoided changes. I left bad code in place because changing it was risky. Refactoring without tests is what's actually slow. It compounds with every month.

When TDD Doesn't Help

Test-driven development isn't always the right tool. Spikes and prototypes (code you'll throw away) don't need tests. Exploratory work where you don't know the shape of the solution yet. UI code where the behavior is visual, not logical. One-off scripts. Code with genuinely unstable requirements that change every sprint.

TDD pays off when you're building something you'll maintain and refactor over time. If the code won't survive the week, write the minimum.

Monday Morning

Take one function you wrote recently. Rewrite its tests to verify behavior (what it does) instead of implementation (how it's organized).

Watch what happens to your code.

Three years ago, I was afraid to refactor. Now I do it aggressively, because my tests give me permission. Test-driven development gave me a permission slip I didn't know I needed.

Next time, I'll show you how to build Value Objects that make invalid states impossible, and how testing them first makes the design emerge naturally.

If you found this useful, the next article in the series shows how to build Value Objects that make invalid states impossible. It comes out in a couple of weeks. You can subscribe via RSS or Atom to catch it when it lands.

— Corentin

Docker 29 Broke Traefik — Here's the Fix (and Why It Happened)

Mon, 04 May 2026 00:00:00 GMT

Docker 29 Broke Traefik — Here's the Fix (and Why It Happened)

The "Everything Is Down" Moment

You ran apt upgrade on a Sunday evening on your Linux server. Seemed harmless. Docker updated to v29.0.0. You rebooted because why not.

Now every service behind your Traefik reverse proxy returns a 404. Or a 502. Or just… nothing. Your monitoring is screaming. Your personal projects are unreachable. Your CI pipeline is stuck. And if you're running Coolify, Dokploy, or any platform that bundles Traefik, you can't even access the management UI to fix it.

You check the Traefik logs and see this, repeating every few seconds:

ERR Failed to retrieve information of the docker client and server host
  error="Error response from daemon: client version 1.24 is too old.
  Minimum supported API version is 1.44, please upgrade your client
  to a newer version" providerName=docker

Your first thought: "I didn't change anything in Traefik."

You're right. You didn't. Docker changed something under you.

Let me save you the debugging. Here's the fix.

The 30-Second Fix

This applies to Linux hosts running Docker Engine directly. If you're using Docker Desktop (macOS or Windows), you're not affected — updates are bundled automatically.

If your services are down right now and you need them back immediately, do this:

Edit (or create) /etc/docker/daemon.json:

sudo nano /etc/docker/daemon.json

Add this:

{
	"min-api-version": "1.24"
}

If you already have content in daemon.json (storage driver, log driver, etc.), merge the key into the existing object. Don't overwrite your whole config:

{
	"log-driver": "json-file",
	"log-opts": {
		"max-size": "10m",
		"max-file": "3"
	},
	"min-api-version": "1.24"
}

The most common mistake is a missing or trailing comma. JSON is strict about this.

Then restart Docker:

sudo systemctl restart docker

Important: Restarting Docker stops all running containers. If your containers have a restart policy (restart: unless-stopped or restart: always), they'll come back on their own. If not, you'll need to start them manually:

docker compose up -d

Docker Swarm: This fix works the same on Swarm nodes. Apply it to all managers and workers. Swarm services will reschedule automatically after the daemon restarts.

Rootless Docker? The config file lives at ~/.config/docker/daemon.json and you restart with systemctl --user restart docker.

That's it. Your services should come back within seconds. Traefik will reconnect to the Docker daemon, discover your containers, and routes will light up again.

Verifying it worked

Check Traefik's logs:

docker logs traefik --tail 20

You should no longer see the client version 1.24 is too old error. If you see Configuration received from provider docker, that's a positive confirmation (this message only appears at INFO log level, which may not be your default).

The strongest verification is the API version check below.

Confirm your Docker API versions:

docker version --format '{{.Server.MinAPIVersion}}'

This should now return 1.24 instead of 1.44.

Why this works: The min-api-version setting overrides Docker's built-in floor. It tells the daemon to accept connections from clients using API version 1.24 and above, restoring the backward compatibility that Docker 29 removed. This config change keeps Docker 29 fully operational while re-enabling communication with older API clients. You're still running Docker 29 with all its features — you've just told it to accept a wider range of client versions.

Is this safe? For now, yes. This setting only changes how Docker negotiates API versions with clients. The API version is a negotiation protocol that controls feature detection between client and server. It doesn't touch encryption or authentication. The risk is that older API clients might miss newer fields or features, but Traefik only needs to read container labels and inspect networks. It doesn't need API 1.44 for that.

The Real Fix: Update Traefik

The daemon.json workaround is a bandage. The actual fix is updating Traefik to v3.6.1 or later.

Traefik v3.6.1, released shortly after this incident, added automatic Docker API version negotiation. Instead of hardcoded v1.24, Traefik now queries the daemon for its supported version range and picks the highest mutually compatible version. This is how the Docker SDK is supposed to work.

If you manage Traefik directly (docker-compose, Dockerfile, etc.), change your image tag:

services:
  traefik:
    image: traefik:v3.6.1

Or pull the latest:

docker pull traefik:v3.6.1

Then restart:

docker compose up -d traefik

If you're on the older standalone docker-compose, the command is docker-compose up -d traefik.

Once Traefik is on 3.6.1+, you can remove the "min-api-version" workaround from daemon.json and restart Docker again. Everything will keep working because Traefik now negotiates the API version correctly on its own.

If you can't update Traefik directly

If you're running a platform that bundles Traefik (Coolify, Dokploy, Appwrite, CapRover), you might not control the Traefik image version. In that case:

Check if your platform has an update. Most of them shipped patches within days.
Use the daemon.json workaround until the platform updates its bundled Traefik.
Check the platform's GitHub issues. This broke so many deployments that every affected platform has a thread about it.

What Actually Happened

Let me explain the technical chain of events, because understanding it helps you prevent similar breakages.

Docker's API version floor

Docker Engine exposes a versioned API. Clients (CLI, SDK, third-party tools) negotiate which version to use during the initial handshake. Before Docker 29, the daemon supported API versions going all the way back to v1.24, which was introduced with Docker 1.12 in 2016. Nearly a decade of backward compatibility.

With Docker Engine v29.0.0 (released November 10, 2025), Docker raised the minimum supported API version to v1.44. API v1.44 corresponds to Docker Engine v25, released in late 2023.

This was deliberate and documented. Docker's deprecation policy states that API versions are supported for a limited number of major releases. API 1.24 had been deprecated for years. Docker 29 finally enforced the deprecation.

The result: any tool compiled against an older Docker SDK that defaults to API v1.24 as its negotiation floor gets rejected immediately. No fallback. No graceful degradation. A hard error: client version 1.24 is too old.

Why Traefik specifically exploded

Traefik uses Docker's official Go SDK (github.com/docker/docker/client) for container discovery. It reads container labels, figures out routing rules, and dynamically updates its configuration.

The problem is in how the Go Docker SDK initializes. When you create a new client with client.NewClientWithOpts(), the SDK defaults to negotiating from API v1.24. This has been the SDK's default since it was written. It was never a problem because Docker always accepted v1.24.

Docker 29 stopped accepting it. The negotiation fails before Traefik can even list containers. No containers discovered means no routes configured means every request hits a 404 or 502.

This affected all Traefik versions through v3.6.0, including v2.x. Traefik was using an SDK default (API v1.24) that Docker 29 no longer accepts.

It's Not Just Traefik

Traefik got the most attention because a broken reverse proxy takes down everything behind it. But Docker 29's API floor broke a wide range of tools:

Tool	What Broke	Fix
Traefik (v3.6.0 and earlier)	Container discovery, all routing	Update to v3.6.1+
Portainer (before v2.33.5)	Can't connect to Docker environments	Update to v2.33.5+ (LTS) or v2.36.0+ (STS)
Watchtower	Can't query containers for updates	Switch to `nickfedor/watchtower` (original `containrrr/watchtower` is unmaintained)
LazyDocker	Docker connection fails	Update to v0.24.2+
Dokploy	Bundled Traefik fails	Update Traefik image to v3.6.1
Coolify	Bundled Traefik fails	Check for Coolify update
Appwrite	Bundled Traefik v2.11 fails	Update to Appwrite version using Traefik v3
JetBrains IDEs	Docker plugin throws 400 errors	Update IDE / Docker plugin
Spring Boot (Testcontainers)	docker-java defaults to API v1.32	Update to Spring Boot 3.5.8+ (includes workaround), or set `api.version=1.44` in `docker-java.properties`
Swarmpit	Docker API calls fail	No fix: project is archived
CapRover	Docker API connectivity	Check for update
CasaOS	Docker API connectivity	Check for update
cAdvisor	Metrics collection stops	Update to v0.53.0+

The pattern repeats across every entry: the tool uses a Docker SDK with a hardcoded minimum version below 1.44, Docker 29 rejects the connection, and the tool can't communicate with the daemon.

Swarmpit is the saddest entry on that list. The project is archived. No fix is coming. Docker 29 finished what abandonment started.

Timeline

For the record, here's how this played out:

Nov 10, 2025 — Docker Engine v29.0.0 released. Minimum API version raised to v1.44. Traefik immediately breaks for everyone who auto-updates.
Nov 10-11, 2025 — Reports flood in. Traefik GitHub issue #12253 is filed. The Docker Community Forums thread explodes. The Traefik community forum fills up. Coolify, Dokploy, and Appwrite all see the same issue.
Nov 11, 2025 — Community identifies the daemon.json workaround and the DOCKER_MIN_API_VERSION environment variable as immediate workarounds.
Nov 12, 2025 — Traefik maintainer PR #12256 adds automatic API version negotiation. Community test builds appear on Docker Hub.
Nov 13-14, 2025 — Traefik v3.6.1 released with the fix. Portainer ships v2.33.5 and v2.36.0 with updated API support. LazyDocker ships v0.24.2.
Nov 2025 onward — The rest of the tooling world catches up. Spring Boot, JetBrains, and others ship fixes over the following weeks.

From "everything is broken" to "official fix available": roughly 48 hours. Solid open-source incident response. But if you were one of the people whose services went down on a Sunday night, it probably felt a lot longer.

Takeaways

For your infrastructure

Pin your Docker version in production. Docker 29's release notes documented the API version change. But if Docker auto-updates via your package manager (as it does on Ubuntu with the official repository), you get the breaking change whether you read the notes or not.

On Ubuntu/Debian:

sudo apt-mark hold docker-ce docker-ce-cli containerd.io

When you're ready to upgrade, unpin:

sudo apt-mark unhold docker-ce docker-ce-cli containerd.io

Pin your Traefik version too. Using traefik:latest means you get updates automatically, which is usually fine, but it also means you can't predict when behavior changes. Use explicit version tags like traefik:v3.6.1.

Keep a daemon.json backup. This file is tiny but critical. If you have it backed up or version-controlled, you can apply the workaround in under a minute instead of googling for it.

For the broader picture

A huge number of Docker-adjacent tools depended on an API version Docker deprecated years ago. The SDK default of v1.24 was never updated because Docker was always backward-compatible. Tools that were actively maintained shipped fixes within days. Tools that weren't (Swarmpit, older CasaOS setups) are now permanently broken on Docker 29+.

The min-api-version escape hatch is Docker acknowledging the pain with a temporary lever. Eventually, even that setting may go away.

If you maintain a tool that talks to Docker, check your SDK's default API version negotiation. If it's hardcoded to a minimum, make it dynamic. The Traefik PR that fixed this was a few dozen lines of code.

Update Traefik. Pin your Docker version. The order is up to you.

Why I Built ZaString

Thu, 30 Apr 2026 00:00:00 GMT

ZaString started with a frustration. I was working on a high-throughput system in C#, and every profiler run showed the same thing: strings everywhere. Allocations piling up. The garbage collector working overtime. Not because the code was wrong, but because strings in C# are immutable and every operation creates a new one.

The Problem With Strings

Here's what I mean. Take something innocent-looking:

var label = $"User {name} ({age}) logged in at {DateTime.Now:t}";

One line. Clean. Readable. And underneath, it allocates. The interpolated string creates a string on the heap. The DateTime.Now:t formatting creates another. The name substring reference adds overhead. Even something as simple as str1 + str2 doesn't modify in place. It allocates a brand new string with the combined content.

In C#, strings are immutable. That's not a flaw. It's a deliberate design choice that makes them thread-safe and predictable. But immutability has a cost: every operation that "changes" a string actually creates a new one. Concatenation, formatting, trimming, splitting, they all allocate.

For most code, this is fine. The garbage collector handles it. You never notice. But in hot paths like game loops, UI rendering, or request handlers processing thousands per second, those allocations add up. I was working with ImGui, where every frame builds strings for labels, tooltips, and debug output. Frame after frame, sixty times a second. The profiler wasn't subtle about it: strings dominated the allocation graph.

I tried the usual fixes. StringBuilder for concatenation. string.Create() for pre-sized allocations. ArrayPool<char> for reuse. They helped, but they all still allocated on the heap eventually. The fundamental issue was that the output was always a string, and strings live on the heap.

I didn't want to optimize allocation. I wanted to eliminate it.

Enter Span<T>

.NET 2.1 introduced Span<T>, and it changed what was possible.

A Span<T> is a view over a contiguous region of memory. It can wrap a managed array, a stack-allocated buffer, or unmanaged memory. Crucially, it's a ref struct. That means it lives on the stack. It can't be boxed, can't be stored in a field, can't be used across async boundaries. The runtime enforces this at compile time.

What you get in exchange is zero-allocation memory access. Slicing a span doesn't copy. It just creates a new view with an adjusted offset and length. Writing into a span modifies the underlying memory directly. No GC pressure. No hidden allocations.

Span<char> buffer = stackalloc char[64];
var written = "Hello".AsSpan().CopyTo(buffer);
var slice = buffer[..5]; // zero-copy view

This was the insight: instead of building strings and letting the GC clean up, write directly into a buffer you already own. The buffer can be on the stack for small strings, or pooled for larger ones. When you're done, you have a ReadOnlySpan<char>, which is just a view over the result with no heap allocation at all.

The catch is that working with raw spans is verbose. There's no fluent API, no formatting helpers, no interpolation. You're manually tracking offsets and calling TryFormat on every value. It works, but it doesn't feel like writing C#. It feels like assembly with extra steps.

I wanted the ergonomics of StringBuilder with the allocation profile of Span<T>. That's where ZaString came from.

The Design Philosophy

The core idea behind ZaString is simple: zero-allocation should feel normal.

Compare the two:

// StringBuilder — 146 ns, 480 B allocated
var sb = new StringBuilder();
sb.Append("Name: ").Append("John").Append(", Age: ").Append(25);
var result = sb.ToString();

// ZaSpanStringBuilder — 115 ns, 0 B allocated
Span<char> buffer = stackalloc char[50];
var builder = ZaSpanStringBuilder.Create(buffer);
builder.Append("Name: ").Append("John").Append(", Age: ").Append(25);
var result = builder.AsSpan();

Same shape. Same chaining. Same mental model. But the second version never touches the heap. The buffer is stack-allocated, the builder is a ref struct, and AsSpan() returns a view, not a copy.

The fluent API was non-negotiable. Every Append returns ref this, so you can chain calls without creating intermediate references. It supports all the types you'd expect: strings, chars, numbers, booleans, dates, with formatting and culture providers:

builder.Append("Pi: ").Append(Math.PI, "F2")
       .Append(", Date: ").Append(DateTime.Now, "yyyy-MM-dd");

C# 10's interpolated string handlers make it even cleaner:

builder.Append($"User: {name}, Age: {age}, Pi: {Math.PI:F2}");

The handler intercepts the interpolation at compile time and routes each placeholder directly into the span buffer. No intermediate string. No Format() call. Just writes.

I also wanted safety. Raw spans are dangerous. Write past the end and you corrupt memory. ZaString provides TryAppend variants that return false instead of throwing when the buffer is full. TryAppendLine is atomic: if there isn't room for both the content and the newline, nothing gets written. No partial state. No corruption.

The library grew from there. Escape helpers for JSON, HTML, URLs, CSV. Path and query parameter builders. A pooled builder for cases where the stack isn't enough. A UTF-8 writer for scenarios that need bytes instead of chars. Each piece follows the same principle: write into a buffer you control, return a view, allocate nothing.

What I Learned

The biggest lesson was about tradeoffs. Zero-allocation is not free. It's a constraint, and constraints shape design. ref struct limitations mean you can't store a builder in a field, pass it to async methods, or return it from a function that escapes the stack. You have to think about buffer sizes upfront. You lose the convenience of string being everywhere in the .NET ecosystem.

But those constraints also force clarity. When you have to declare your buffer size, you think about how much data you actually handle. When you can't allocate, you design around reuse. The code becomes more intentional.

I also learned that zero-allocation isn't always the answer. For a one-off log message, string interpolation is fine. The GC won't notice. ZaString is for the hot paths, the code that runs thousands of times per frame, per request, per second. The profiler tells you where those paths are. Trust the profiler.

On API design, I found that the best abstraction is the one you forget is there. If someone can read ZaString code and not realize it's zero-allocation, that's a win. The ergonomics should be invisible. The performance should be the default, not something you opt into with ceremony.

And honestly? There's a specific kind of joy in running a benchmark and seeing the allocation column read 0 B. Not "reduced." Not "acceptable." Zero. The garbage collector has nothing to do. The memory doesn't move. The result is just... there, in the buffer, where you put it.

Sometimes the best code is the code that doesn't happen.

ZaString is available on GitHub and NuGet.

Generic Methods Coming to Go

Mon, 27 Apr 2026 00:00:00 GMT

Generic Methods Coming to Go

"We do not anticipate that Go will ever add generic methods."

— Go FAQ, for roughly a decade.

That line sat in the FAQ like a monument. It wasn't a "not yet." It was a "never." And for years, every time someone opened an issue asking for type parameters on methods, that quote was the mic drop that ended the conversation.

Then, in January 2026, Robert Griesemer opened proposal #77273: "Generic methods for concrete types." The label was added: Proposal-Accepted.

Before you get too excited: this is an accepted proposal, not a shipped feature. You cannot use this in any Go release today. That said, the path forward is clear, and understanding what's coming, and what isn't, is worth your time now.

The Status Quo: The Anti-Pattern We All Live With

Since Go 1.18 gave us generics, there's been a frustrating gap. You can write generic functions and generic types, but you can't combine them into generic methods. The method set of a generic type can only contain non-generic methods.

This means many post-1.18 Go codebases have functions like these scattered around:

type Cache[K comparable, V any] struct {
    items map[K]V
}

// This is fine: non-generic method on a generic type.
func (c *Cache[K, V]) Get(key K) (V, bool) {
    v, ok := c.items[key]
    return v, ok
}

// But you can't do this:
// func (c *Cache[K, V]) Transform[T any](fn func(V) T) []T { ... }

// So you end up with this:
func Transform[K comparable, V any, T any](c *Cache[K, V], fn func(V) T) []T {
    result := make([]T, 0, len(c.items))
    for _, v := range c.items {
        result = append(result, fn(v))
    }
    return result
}

That last function takes the receiver as its first argument. It's not a method. It doesn't show up on autocomplete, can't be chained, and breaks the left-to-right reading flow that methods provide. cache.Transform(...) reads naturally; Transform(cache, ...) reads like inside-out code. This isn't just aesthetics: it makes APIs harder to discover, document, and compose.

Why This Became Possible

The reason generic methods were blocked for so long wasn't technical laziness. It was a genuine design problem.

The original Type Parameters proposal discussed generic methods and rejected them. The reasoning went like this: if methods can have type parameters, then interface methods should too. And generic interface methods are genuinely hard. They break type erasure, create dynamic dispatch problems, and don't play well with reflection. So the whole idea was shelved.

What changed? Griesemer's insight, captured in the proposal, is disarmingly simple:

"Concrete methods are a language feature that is useful in itself, irrespective of interfaces."

Methods do two things: they implement interfaces, and they organize code on a type. The Go team had been conflating these two roles. Once you separate them, the problem dissolves:

// This is what's being added (concrete type, generic method):
func (s *MySlice[E]) Map[T any](fn func(E) T) []T

// This is NOT being added (interface method with type parameters):
type Reader[E any] interface {
    Read[E]()  // Still impossible
}

The first one is straightforward. The compiler knows the concrete type at compile time, so it can statically resolve the method using the existing type argument mechanism. No dynamic dispatch, no reflection, no runtime complexity.

The second one remains unsolved because Reader[E] doesn't name a single method. It names a family of methods, and Go's interface satisfaction model can't handle that.

This decoupling is the key insight. Generic methods on concrete types were always implementable. They were just held hostage to the harder problem of generic interface methods.

The history matters here. Issue #49085, opened in October 2021 by the community, accumulated over 900 positive reactions. It was the primary pressure point. Issue #50981 followed in February 2022 with simpler motivating examples. Both were effectively deferred. Until now.

The Syntax

The grammar change is minimal. The method declaration production gains an optional type parameter list:

MethodDecl = "func" Receiver MethodName [ TypeParameters ] Signature [ FunctionBody ]

In practice:

type Stack[T any] struct {
    items []T
}

// Regular method on a generic type (valid since Go 1.18):
func (s *Stack[T]) Push(items ...T) {
    s.items = append(s.items, items...)
}

// Generic method with its own type parameter: new!
func (s *Stack[T]) Map[U any](fn func(T) U) *Stack[U] {
    result := &Stack[U]{}
    for _, item := range s.items {
        result.Push(fn(item))
    }
    return result
}

// Calling it:
nums := &Stack[int]{}
nums.Push(1, 2, 3)
strs := nums.Map[string](func(n int) string { return strconv.Itoa(n) })

// Type inference works:
strs = nums.Map(strconv.Itoa)

There's a subtler grammar change too: TypeArgs moves from Operand to PrimaryExpr. This is what makes expr.Method[T](args) parseable. The type arguments attach to the method call expression, not just to identifiers.

Method expressions and method values work as you'd expect, with one catch:

// Method expression: Stack[int].Map produces a generic function
// with signature [U any](*Stack[int], func(int) U) *Stack[U]

// Method value: s.Map produces [U any] func(func(int) U) *Stack[U]

// But this is INVALID: you must instantiate the type first:
// Stack.Map[U any]  // ERROR: Stack is not instantiated

The Boundary: What This Does NOT Do

After seeing the syntax, the natural next question is: "Can I use this in interfaces?"

No.

Generic interface methods are explicitly out of scope. This is a feature that solves the right problem first. The interface problem is a different, harder one, and leaving it unsolved here doesn't preclude a future proposal.

The Case Against

I'd be dishonest if I didn't engage with the real objections.

"Go said never. Why should we trust them now?"

The FAQ reversal is uncomfortable. When language designers draw a line, reversing it risks eroding trust. Some developers chose Go because of its restraint. Adding features incrementally is one thing; reversing explicit "never" statements is another.

My read: the original resistance was principled but overly conservative. The reasoning was "generic methods require generic interface methods," and that premise turned out to be false. Changing course when you discover a false premise isn't flip-flopping. It's good engineering. The Go team has a track record of shipping minimal, well-thought-out language changes. I'd rather they correct a mistake than defend it out of pride.

"This adds complexity for marginal benefit."

This is the strongest objection. Go's complexity budget is real. Every new feature makes the language harder to learn, harder to tool, and harder to reason about. Is the convenience of cache.Transform(...) worth the cognitive cost of another grammar production?

I think so, for two reasons. First, the implementation cost is surprisingly low. The compiler can statically resolve generic methods on concrete types without any runtime changes. This isn't adding a new dispatch mechanism; it's removing a parsing restriction. Second, the ergonomic benefit is disproportionate. I keep running into the (receiver, typeParam) pattern in post-1.18 Go codebases. Fixing it at the language level eliminates a class of API awkwardness that I encounter daily.

"This opens the door to generic interface methods, and then we're Java."

Slippery slope arguments are easy to make. But the proposal explicitly addresses this: "It also doesn't preclude the implementation of generic interface methods at some point, should we find an acceptable implementation solution." This is honest. It doesn't promise never, and it doesn't promise soon. It leaves the door open for a future proposal that makes its own case.

I'd argue the opposite slippery slope: if Go doesn't ship generic methods on concrete types, the pressure for full generic interface methods only grows. Shipping this targeted feature might actually reduce demand for the more complex one by solving the practical pain point.

Practical Examples

Let me show you what this actually looks like in code you might write.

Type-Safe Builders

type Query[T any] struct {
    filters []func(T) bool
    limit   int
}

func NewQuery[T any]() *Query[T] {
    return &Query[T]{limit: -1}
}

func (q *Query[T]) Where(pred func(T) bool) *Query[T] {
    q.filters = append(q.filters, pred)
    return q
}

func (q *Query[T]) Limit(n int) *Query[T] {
    q.limit = n
    return q
}

func (q *Query[T]) Run(items []T) []T {
    // Applies all filters and returns matching items (implementation omitted)
}

// Usage: notice how the type parameter flows through the chain:
type User struct {
    Name string
    Age  int
}

users := []User{{"Alice", 30}, {"Bob", 25}, {"Charlie", 30}}
activeAdults := NewQuery[User]().
    Where(func(u User) bool { return u.Age >= 18 }).
    Where(func(u User) bool { return u.Name != "" }).
    Limit(10).
    Run(users)

Today, you'd need to make Where a package-level function that takes the query as its first argument. The chaining breaks. The type parameter doesn't flow naturally. With generic methods, the builder pattern becomes first-class.

GroupBy: A Method-Level Type Parameter

Here's a case where the method genuinely needs its own type parameter, unrelated to the receiver's:

type Table[R any] struct {
    rows []R
}

func (t *Table[R]) GroupBy[K comparable](keyFn func(R) K) map[K][]R {
    groups := make(map[K][]R)
    for _, row := range t.rows {
        k := keyFn(row)
        groups[k] = append(groups[k], row)
    }
    return groups
}

// Usage:
orders := &Table[Order]{rows: allOrders}
byCustomer := orders.GroupBy[string](func(o Order) string { return o.CustomerID })

The type parameter K belongs to GroupBy, not to Table. This is the kind of thing you simply cannot express as a method today. With generic methods, it becomes natural.

The `List[E].Format[F]` Pattern

This is the motivating example from the proposal itself: a method that introduces its own type parameter independent of the type's parameter:

type List[E any] struct {
    elements []E
}

func (l *List[E]) Format[F any](formatter func(E) F) []F {
    result := make([]F, len(l.elements))
    for i, e := range l.elements {
        result[i] = formatter(e)
    }
    return result
}

// E is string, F is int: two independent type parameters
names := &List[string]{elements: []string{"hello", "world"}}
lengths := names.Format[int](func(s string) int { return len(s) })
// lengths == []int{5, 5}

This is the case that's truly impossible to express cleanly today. The method needs a type parameter (F) that's unrelated to the type's parameter (E). A package-level function can do it, but then you lose the method call syntax entirely.

How to Prepare Today

You can't use generic methods yet, but you can write code that migrates trivially when they land.

1. Wrap your (receiver, typeParam) functions on types.

If you have:

func Transform[K comparable, V any, T any](c *Cache[K, V], fn func(V) T) []T {
    // ...
}

Keep the type as the first argument. When generic methods ship, the migration is mechanical: move the function onto the type, drop the receiver parameter, done. The function body doesn't change.

2. Stop reaching for interface{} workarounds.

If you're using empty interfaces to work around the inability to write generic methods, stop. Design your types with proper type parameters now. The code will work today with package-level functions and become cleaner tomorrow with methods.

3. Name your helper functions after the future method.

If you plan to add a Transform method to your Cache type, name the package-level function Transform, not TransformCache or ApplyToCache. Mechanical migration depends on name alignment.

When Will It Land?

Here's where things stand:

Parser: Already handles type parameters on methods (parses them, currently rejects with an error). The change to accept them is trivial.
Type checker: Needs modifications to remove the current restriction and handle method expressions/values with type parameters. In progress.
Compiler backend: Generic method calls on concrete types can be statically resolved and rewritten as generic function calls. This is well-understood: no new dispatch mechanism needed.
Export/import format: This is the hardest part. The serialized format for compiled packages needs to represent generic methods, and changing it affects tooling across the ecosystem (gopls, vulncheck, build cache compatibility). This is what determines the timeline.

The x/tools repository already has a tracking issue (#77549) for generic method support across the toolchain. The Go team typically allows one or two release cycles for tooling to catch up after a language change.

Go 1.27 or 1.28 feels realistic. This is a well-scoped change with a clear implementation path, not a years-long research project like the original generics design.

My Take

I'm genuinely excited about this, and not just because I get to delete a category of awkward functions from my codebases.

What I appreciate about this proposal is the discipline. The Go team could have swung for the fences and tried to solve generic interface methods too. Instead, they identified the part that's well-understood, implementable, and high-value, and they scoped just that. That's the kind of restraint that made Go worth using in the first place.

What I'd warn against: don't use generic methods as a hammer. If a package-level generic function reads clearly, keep it as a function. Methods aren't inherently better. They're better when they improve API discoverability and composability. Use the feature where it earns its complexity budget, not everywhere it's syntactically legal.

Go's relationship with generics has been cautious, sometimes frustratingly so. But this proposal feels like the right next step: small, principled, and immediately useful. The FAQ was wrong. That's okay. Recognizing a mistake and correcting it is better than defending it indefinitely.

I Write, I Code, I Explore — Why Verbs, Not Nouns

Sun, 26 Apr 2026 00:00:00 GMT

I was listening to one of my favorite episodes of the ChessMood Podcast from my friend GM Avetik, How you can perform at your peak like top athletes do | Todd Herman on ChessMood Podcast, and around 01h07 one small idea from it kept following me around.

He said he prefers verbs to nouns when he talks about himself.

Not "I'm a coach," but "I coach." Not "I'm an entrepreneur," but "I build." Not "I'm an author," but "I write."

The more I sat with that, the more it explained a tension I've felt for years.

There are days when "developer" feels too heavy a word for me. I can ship something useful in the morning, get stuck on a dumb bug in the afternoon, and end the day feeling like I should hand back the badge. The same thing happens with writing. If I tell myself "I am a writer," every weak paragraph feels like evidence against me. But if I say "I write," the whole thing softens. I wrote today. Some of it was good. Some of it wasn't. Both can be true.

That's what I like about verbs: they leave room for motion.

Nouns can be useful. They help other people understand roughly where you live in the world. But they also freeze you at strange moments. They can turn yesterday's result into today's identity. "I am this kind of person." "I am this level." "I am this role." And once that sentence hardens, you start protecting it. You hesitate to be bad at something new because it might threaten the name you've chosen for yourself.

Verbs don't ask for that kind of defense. Verbs ask for practice.

"I write" means I can write something sharp one day and clumsy the next. "I code" means I can build a feature, break something obvious, learn, fix it, and keep going. The verb doesn't collapse because the performance wasn't perfect. In a quiet way, it's a more merciful grammar.

What stayed with me most from Todd Herman's framing was that this changes across contexts. The verb that describes how I work is not always the same one that describes how I love people. In one part of life, maybe I build. In another, I listen. In another, I encourage. The noun tries to gather everything into one static identity. The verb asks a better question: what am I actually doing here?

That question feels closer to how life is really lived.

I don't want to be trapped by a title that only describes me on my best days. I'd rather use words that still fit on ordinary days. I wrote bad code today. I rewrote a paragraph five times. I explored an idea without being sure it would go anywhere. None of that cancels the practice. It is the practice.

And there is something quietly reassuring in that. If a title disappears, the verb often remains. If one day I stop calling myself a developer, I can still build. If I stop calling myself a writer, I can still write. The label may change with the season. The work can keep moving.

So I find myself trusting verbs more.

Not because nouns are evil, and not because language alone can save us, but because verbs keep me closer to the living part of things. They remind me that I am not a finished category. I am a person in motion, practicing.

Go Pipeline Pattern: Turning Streams into Useful Data

Fri, 24 Apr 2026 00:00:00 GMT

Pipeline Pattern in Go

Introduction

Sometimes, the hard part of concurrent programming is not making things run in parallel. The hard part is keeping the flow of data understandable.

A pipeline is a simple way to do that. Instead of putting every responsibility inside one large loop, you split the work into small stages. Each stage receives values from a channel, does one transformation, and sends the result to the next stage.

source -> parse -> filter -> enrich -> sink

In Go, this pattern feels natural because goroutines and channels already give us the building blocks. A generator can create the initial stream, each pipeline stage can transform it, and a final consumer can collect or print the result.

This article continues the Go Patterns series after Producer-Consumer, Generator, and Worker Pool. The goal is not to build a framework. The goal is to learn how to structure data processing without turning one function into a pile of responsibilities.

When to Use

Use the Pipeline Pattern when data moves through multiple steps:

Processing log streams
Reading, validating, and transforming CSV rows
Cleaning API responses before storing them
Building small ETL-like flows
Splitting parsing, filtering, enrichment, and reporting into separate responsibilities

The pattern is especially useful when each step can be described as:

take a stream of values, transform or filter it, and return another stream of values.

Why Use It

Pipelines are useful because they keep each stage focused.

Separation of concerns: parsing, filtering, and reporting do not live in the same loop
Composability: stages can be reused and reordered
Natural backpressure: channels slow down upstream stages when downstream stages cannot keep up
Testability: each stage can be tested with a small input channel
Readable concurrency: the data flow is visible from the stage composition

The pattern is not magic. It is mostly discipline: one stage, one responsibility.

How It Works

A pipeline stage usually has this shape:

func stage(in <-chan Input) <-chan Output {
    out := make(chan Output)

    go func() {
        defer close(out)

        for value := range in {
            out <- transform(value)
        }
    }()

    return out
}

The stage receives a read-only channel, creates its own output channel, starts a goroutine, then closes the output channel when the input channel is exhausted.

This gives us a chain:

raw := source()
parsed := parse(raw)
filtered := filter(parsed)
enriched := enrich(filtered)
sink(enriched)

Simple Example

Before looking at logs, let's start with a tiny pipeline:

numbers -> square -> keep even -> print

package main

import "fmt"

func numbers(max int) <-chan int {
    out := make(chan int)

    go func() {
        defer close(out)

        for i := 1; i <= max; i++ {
            out <- i
        }
    }()

    return out
}

func square(in <-chan int) <-chan int {
    out := make(chan int)

    go func() {
        defer close(out)

        for n := range in {
            out <- n * n
        }
    }()

    return out
}

func keepEven(in <-chan int) <-chan int {
    out := make(chan int)

    go func() {
        defer close(out)

        for n := range in {
            if n%2 == 0 {
                out <- n
            }
        }
    }()

    return out
}

func main() {
    values := numbers(10)
    squared := square(values)
    evenSquares := keepEven(squared)

    for n := range evenSquares {
        fmt.Println(n)
    }
}

Each function owns one small part of the work. numbers produces values, square transforms them, keepEven filters them, and main consumes the final stream.

That is the pipeline pattern in its smallest useful form.

Real-World Example: Log Processing Pipeline

Now let's use a more realistic example.

Imagine we receive raw log lines and want to turn them into useful information. We can model that as a pipeline:

raw log lines -> parse logs -> filter errors -> enrich logs -> print report

For a complete runnable version, this example needs:

import (
    "fmt"
    "strings"
    "time"
)

First, define the data we want to pass between stages:

type RawLog string

type LogEntry struct {
    Timestamp time.Time
    Level     string
    Service   string
    Message   string
    Alert     bool
}

The source stage sends raw log lines:

func logSource(lines []string) <-chan RawLog {
    out := make(chan RawLog)

    go func() {
        defer close(out)

        for _, line := range lines {
            out <- RawLog(line)
        }
    }()

    return out
}

The parser turns each raw line into a structured LogEntry:

func parseLogs(in <-chan RawLog) <-chan LogEntry {
    out := make(chan LogEntry)

    go func() {
        defer close(out)

        for raw := range in {
            parts := strings.SplitN(string(raw), "|", 4)
            if len(parts) != 4 {
                continue
            }

            timestamp, err := time.Parse(time.RFC3339, parts[0])
            if err != nil {
                continue
            }

            out <- LogEntry{
                Timestamp: timestamp,
                Level:     parts[1],
                Service:   parts[2],
                Message:   parts[3],
            }
        }
    }()

    return out
}

The filter stage keeps only errors:

func filterErrors(in <-chan LogEntry) <-chan LogEntry {
    out := make(chan LogEntry)

    go func() {
        defer close(out)

        for entry := range in {
            if entry.Level == "ERROR" {
                out <- entry
            }
        }
    }()

    return out
}

The enrichment stage adds a small piece of derived information:

func enrichLogs(in <-chan LogEntry) <-chan LogEntry {
    out := make(chan LogEntry)

    go func() {
        defer close(out)

        for entry := range in {
            entry.Alert = entry.Service == "payment" || entry.Service == "auth"
            out <- entry
        }
    }()

    return out
}

Finally, the sink consumes the enriched entries:

func printReport(in <-chan LogEntry) {
    for entry := range in {
        alert := ""
        if entry.Alert {
            alert = " [ALERT]"
        }

        fmt.Printf("%s %s: %s%s\n", entry.Service, entry.Level, entry.Message, alert)
    }
}

The full pipeline becomes very readable:

func main() {
    lines := []string{
        "2026-04-24T10:00:00Z|INFO|api|request completed",
        "2026-04-24T10:00:01Z|ERROR|payment|card authorization failed",
        "2026-04-24T10:00:02Z|ERROR|worker|job timeout",
        "2026-04-24T10:00:03Z|ERROR|auth|invalid token",
    }

    raw := logSource(lines)
    parsed := parseLogs(raw)
    errors := filterErrors(parsed)
    enriched := enrichLogs(errors)

    printReport(enriched)
}

The important part is not the log format. The important part is that the flow is explicit.

Each stage can be read, tested, and replaced independently.

Error Handling

The log parser above silently skips invalid lines. That keeps the example small, but it is not always what you want in production.

Two common approaches are:

Send errors to a separate error channel
Pass a result type through the pipeline

For example:

type LogResult struct {
    Entry LogEntry
    Err   error
}

This makes failures explicit without panicking inside a goroutine. It also lets the final consumer decide whether to log, count, retry, or ignore invalid records.

Cancellation

The examples above work for finite inputs. For long-running pipelines, use context.Context so every stage can stop when the caller is done.

The shape usually looks like this:

func parseLogs(ctx context.Context, in <-chan RawLog) <-chan LogEntry {
    out := make(chan LogEntry)

    go func() {
        defer close(out)

        for {
            select {
            case <-ctx.Done():
                return
            case raw, ok := <-in:
                if !ok {
                    return
                }

                entry, ok := parseLog(raw)
                if ok {
                    out <- entry
                }
            }
        }
    }()

    return out
}

Without cancellation, a pipeline that reads from a never-ending source can leak goroutines when the consumer stops early.

Best Practices and Pitfalls

Best Practices:

Keep each stage focused on one responsibility
Return receive-only channels (<-chan T) from stages
Close the output channel from the goroutine that writes to it
Use context.Context for long-running or cancellable pipelines
Test each stage independently with small input channels

Pitfalls:

Forgetting to close output channels
Stopping early without cancelling upstream stages
Creating too many tiny stages that hide simple logic
Mixing parsing, filtering, enrichment, and reporting in one function
Assuming ordering will stay the same if you later parallelize a stage

Related Patterns

Generator Pattern: Creates the initial stream of values
Producer-Consumer Pattern: Separates production from consumption
Worker Pool Pattern: Parallelizes expensive stages
Fan-Out/Fan-In Pattern: Distributes one stage across multiple workers and merges the results

Summary

The Pipeline Pattern is one of the most readable ways to structure data processing in Go. It lets you split a flow into small stages, connect them with channels, and keep each responsibility isolated.

It works well when data naturally moves through a sequence: read, parse, filter, enrich, report.

The pattern is also a bridge to more advanced concurrency designs. Once one stage becomes too slow, you can combine a pipeline with a Worker Pool or Fan-Out/Fan-In to parallelize only that part of the flow.

Testing Pipelines

The best pipeline tests treat each stage as a pure function: send a channel in, get a channel out. The goroutines are implementation details. I wrote about that approach in TDD Isn't About Bugs — It's Your Permission to Refactor.

Series Navigation

This article is part of the Go Patterns series:

Previous: Mastering the Worker Pool Pattern in Go
Series: Go Patterns

If you want to experiment with the code examples, you can find them on my GitHub repository.

Flexible Approaches to Worker Pools in Go

Thu, 12 Dec 2024 00:00:00 GMT

Flexible Approaches to Worker Pools in Go

Introduction

While the standard Worker Pool pattern is powerful, Go's flexibility allows for alternative approaches to concurrent task processing. This article explores the Shared Semaphore method and discusses the use of third-party libraries for managing concurrency in Go applications.

Shared Semaphore Method

The Shared Semaphore method uses a buffered channel as a semaphore to limit concurrency across various parts of an application.

package main

import (
	"fmt"
	"sync"
	"time"
)

func processWithSemaphore(tasks []int, maxConcurrency int) {
	sem := make(chan struct{}, maxConcurrency)
	var wg sync.WaitGroup

	for _, task := range tasks {
		wg.Add(1)
		sem <- struct{}{} // Acquire semaphore
		go func(task int) {
			defer wg.Done()
			defer func() { <-sem }() // Release semaphore
			processTask(task)
		}(task)
	}

	wg.Wait()
}

func processTask(task int) {
	fmt.Printf("Processing task %d\n", task)
	time.Sleep(time.Second) // Simulate work
	fmt.Printf("Completed task %d\n", task)
}

func main() {
	tasks := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
	processWithSemaphore(tasks, 3)
}

Advantages:

Simple and lightweight implementation
Scales to zero when not in use
Can be used in multiple places without increasing complexity

Use Case:

Ideal for scenarios where you need to limit concurrency across various parts of your application without maintaining a persistent worker pool.

Third-Party Libraries

While implementing your own worker pool is often the best approach, there are some third-party libraries that can be useful in certain situations:

Ants: A high-performance goroutine pool in Go, providing features like automatic scaling and reuse of goroutines.
sourcegraph/conc: A package for structured concurrency in Go, offering higher-level abstractions for common concurrency patterns.

These libraries can be beneficial when you need advanced features or when working on complex concurrent systems. However, it's important to note that in approximately 90% of cases, maintaining your own worker pool implementation is preferable. This approach gives you more control, better understanding of your concurrency model, and avoids unnecessary dependencies.

Choosing the Right Approach

Consider the following factors when deciding on your concurrency approach:

Assess your concurrency needs:

Do you need to limit concurrency across multiple parts of your application?
Is your workload consistent or variable?

Evaluate your task characteristics:

Are tasks short-lived or long-running?
Do you need fine-grained control over task execution?

Consider your application architecture:

Is simplicity a priority?
Do you need to scale workers dynamically?

Decision tree:

If (need to limit concurrency in specific sections) AND (variable workload): Use Shared Semaphore
Else if (consistent workload) AND (need fine-grained control): Use Standard Worker Pool
Else if (need advanced features) AND (complexity is justified): Consider third-party libraries like Ants or sourcegraph/conc
Else: Implement and maintain your own worker pool

This decision process helps guide your choice based on your specific context and requirements, ensuring you choose the most appropriate concurrency pattern for your needs while prioritizing simplicity and control in most cases.

Summary

While the standard Worker Pool pattern is versatile, Go's concurrency model allows for flexible approaches like the Shared Semaphore method. This alternative can be particularly useful for applications with varying concurrency needs across different components. Third-party libraries offer advanced features but should be used judiciously, as maintaining your own worker pool often provides the best balance of control and simplicity.

Disclaimer

This article provides an overview of flexible approaches to worker pools in Go. While these patterns are powerful, it's important to consider the specific needs of your application when implementing them. For production use, additional error handling and optimizations may be necessary.

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.

Mastering the Worker Pool Pattern in Go

Tue, 10 Dec 2024 00:00:00 GMT

Worker Pool Pattern in Go

Introduction

The Worker Pool pattern is a fundamental concurrency design in Go that efficiently manages a pool of worker goroutines to process tasks from a shared queue. This pattern excels at handling a large number of independent tasks concurrently while maintaining precise control over system resources and performance.

When to Use

Processing a large number of independent tasks that can be parallelized
Limiting the number of concurrent operations to prevent resource exhaustion
Balancing workload across multiple processors or cores
Managing CPU-bound or I/O-bound tasks efficiently
Handling batch processing operations with controlled parallelism

Why to Use

Controls resource utilization by maintaining a fixed number of workers
Improves performance through efficient parallel processing
Prevents system overload by limiting concurrent operations
Enhances application scalability and throughput
Maintains predictable resource usage patterns

How it Works

The Worker Pool pattern consists of three essential components:

A pool of worker goroutines that process tasks concurrently
A job queue (input channel) that holds pending tasks
A results queue (output channel) that collects processed results

Workers continuously pull tasks from the job queue, process them independently, and send results to the results queue. This design ensures efficient task distribution and controlled concurrency.

Simple Example

func worker(id int, jobs <-chan int, results chan<- int) {
    for job := range jobs {
        fmt.Printf("Worker %d processing job %d\n", id, job)
        time.Sleep(time.Second) // Simulating work
        results <- job * 2
    }
}

func main() {
    const numJobs = 5
    const numWorkers = 3

    jobs := make(chan int, numJobs)
    results := make(chan int, numJobs)

    // Start worker pool
    for w := 1; w <= numWorkers; w++ {
        go worker(w, jobs, results)
    }

    // Send jobs to the workers
    for j := 1; j <= numJobs; j++ {
        jobs <- j
    }
    close(jobs)

    // Collect and print results
    for a := 1; a <= numJobs; a++ {
        result := <-results
        fmt.Printf("Job result: %d\n", result)
    }
}

Real-World Example: Image Processor

Let's consider a scenario where we need to process multiple images concurrently:

type Job struct {
    ID       int
    ImageURL string
    Size     int
}

type Result struct {
    JobID     int
    ImageURL  string
    NewSize   int
    Error     error
    TimeSpent time.Duration
}

func imageProcessor(id int, jobs <-chan Job, results chan<- Result) {
    for job := range jobs {
        startTime := time.Now()

        fmt.Printf("Worker %d processing image %d from %s\n", id, job.ID, job.ImageURL)

        result := Result{
            JobID:    job.ID,
            ImageURL: job.ImageURL,
            NewSize:  job.Size,
        }

        // Simulate image processing with realistic steps
        err := processImage(job)
        if err != nil {
            result.Error = err
            results <- result
            continue
        }

        result.TimeSpent = time.Since(startTime)
        results <- result
    }
}

func processImage(job Job) error {
    // Simulate various image processing steps
    time.Sleep(time.Duration(rand.Intn(500)) * time.Millisecond)

    // Simulate potential errors
    if rand.Float32() < 0.1 {
        return fmt.Errorf("failed to process image %d: simulation error", job.ID)
    }

    return nil
}

func main() {
    numCPU := runtime.NumCPU()
    runtime.GOMAXPROCS(numCPU)
    numWorkers := numCPU * 2 // Use 2 workers per CPU core
    const numJobs = 10

    jobs := make(chan Job, numJobs)
    results := make(chan Result, numJobs)

    // Initialize worker pool
    for w := 1; w <= numWorkers; w++ {
        go imageProcessor(w, jobs, results)
    }

    // Send image processing jobs
    for j := 1; j <= numJobs; j++ {
        jobs <- Job{
            ID:       j,
            ImageURL: fmt.Sprintf("https://example.com/image%d.jpg", j),
            Size:     100 * j, // Varying sizes
        }
    }
    close(jobs)

    // Collect and handle results
    for a := 1; a <= numJobs; a++ {
        result := <-results
        if result.Error != nil {
            fmt.Printf("Error processing image %d: %v\n", result.JobID, result.Error)
        } else {
            fmt.Printf("Successfully processed image %d to size %dpx in %v\n",
                result.JobID, result.NewSize, result.TimeSpent)
        }
    }
}

Best Practices and Pitfalls

Best Practices:

Always close the channel when generation is complete
Use buffered channels when appropriate to prevent blocking
Include monitoring and logging for production environments
Implement graceful shutdown mechanisms
Size your worker pool based on available system resources

Pitfalls:

Creating too many workers, leading to resource exhaustion
Not handling worker failures or panics
Forgetting to close channels properly
Missing timeout mechanisms for long-running tasks
Inefficient job distribution strategies

Summary

The Worker Pool pattern is a powerful tool in Go's concurrency toolkit, offering a balanced approach to parallel processing. By maintaining a fixed number of workers, it prevents resource exhaustion while maximizing throughput. The pattern is particularly valuable in real-world scenarios such as image processing, batch operations, and API request handling, where controlled concurrent processing is essential for optimal performance and resource utilization.

Disclaimer

This article provides an introduction to the Worker Pool pattern in Go. While the pattern is powerful, it's important to consider the specific needs of your application when implementing it. For production use, additional error handling and optimizations may be necessary.

Testing Worker Pools

If you test a worker pool, focus on the contract: jobs go in, results come out. The number of goroutines is an implementation detail. I wrote about that in TDD Isn't About Bugs — It's Your Permission to Refactor.

Series Navigation

This article is part of the Go Patterns series:

Previous: Mastering the Generator Pattern in Go
Next: Go Pipeline Pattern: Turning Streams into Useful Data
Series: Go Patterns

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.

Advanced Generator Pattern: Consuming and Testing Data Streams

Sun, 08 Dec 2024 00:00:00 GMT

Advanced Generator Pattern: Consuming and Testing Streams

:::warning[Difficulty Level] Advanced :::

Introduction

Expanding on our previous discussions of the Generator pattern, we'll explore two advanced applications: consuming large datasets lazily and simulating data streams for testing. These techniques are crucial for efficient data processing and robust application testing.

When to Use

Processing large datasets that don't fit in memory
Simulating data sources for testing
Implementing ETL (Extract, Transform, Load) processes
Creating reproducible test scenarios for data processing pipelines

Why to Use

Memory Efficiency: Process large datasets without loading everything into memory
Testability: Create controlled environments for testing data processing logic
Flexibility: Easily switch between real and simulated data sources
Reproducibility: Generate consistent test cases for data processing scenarios

How it Works

Create generator functions that yield data items one at a time
Use channels to stream data from the source to the consumer
Implement lazy loading for large datasets
Create mock data generators for testing scenarios

Example 1: Lazy Loading of Large Datasets

type DataItem struct {
    ID   int
    Data string
}

// lazyDataLoader simulates loading a large dataset lazily
func lazyDataLoader(filePath string) <-chan DataItem {
    out := make(chan DataItem)
    go func() {
        defer close(out)
        // Simulate opening a large file
        fmt.Printf("Opening file: %s\n", filePath)

        // Simulate reading the file line by line
        for i := 0; i < 1000000; i++ {
            // Simulate processing delay for each item
            time.Sleep(1 * time.Millisecond)
            out <- DataItem{
                ID:   i + 1,
                Data: fmt.Sprintf("Data from line %d", i+1),
            }
            if i%100000 == 0 {
                fmt.Printf("Processed %d items\n", i)
            }
        }
    }()
    return out
}

func processData(data <-chan DataItem) {
    for item := range data {
        // Simulate data processing
        processedData := fmt.Sprintf("Processed: %s (ID: %d)", item.Data, item.ID)
        fmt.Println(processedData)
    }
}

func main() {
    dataStream := lazyDataLoader("large_dataset.txt")
    processData(dataStream)
}

This example demonstrates lazy loading of a large dataset, processing items one at a time without loading the entire dataset into memory.

Example 2: Simulating Data Streams for Testing

type DataItem struct {
    ID   int
    Data string
}

// mockDataStream simulates a data source (e.g., a file, queue, or network stream)
func mockDataStream(count int) <-chan DataItem {
    out := make(chan DataItem)
    go func() {
       defer close(out)
       for i := 0; i < count; i++ {
          // Simulate reading from a data source
          time.Sleep(100 * time.Millisecond)
          out <- DataItem{
             ID:   i + 1,
             Data: fmt.Sprintf("Data-%d", i+1),
          }
       }
    }()
    return out
}

// dataGenerator consumes the mock stream and yields processed data
func dataGenerator(stream <-chan DataItem) <-chan string {
    out := make(chan string)
    go func() {
       defer close(out)
       for item := range stream {
          // Process the data item
          processedData := fmt.Sprintf("Processed: %s (ID: %d)", item.Data, item.ID)
          out <- processedData
       }
    }()
    return out
}

type StreamGenerator struct{}

func (g StreamGenerator) Execute() {
    // Create a mock data stream
    dataStream := mockDataStream(10)

    // Create a generator to process the stream
    processedDataGen := dataGenerator(dataStream)

    // Consume and print the processed data
    for data := range processedDataGen {
       fmt.Println(data)
    }
}

This example demonstrates a more structured approach to using the Generator pattern for testing data processing pipelines:

mockDataStream simulates a data source by generating items with controlled timing
dataGenerator shows how to process a stream of data items and transform them
The StreamGenerator type provides a clean interface for executing the pipeline and can be replaced with real data sources in production using DI (Dependency Injection)
Each stage of the pipeline is clearly separated and testable

Best Practices and Pitfalls

Best Practices:

Use buffered channels for improved performance when processing large streams
Implement timeout mechanisms for long-running operations
Use the context package for cancellation in long-running generators
Create configurable mock generators for diverse test scenarios

Pitfalls:

Not handling errors or edge cases in data generation
Overlooking resource cleanup in generators (e.g., closing file handles)
Creating overly complex mock generators that don't reflect real-world scenarios
Ignoring performance implications in lazy loading implementations

Summary

The Generator pattern proves invaluable for both consuming large datasets efficiently and creating robust test environments for data processing logic. By leveraging Go's concurrency features, we can create flexible, memory-efficient, and testable data processing pipelines that can handle real-world scenarios and simulated test cases alike.

Disclaimer

While these examples demonstrate the power of the Generator pattern for data processing and testing, real-world implementations may require additional error handling, resource management, and optimizations. Always consider the specific requirements and constraints of your application when applying these patterns.

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.

<div style="font-size: 0.875rem; color: color-mix(in oklch, var(--text-base) 75%, transparent);"> <span property="dct:title">Educational Go Patterns </span> by <a rel="cc:attributionURL dct:creator" property="cc:attributionName" href="https://corentings.dev">Corentin Giaufer Saubert</a> is licensed under <a href="https://creativecommons.org/licenses/by-nc-nd/4.0/?ref=chooser-v1" target="_blank" rel="license noopener noreferrer">Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International </a><br/> The code examples are licensed under the <a href="https://opensource.org/licenses/MIT" target="_blank" rel="license noopener noreferrer">MIT License</a>. The banner image has been created by (DALL·E) and is licensed under the same license as the article and other graphics. </div>

Advanced Generator Pattern in Go: Test Data Generation

Fri, 06 Dec 2024 00:00:00 GMT

Advanced Generator Pattern: Test Data for Web Services

Introduction

Building upon our previous exploration of the Generator pattern, let's dive into a more advanced real-world application: generating test data for a web service. This pattern is particularly useful for creating large datasets to stress test APIs or simulate high-load scenarios.

When to Use

Stress testing web services
Simulating high-load scenarios for databases
Creating diverse datasets for QA environments
Benchmarking system performance

Why to Use

Scalability: Easily generate large volumes of test data
Customization: Tailor data generation to specific test scenarios
Realism: Create data that closely mimics production patterns
Efficiency: Generate data on-the-fly, reducing storage needs

How it Works

Define structures representing your API's data models
Create generator functions for each data type
Combine generators to create complex, interrelated data sets
Use channels to stream generated data to consumers (e.g., API clients)

Advanced Example: E-commerce API Test Data Generator

type Product struct {
    ID    int
    Name  string
    Price float64
}

type Order struct {
    ID       int
    UserID   int
    Products []Product
    Total    float64
}

func productGenerator(count int) <-chan Product {
    out := make(chan Product)
    go func() {
        defer close(out)
        for i := 0; i < count; i++ {
            out <- Product{
                ID:    i + 1,
                Name:  fmt.Sprintf("Product-%d", i+1),
                Price: 10.0 + float64(i),
            }
        }
    }()
    return out
}

func orderGenerator(userCount, orderPerUser int, products <-chan Product) <-chan Order {
    out := make(chan Order)
    go func() {
        defer close(out)
        var orderID int
        for userID := 1; userID <= userCount; userID++ {
            for i := 0; i < orderPerUser; i++ {
                orderID++
                var orderProducts []Product
                var total float64
                for j := 0; j < rand.Intn(5)+1; j++ {
                    product := <-products
                    orderProducts = append(orderProducts, product)
                    total += product.Price
                }
                out <- Order{
                    ID:       orderID,
                    UserID:   userID,
                    Products: orderProducts,
                    Total:    total,
                }
            }
        }
    }()
    return out
}

func main() {
    productChan := productGenerator(1000)
    orderChan := orderGenerator(100, 5, productChan)

    // Simulate sending orders to an API
    for order := range orderChan {
        // In a real scenario, you'd send this to your API
        fmt.Printf("Sending order %d for user %d with total $%.2f\n", order.ID, order.UserID, order.Total)
    }
}

This example demonstrates a more complex use of the Generator pattern to create realistic test data for an e-commerce API. It generates products and orders, simulating a scenario where multiple users are placing orders with varying numbers of products.

Best Practices and Pitfalls

Best Practices:

Use buffered channels for improved performance when generating large datasets
Implement cancellation mechanisms for long-running generators
Consider using worker pools for parallel data generation in complex scenarios
Seed random number generators for reproducible test data

Pitfalls:

Generating more data than necessary, leading to increased test times
Not closing channels properly, causing goroutine leaks
Overlooking edge cases in data generation, leading to incomplete test coverage
Generating unrealistic data that doesn't reflect real-world scenarios

Summary

The advanced application of the Generator pattern for test data generation showcases its power in creating scalable, customizable, and efficient solutions for testing web services. By leveraging Go's concurrency features, we can create sophisticated data generation pipelines that closely mimic real-world scenarios, enabling thorough and realistic testing of our systems.

Disclaimer

This article expands on the Generator pattern with a focus on test data generation. While the example provided is more complex, it's still simplified for educational purposes. In real-world applications, additional considerations such as data variety, error handling, and integration with actual API endpoints would be necessary.

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.

Mastering the Generator Pattern in Go

Tue, 03 Dec 2024 00:00:00 GMT

Generator Pattern in Go

Introduction

The Generator pattern in Go is a powerful concurrency pattern used to create functions that produce a sequence of values. It leverages Go's goroutines and channels to generate data asynchronously, providing an elegant way to work with streams of data or implement iterators.

When to Use

Creating sequences of numbers or data
Implementing iterators
Processing streams of data
Generating test data
Simulating real-time data sources

Why to Use

Lazy Evaluation: Values are generated on-demand, saving memory
Encapsulation: Hides the complexity of data generation
Concurrency: Allows for asynchronous data production
Flexibility: Can generate infinite or finite sequences
Composability: Generators can be chained or combined easily

How it Works

A function creates and returns a channel
The function starts a goroutine that sends values through the channel
The caller receives values from the channel, typically using a for-range loop

Simple Example

func evenGenerator(max int) <-chan int {
    out := make(chan int)
    go func() {
        for i := 0; i <= max; i += 2 {
            out <- i
        }
        close(out)
    }()
    return out
}

func main() {
    for num := range evenGenerator(10) {
        fmt.Println(num)
    }
}

This example creates a generator that produces even numbers up to a specified maximum. The evenGenerator function returns a receive-only channel (<-chan int). It starts a goroutine that sends even numbers through the channel and closes it when done.

Real-World Example: Log Line Generator

Let's consider a scenario where we need to generate sample log lines for testing a log analysis system.

type LogEntry struct {
    Timestamp time.Time
    Level     string
    Message   string
}

func logGenerator(count int) <-chan LogEntry {
    out := make(chan LogEntry)
    go func() {
        levels := []string{"INFO", "WARNING", "ERROR"}
        messages := []string{
            "User logged in",
            "Failed login attempt",
            "Database connection lost",
            "API request received",
            "Cache miss",
        }
        for i := 0; i < count; i++ {
            out <- LogEntry{
                Timestamp: time.Now().Add(time.Duration(i) * time.Second),
                Level:     levels[rand.Intn(len(levels))],
                Message:   messages[rand.Intn(len(messages))],
            }
            time.Sleep(100 * time.Millisecond) // Simulate delay between log entries
        }
        close(out)
    }()
    return out
}

func main() {
    for entry := range logGenerator(5) {
        fmt.Printf("[%s] %s: %s\n", entry.Timestamp.Format(time.RFC3339), entry.Level, entry.Message)
    }
}

This generator creates a stream of log entries, simulating real-world log generation. It's useful for testing log processing systems, allowing developers to generate a controlled stream of diverse log entries.

Best Practices and Pitfalls

Best Practices:

Always close the channel when generation is complete
Use receive-only channels (<-chan) as return types
Consider using context for cancellation in long-running generators
Implement error handling for robust generators

Pitfalls:

Forgetting to close channels, leading to goroutine leaks
Creating infinite generators without proper termination conditions
Blocking indefinitely on channel operations without timeout mechanisms
Overusing generators for simple, finite sequences where slices might suffice

Related Patterns

Pipeline Pattern: Often used in conjunction with generators to process data streams
Fan-Out/Fan-In Pattern: Can distribute generator output to multiple consumers
Iterator Pattern: Generators can be seen as concurrent iterators

Summary

The Generator pattern in Go provides a powerful way to create sequences or streams of data asynchronously. By leveraging goroutines and channels, it offers lazy evaluation, encapsulation of complex logic, and seamless integration with Go's concurrency model. Whether you're working with infinite sequences, simulating data sources, or implementing iterators, the Generator pattern offers a flexible and efficient solution.

Disclaimer

This article provides an introduction to the Generator pattern in Go. While the pattern is powerful, it's important to consider the specific needs of your application when implementing it. For production use, additional error handling and optimizations may be necessary.

Testing Generators

Generators are easy to test when you treat them as contracts: send input, receive output. The goroutine is an implementation detail. I wrote about why that matters in TDD Isn't About Bugs — It's Your Permission to Refactor.

Series Navigation

This article is part of the Go Patterns series:

Previous: Understanding the Producer-Consumer Pattern in Go
Next: Mastering the Worker Pool Pattern in Go
Series: Go Patterns

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.

Producer-Consumer in Go: Beyond the Basics

Mon, 02 Dec 2024 00:00:00 GMT

Producer-Consumer in Go: Beyond the Basics

In our previous article, we explored the fundamentals of the Producer-Consumer pattern in Go. Today, we'll take it a step further by examining more practical scenarios and introducing buffered channels - a powerful feature that can significantly improve your concurrent applications.

Buffered Channels: A Game Changer

While regular channels provide immediate synchronization between producers and consumers, sometimes we need more flexibility. Buffered channels act like a small warehouse, temporarily storing items when producers are faster than consumers or when we want to batch process items.

Let's see how they work:

// Unbuffered channel - synchronous
ch := make(chan int)

// Buffered channel - can hold up to 5 items
bufferedCh := make(chan int, 5)

The key difference? With a buffered channel, producers can send up to 5 items without waiting for a consumer to receive them. This can significantly improve performance in certain scenarios.

Real-World Example: Log Processing System

Imagine building a log processing system for a busy web application. Logs come in rapidly during peak hours, but we want to process them in batches for efficiency. This is a perfect use case for buffered channels.

type LogEntry struct {
    Timestamp time.Time
    Level     string
    Message   string
}

func logGenerator(logs chan<- LogEntry) {
    // Simulate incoming logs
    for i := 0; i < 10 ; i++{
        log := LogEntry{
            Timestamp: time.Now(),
            Level:     []string{"INFO", "WARNING", "ERROR"}[rand.Intn(3)],
            Message:   fmt.Sprintf("Event #%d occurred", i),
        }
        logs <- log
        time.Sleep(100 * time.Millisecond) // Simulate varying log frequencies
    }
    close(logs)
}

func logProcessor(logs <-chan LogEntry) {
    batch := make([]LogEntry, 0, 3) // Process logs in batches of 3

    for log := range logs {
        batch = append(batch, log)

        if len(batch) == 3 {
            // Process batch
            processLogBatch(batch)
            batch = batch[:0] // Clear the batch
        }
    }

    // Process remaining logs
    if len(batch) > 0 {
        processLogBatch(batch)
    }
}

func processLogBatch(batch []LogEntry) {
    fmt.Println("Processing batch of logs:")
    for _, log := range batch {
        fmt.Printf("[%s] %s: %s\n",
            log.Timestamp.Format("15:04:05"),
            log.Level,
            log.Message)
    }
    fmt.Println("Batch processing complete\n")
}

func main() {
    // Buffer size of 5 to handle burst of logs
    logs := make(chan LogEntry, 5)

    go logGenerator(logs)
    logProcessor(logs)
}

When you run this program, you'll see output similar to this:

Processing batch of logs:
[14:23:45] INFO: Event #0 occurred
[14:23:45] WARNING: Event #1 occurred
[14:23:45] ERROR: Event #2 occurred
Batch processing complete

Processing batch of logs:
[14:23:46] INFO: Event #3 occurred
[14:23:46] WARNING: Event #4 occurred
[14:23:46] INFO: Event #5 occurred
Batch processing complete

Processing batch of logs:
[14:23:47] ERROR: Event #6 occurred
[14:23:47] INFO: Event #7 occurred
[14:23:47] WARNING: Event #8 occurred
Batch processing complete

Processing remaining logs:
[14:23:48] INFO: Event #9 occurred
Batch processing complete

Understanding the Benefits

This implementation showcases several advanced concepts:

Burst Handling: The buffered channel (size 5) can handle bursts of logs without blocking the producer, even if the consumer is busy processing a batch.
Batch Processing: Instead of processing each log immediately, we batch them for more efficient processing. This is common in real-world scenarios where batching can reduce database writes or API calls.
Graceful Shutdown: The system handles remaining logs before shutting down, ensuring no data is lost.
Flexible Processing: The batch size (3) is independent of the channel buffer size (5), allowing us to optimize both independently.

When to Use Buffered Channels

Buffered channels are particularly useful when:

Your producers and consumers work at different speeds
You want to batch process items for efficiency
You need to handle burst of data without blocking producers
You want to improve performance by reducing synchronization overhead

However, remember that buffered channels don't solve all problems. They can hide deadlocks and make it harder to reason about your program's behavior if used incorrectly.

Tips for Success

Choose buffer sizes carefully—too small negates the benefits, too large can hide problems
Monitor channel capacity in production using len(ch) and cap(ch)
Consider using contexts for cancellation and timeouts
Always handle the remaining items when shutting down

What's Next?

Now that you understand buffered channels and batch processing, you're ready to explore more advanced patterns like Fan-Out/Fan-In or the Pipeline pattern. Stay tuned for our next article where we'll dive into those topics!

Conclusion

The Producer-Consumer pattern becomes even more powerful when combined with buffered channels and batch processing. While staying true to Go's simplicity, these features allow us to build more efficient and resilient systems.

Remember: The key to mastering these patterns is practice. Try modifying the log processing example to handle different batch sizes or add error handling. The possibilities are endless!

Understanding the Producer-Consumer Pattern in Go

Sat, 30 Nov 2024 00:00:00 GMT

Producer-Consumer Pattern in Go

Introduction

The Producer-Consumer pattern is a fundamental concurrency pattern in Go that elegantly separates the production of data from its consumption. By using channels as intermediaries, this pattern creates a robust foundation for concurrent data processing.

When to Use

When you need to separate data generation from its processing
In scenarios where production and consumption rates differ
When scaling producers and consumers independently is desired
For managing workload distribution in concurrent systems

Why Use It

This pattern offers several compelling advantages:

Modularity: Clear separation between data production and consumption logic
Flexible Scaling: Easy to add more producers or consumers as needed
Buffer Management: Handles different processing rates naturally
Resource Control: Better management of system resources through controlled data flow

How it Works

The pattern consists of three main components:

Producers: Goroutines that generate data
Queue: A channel that acts as a buffer between producers and consumers
Consumers: Goroutines that process the data

Simple Example

func producer(ch chan<- int) {
    for i := 0; i < 5; i++ {
        ch <- i
        fmt.Printf("Produced: %d\n", i)
    }
    close(ch)
}

func consumer(ch <-chan int) {
    for num := range ch {
        fmt.Printf("Consumed: %d\n", num)
    }
}

func main() {
    ch := make(chan int)
    go producer(ch)
    consumer(ch)
}

Real-World Example: Web Scraper

type Page struct {
    URL     string
    Content string
}

func scraper(urls []string, pages chan<- Page) {
    for _, url := range urls {
        // Simulate web scraping
        time.Sleep(100 * time.Millisecond)
        pages <- Page{
            URL:     url,
            Content: fmt.Sprintf("Content from %s", url),
        }
    }
    close(pages)
}

func processor(pages <-chan Page, results chan<- string) {
    for page := range pages {
        // Simulate content processing
        time.Sleep(200 * time.Millisecond)
        results <- fmt.Sprintf("Processed %s: %s", page.URL, page.Content)
    }
    close(results)
}

func main() {
    urls := []string{"https://example1.com", "https://example2.com"}
    pages := make(chan Page)
    results := make(chan string)

    go scraper(urls, pages)
    go processor(pages, results)

    for result := range results {
        fmt.Println(result)
    }
}

Best Practices and Pitfalls

Best Practices:

Always use directional channel types (chan<- for send-only, <-chan for receive-only)
Consider buffered channels when producers and consumers work at different speeds
Implement proper error handling and propagation
Use context for cancellation when needed

Common Pitfalls:

Forgetting to close channels
Not handling backpressure when producers are faster than consumers
Creating deadlocks by improper channel management
Memory leaks from unclosed goroutines

Related Patterns

Pipeline Pattern: For multi-stage data processing
Worker Pool Pattern: For parallel task processing
Fan-Out/Fan-In Pattern: For distributed workload processing

Summary

The Producer-Consumer pattern is a cornerstone of concurrent programming in Go. Its simplicity and effectiveness make it an excellent starting point for learning concurrency patterns. By separating concerns and managing data flow through channels, it provides a clean and efficient way to handle concurrent data processing tasks.

Series Navigation

This article is part of the Go Patterns series:

Next: Mastering the Generator Pattern in Go
Series: Go Patterns

Disclaimer

This article is a simple introduction to the Producer-Consumer pattern in Go. For more advanced use cases and optimizations, consider exploring additional resources and best practices in concurrent programming. I may write more articles on this topic in the future, so stay tuned! 🚀 <br/> If you want to have a look at the code examples, you can find them on my GitHub repository.

My Experience at the Karen Asrian Memorial Tournament

Fri, 17 May 2024 00:00:00 GMT

Introduction

My recent trip to Armenia wasn't just a vacation—it was a chess pilgrimage! I participated in the prestigious Karen Asrian Memorial, a chess tournament that pushed me to my limits and left me with unforgettable memories.

The Tournament

The Karen Asrian Memorial is a tournament dedicated to a chess legend. GM Avetik Grigoryan wrote a fantastic article about him, which I highly advise you to read: <a rel='noopener noreferrer' aria-labal="In Memory of GM Karen Asrian" href="https://chessmood.com/blog/karen-asrian">In Memory of GM Karen Asrian</a>. One of the tournament's highlights was meeting Levon Aronian, a true chess superstar. I was fortunate to meet such a player in person.

The tournament format consisted of nine rounds spread over nine days. Each game was a marathon, lasting between 3 and 4.5 hours.

My performance

Overall, this was my most robust performance yet. I beat a player rated over 2150 and even faced a Grandmaster in a classical chess game for the first time. While I achieved a dominant position, time pressure ultimately led to a bittersweet loss.

At the beginning of the tournament, I was rated 1803 Elo and was the 67th player over 73 according to the initial ranking. I finished with 3.5 points over 9, half a point more than my initial goal. I earned 34 fide elo points and a performance of 2000 elo! My final rank is 57th, meaning I had an excellent performance.

Chessmood

Speaking of improvement, a big part of my chess journey wouldn't have been possible without the Chessmood team. Their training platform helped me climb over 300 points in online blitz and rapid chess ratings. This translated into real-world results, allowing me to consistently defeat players rated around 2000 Elo. I also had the opportunity to meet them in person. I played in the tournament with IM David Shahinyan, who helped me improve recently. Furthermore, I could visit Chessmood's office, discover their waiting room, and have a great barbecue with them!

Conclusion

This Armenian adventure tested my skills and reminded me of the power of dedicated practice. It left me eager to return and continue my chess journey!

If you want to know more about what I discovered in Armenia besides chess, stay tuned for my next article! I would like to share some insights about the country and its culture.

Solving the Sum of Squares Problem: Optimizing Performance

Wed, 09 Aug 2023 00:00:00 GMT

Disclaimer: Enhancing Algorithm Discussions

Before delving into the main topic, I want to express my respect for the original post's intent. My response aims to provide slight corrections that can benefit readers seeking accurate information. In the spirit of shared learning, I intend to contribute constructively rather than criticize. <br />Clarifications and alternative viewpoints can foster a deeper understanding of complex concepts. Let's continue engaging in open discussions, embracing diverse insights as we collectively refine our knowledge.

Introduction

Greetings, fellow Gophers! Are you ready to explore the intriguing world of Goroutines and unravel the mysteries of the sum of squares problem? Recently, a friend learning Go shared a <a href={'https://medium.com/@ShivamSouravJha/golang-only-things-i-know-for-the-interview-4322d29d67a3'} style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener">medium article</a> with me. It discussed Goroutines and how they can be used to solve a simple problem: calculating the sum of square numbers in a given array.

In this blog post, we'll delve into how Goroutines, channels, and data structures in Golang can be used to tackle the sum of squares problem. We'll also discuss common pitfalls and inefficient practices that can hinder performance. Fear not, my friends, for we shall also unveil secrets to optimizing your code and achieving exceptional performance. So, let's embark on this quest together!

The Sum of Squares Problem: A simple problem

The medium article of my friend addressed the following problem: "6. Write a program that returns the sum of the squares of each element of an array in Golang."

Easy enough. We need to iterate through every element in the array and add the square of each one.

func simpleSumSquare(items []int) int {
	total := 0 // total sum
	for i := 0; i < len(items); i++ {
		total += items[i] * items[i] // square the item and add it to the total
	}
	return total // return the total sum
}

Can we simplify it further? The code is reasonably straightforward. However, can we make it faster?

Goroutines: Harnessing the Power of Concurrency

Goroutines, the superheroes of Golang, are lightweight threads that enable concurrent task execution. By using Goroutines, we can execute multiple functions simultaneously, making our code more efficient and faster. But with great power comes great responsibility.<br /> One common mistake in solving the sum of squares problem is the improper use of Goroutines. Some developers create too many Goroutines without proper synchronization, leading to chaos and incorrect results. Remember, coordination is crucial!

Goroutines: Common pitfall

When solving the sum of squares problem, the choice of algorithm and data structures significantly impacts performance. The aforementioned medium blog post highlights the use of Goroutines and their potential to substantially improve performance.

func sumSquare(items []int) int {
	number := make(chan int)   // channel for sending numbers
	response := make(chan int) // channel for receiving responses

	var wg sync.WaitGroup // wait group for waiting for all goroutines to finish

	total := 0 // total sum

	// Create a goroutine for each item in the slice
	for _, item := range items {
		wg.Add(1)           // increment the wait group counter
		go func(item int) { // create a goroutine
			defer wg.Done()    // decrement the wait group counter when the goroutine finishes
			sum1 := <-number   // receive a number from the number channel
			sum1 = sum1 * sum1 // square the number
			response <- sum1   // send the result to the response channel
		}(item) // pass the item to the goroutine
		number <- item      // send the item to the number channel
		total += <-response // receive the result from the response channel
	}

	defer close(number)   // close the number channel
	defer close(response) // close the response channel

	wg.Wait() // wait for all goroutines to finish

	return total // return the total sum
}

This code creates a new goroutine for each item in the array and then performs the squaring operation. It uses channels to pass values between the goroutine and the main thread.

However, is this code correct? As discussed in <a href={'../mergesort-parallel'} style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="prefetch noopener">my previous blog post about the mergesort algorithm</a>, there are better practices than spawning numerous goroutines. More than being lightweight is needed to justify using many goroutines; it still uses memory and time, and we must be cautious.

As always, let's benchmark the code!

Benchmarking: The Quest for Ultimate Performance

Benchmarking lets us measure code performance and compare implementations to identify the best.

With Golang's built-in benchmarking tools, we can easily measure the execution time of our code and identify bottlenecks. By tweaking our implementation and experimenting with different approaches, we can achieve optimal performance and revel in our achievements.

Hence, I've created a small function to benchmark our algorithms using randomly generated arrays of varying sizes to observe their performance at different scales:

func benchmarkFramework(b *testing.B, sumFunction func([]int) int) {
	sizes := [][]int{RandomArray(100, 0, 100),
		RandomArray(1000, 0, 1000),
		RandomArray(10000, 0, 10000),
		RandomArray(100000, 0, 100000),
		RandomArray(1000000, 0, 1000000),
	}
	b.ResetTimer()
	for _, size := range sizes {
		b.Run(fmt.Sprintf("%d", len(size)), func(b *testing.B) {
			for i := 0; i < b.N; i++ {
				sumFunction(size)
			}
		})
	}
}

Here are the results:

name	time/op
SimpleSumSquare/100	49.5ns
SimpleSumSquare/1000	450ns
SimpleSumSquare/10000	4.41µs
SimpleSumSquare/100000	43.8µs
SimpleSumSquare/1000000	437µs
.
SumSquare/100	75.2µs
SumSquare/1000	730µs
SumSquare/10000	7.24ms
SumSquare/100000	72.4ms
SumSquare/1000000	739ms

As we can see, the simple algorithm outperforms the complex one! Why? Because spawning too many Goroutines slows down the program and consumes a significant amount of memory.

Data Structures & Algorithms: Choose Wisely

When solving the sum of squares problem, the choice of data structures and algorithms significantly impacts performance.

I decided to develop a more efficient function. Consequently, I used chunks and a Goroutine pool.

A goroutines pool manages a set number of reusable goroutines, reducing overhead in concurrent programs. Chunks break data into segments for parallel processing, optimizing resource use, and enhancing efficiency. Combining these techniques streamlines parallelism, maximizing concurrency benefits.

Here is my code:

func simpleParallelSumSquare(items []int) int {
	const chunkSize = 10000

	if len(items) <= 10000 { // Threshold for small slices
		return simpleSumSquare(items) // Use the simpleSumSquare function
	}

	// Divide the items into chunks
	chunks := make([][]int, 0)
	for i := 0; i < len(items); i += chunkSize {
		end := i + chunkSize // end index for the chunk
		if end > len(items) {
			end = len(items) // last chunk may be smaller than chunkSize
		}
		chunks = append(chunks, items[i:end]) // append the chunk to the chunks slice
	}

	// Create a goroutine for each chunk
	wg := sync.WaitGroup{}
	resultChan := make(chan int, len(chunks)) // channel for receiving results

	for _, chunk := range chunks { // iterate over the chunks
		wg.Add(1)              // increment the wait group counter
		go func(chunk []int) { // create a goroutine
			resultChan <- simpleSumSquare(chunk) // send the result to the result channel
			wg.Done()                            // decrement the wait group counter when the goroutine finishes
		}(chunk) // pass the chunk to the goroutine
	}

	wg.Wait()         // Wait for all goroutines to finish
	close(resultChan) // close the result channel

	// Sum the results
	total := 0
	for partialSum := range resultChan {
		total += partialSum
	}

	return total // return the total sum
}

Can this code be improved further? I could reduce the number of allocations and enhance performance. However, I opted for simplicity to better illustrate pitfalls and good practices related to Goroutines.

Is it genuinely efficient? Let's benchmark it, as always!

name	time/op
SimpleSumSquare/100	49.5ns
SimpleSumSquare/1000	450ns
SimpleSumSquare/10000	4.41µs
SimpleSumSquare/100000	43.8µs
SimpleSumSquare/1000000	437µs
.
SumSquare/100	75.2µs
SumSquare/1000	730µs
SumSquare/10000	7.24ms
SumSquare/100000	72.4ms
SumSquare/1000000	739ms
.
SimpleParallelSumSquare/100	49.5ns
SimpleParallelSumSquare/1000	450ns
SimpleParallelSumSquare/10000	4.41µs
SimpleParallelSumSquare/100000	18.0µs
SimpleParallelSumSquare/1000000	67.2µs

As we can observe, our new code is significantly faster!

Pitfalls and Bad Practices: The Road to Destruction

Ah, the treacherous road of pitfalls and bad practices. One common mistake is the excessive creation and destruction of Goroutines. Creating Goroutines carries a cost; making too many can slow your program and consume unnecessary resources.

Furthermore, we can use unsafe features to improve our code's performance but must not sacrifice maintainability and safety for speed if it's not really required. If you want to learn more about unsafe features available on Golang, you can look at the code published <a href='https://github.com/CorentinGS/go-teaching/tree/main/goroutines_sum_square' style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener nofollow"> here on my Github </a>.

Conclusion: The Sum of Success

Congratulations, my friends! We've journeyed through Goroutines, channels, and data structures, conquering the sum of squares problem. We've learned from our mistakes, optimized our code, and achieved top-notch performance.

But remember, the pursuit of knowledge is everlasting. Keep exploring, experimenting, and pushing the boundaries of what's possible. And always remember, with great power comes great responsibility. So go forth, my fellow Gophers, may your code be swift, your bugs be few, and your adventures be legendary!

References

<a href="https://medium.com/@ShivamSouravJha/golang-only-things-i-know-for-the-interview-4322d29d67a3" style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener nofollow">Medium article</a>
<a href="https://github.com/CorentinGS/go-teaching/tree/main/goroutines_sum_square" style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener">Code snippets</a>

Upgrading to dnf5: A step-by-step guide for Fedora users

Fri, 28 Apr 2023 00:00:00 GMT

Upgrading to dnf5: A guide for Fedora users

:::warning This guide is old and may contain outdated information. :::

Introduction

DNF5 is Fedora's new, faster, and more powerful package manager. Although it is still in development and won't be the default package manager until Fedora 39, you can install it now and start using it. This blog post will show you how to replace DNF with DNF5 on your Fedora system.

What's the difference between dnf5 and dnf ?

DNF is an old, single-threaded package manager with much legacy code. It's written in Python and is usually described as slow by users. DNF5, on the other hand, is a complete rewrite of DNF written in C++. It's multi-threaded, has a better user experience, should be easier to maintain, and is faster.

Why should I upgrade to dnf5?

Upgrading to DNF5 offers several benefits, including improved speed and efficiency. DNF5 is designed to be faster and more efficient than DNF, which can help speed up your system's package installation and update process. Additionally, as it will be the default package manager in Fedora39, starting to use it now and reporting any bugs you encounter will help the developers fix them before the release.

How to upgrade to dnf5?

Step 1: Install dnf5

To install DNF5 from the unstable repository, run the following command:

dnf copr enable rpmsoftwaremanagement/dnf5-unstable ;
dnf install dnf5 dnf5-plugins

If you're using sudo, use this command instead:

sudo dnf copr enable rpmsoftwaremanagement/dnf5-unstable ;
sudo dnf install dnf5 dnf5-plugins

Step 2: Create an alias for dnf5 (optional)

You can create an alias if you want to use DNF5 instead of DNF. Run the following command:

alias dnf="dnf5"

To make this alias permanent, add it to your ~/.bashrc file:

echo "alias dnf=\"dnf5\"" >> ~/.bashrc

Or add it to your ~/.zshrc file if you use zsh:

echo "alias dnf=\"dnf5\"" >> ~/.zshrc

Disclaimer

Remember that DNF5 is still in development and not ready for production use. It may contain bugs and should not be used on production systems. Use it at your own risk. I am not responsible for any damage caused by using DNF5.

Conclusion

DNF5 may still have some bugs, so it's essential to experiment with it and keep this blog post up-to-date with the latest changes. This guide helped you upgrade to DNF5. If you have any questions or suggestions, feel free to contact me on <a href="https://twitter.com/GSCorentinDev" target="_blank" rel="noopener" style="font-weight: 700; text-decoration: underline; color: var(--primary);">Twitter</a> or <a href="https://corentings.dev/discord" target="_blank" rel="noopener" style="font-weight: 700; text-decoration: underline; color: var(--primary);">Discord</a>.

References

<a href="https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5" style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener">Fedora Project</a>
<a href="https://www.reddit.com/r/Fedora/comments/12jv7uc/what_is_the_state_of_affairs_with_fedora_38_and/" target="_blank" rel="noopener" style="font-weight: 700; text-decoration: underline; color: var(--primary);">r/fedora</a>
<a href="https://github.com/rpm-software-management/dnf5/issues/411" style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener">Github issue</a>

Merge Sort using Goroutines

Wed, 11 Jan 2023 00:00:00 GMT

Parallel Merge Sort vs Simple Merge Sort

This is a simple example of how to use goroutines to parallelize a merge sort algorithm. We compare the performance of a simple merge sort algorithm with a parallel merge sort algorithm that uses goroutines.

The Merge Sort Algorithm

The merge sort algorithm is a divide and conquer algorithm that recursively splits the input array into two halves, sorts each half, and then merges the two sorted halves into a single sorted array.

To speed up the merge sort algorithm, we use insertion sort for small subarrays (less than 12 elements).

The implementation of the algorithm uses generics to allow sorting of any type of numbers.

func MergeSort[T Number](items []T) []T {
	size := len(items)
	if size < 2 {
		return items
	}

	if size < K {
		return Insertionsort(items)
	}

	middle := size / 2
	var a = MergeSort(items[:middle])
	var b = MergeSort(items[middle:])

	return merge(a, b)
}

The Merge Sort Algorithm with Goroutines

The parallel merge sort algorithm uses goroutines to sort the two halves of the input array in parallel.

To prevent the creation of too many goroutines, we use a threshold to determine when to use goroutines. If the size of the input array is less than the threshold, we use a simple merge sort algorithm instead of a parallel one.

Here we use a threshold of 512 elements. We can benchmark the performance of the algorithm with different thresholds to find the optimal threshold, but we will not do that in this example.

// ParallelMerge Perform merge sort on a slice using goroutines
func ParallelMerge[T Number](items []T) []T {
	if len(items) < 2 {
		return items
	}

	// Use a simple merge sort algorithm if the size of the input array is less than the threshold
	if len(items) < 512 {
		return MergeSort(items)
	}

	// Create the wait group to wait for the goroutines to finish
	var wg sync.WaitGroup
	wg.Add(1)

	var middle = len(items) / 2  // Find the middle index of the input array
	var a []T                   // Create a slice to hold the first half of the input array
	go func() {                // Create a goroutine to sort the first half of the input array
		defer wg.Done()       // Decrement the wait group counter when the goroutine finishes
		a = ParallelMerge(items[:middle]) // Sort the first half of the input array
	}()
	var b = ParallelMerge(items[middle:]) // Sort the second half of the input array

	wg.Wait() // Wait for the goroutine to finish
	return merge(a, b) // Merge the two sorted halves
}

Benchmarking the Merge Sort Algorithms

Now we can benchmark our merge sort algorithms to compare their performance.

To benchmark the algorithms, we use arrays of different sizes and measure the time it takes to sort the arrays.

To run the benchmark we use the following command:

go test -bench=. -benchtime 5s > benchmark.txt && benchstat benchmark.txt

Benchstat is a tool that can be used to compare the results of benchmarks.

The results of the benchmark are as follows:

name                                   time/op
Mergesort/1000                     14.6µs ± 0%
Mergesort/10000                     473µs ± 0%
Mergesort/100000                   6.33ms ± 0%
Mergesort/1000000                  87.4ms ± 0%

MergesortWithGoroutines/1000       18.7µs ± 0%
MergesortWithGoroutines/10000       217µs ± 0%
MergesortWithGoroutines/100000     2.71ms ± 0%
MergesortWithGoroutines/1000000    29.0ms ± 0%

As we can see, the parallel merge sort algorithm is much faster than the simple merge sort algorithm for large arrays.

Why is the Parallel Merge Sort Algorithm Faster?

The parallel merge sort algorithm is faster because it uses goroutines to sort the two halves of the input array in parallel. The simple merge sort algorithm sorts the two halves of the input array sequentially.

As discussed in the previous article, the cost of creating a goroutine can be high. So we should use a parallel merge sort algorithm only when the size of the input array is large enough to justify the cost of creating goroutines.

Conclusion

In this example, we have seen how to use goroutines to parallelize a merge sort algorithm. We have also benchmarked the performance of the merge sort algorithms to compare their performance. Goroutines are a powerful tool that should be used only when the cost of creating goroutines is justified by the performance improvement.

In this example, the code isn't more complex when using goroutines therefore it's worth using them.

Mastering the Worker Pool Pattern in Go

Code

The code for this article can be found on <a href="https://github.com/CorentinGS/go-teaching/tree/main/goroutines_merge_sort" style="font-weight: 700; text-decoration: underline; color: var(--primary);"> GitHub </a>

Goroutine vs Simple function

Wed, 11 Jan 2023 00:00:00 GMT

Goroutine vs Simple function

This is a simple example of why goroutines might be overkill for some tasks and less efficient than a simple function.

Structures

We got a simple structure that contains sensitive information that we don't want to be exposed to the outside world. Therefore, we created a second structure that hides the sensitive information and only exposes the information we want to be public.

// Pineapple is a struct that represents a database object with sensitive data that should be hidden
type Pineapple struct {
	Paro       string `faker:"name"`
	Turkey     string `faker:"name"`
	Banana     string `faker:"name"`
	Age        int    `faker:"number"`
	Size       int    `faker:"number"`
	IsAlive    bool
	ID         uint
	SecretCode []byte
	Created    time.Time
	Updated    time.Time
}

// SafePineApple is a struct that represents a Pineapple object without sensitive data
type SafePineApple struct {
    Paro    string
    Turkey  string
    Banana  string
    IsAlive bool
    Age     int
    ID      uint
}

Conversion

We need to convert our Pineapple object to a SafePineApple object. We can do this by creating a method on the Pineapple struct that returns a SafePineApple object.

// ToSafePineApple converts a Pineapple object to a SafePineApple object
func (p *Pineapple) ToSafePineApple() SafePineApple {
	return SafePineApple{
		Paro:    p.Paro,
		Turkey:  p.Turkey,
		Banana:  p.Banana,
		IsAlive: p.IsAlive,
		Age:     p.Age,
		ID:      p.ID,
	}
}

Use case

In our use case we have an array of Pineapple objects coming from our database that we want to convert to SafePineApple objects and store them in a new array. The order of the objects in the array should be the same as the original array as it has already been sorted by a sql query.

Simple function

We can do this by creating a simple function that takes an array of Pineapple objects and returns an array of SafePineApple objects.

// SimpleConvertPineApplesToSafety converts an array of Pineapple objects to an array of SafePineApple objects
func SimpleConvertPineApplesToSafety(pineapples []Pineapple) []SafePineApple {
	safePineApples := make([]SafePineApple, len(pineapples))

	for idx, pineapple := range pineapples {
		safePineApples[idx] = pineapple.ToSafePineApple()
	}

	return safePineApples
}

This function is very simple and easy to understand. We loop through the array of Pineapple objects and convert them to SafePineApple objects.

Goroutine without mutex

We can do this by using goroutines to work on the array concurrently and store the results in a new array.

func GoroutinesNoMutexConvertPineApplesToSafety(pineapples []Pineapple) []SafePineApple {
	// Create a slice to store the SafePineApples
	safePineApples := make([]SafePineApple, len(pineapples)/2, len(pineapples))
	safePineApples2 := make([]SafePineApple, len(pineapples)/2)

	var wg sync.WaitGroup // Create a WaitGroup to wait for all goroutines to finish
	wg.Add(1)            // Add 1 to the WaitGroup

	// Create a goroutine to convert the first half of the Pineapple objects
	go func(chunk []Pineapple) {
		defer wg.Done() // Decrement the WaitGroup when the goroutine is done
		for idx, pineapple := range chunk { // Loop through the chunk of Pineapple objects
			safePineApples[idx] = pineapple.ToSafePineApple() // Convert the Pineapple object to a SafePineApple object
		}
	}(pineapples[:len(pineapples)/2]) // Pass the first half of the Pineapple objects to the goroutine

	// Convert the second half of the Pineapple objects in the main thread
	for idx, pineapple := range pineapples[len(pineapples)/2:] {
		safePineApples2[idx] = pineapple.ToSafePineApple()
	}

	// Wait for all goroutines to finish
	wg.Wait()

	// Group both pineapples
	safePineApples = append(safePineApples, safePineApples2...)

	// Return the SafePineApples
	return safePineApples
}

This function is a bit more complex than the simple function. We use a goroutine to convert the first half while the other half is handled by the main thread. We use a WaitGroup to wait for the goroutine to finish before returning the results.

Goroutine with mutex

We can also add mutexes to the goroutine to make it thread safe.

func GoroutinesConvertPineApplesToSafety(pineapples []Pineapple) []SafePineApple {
	// Create a slice to store the SafePineApples
	safePineApples := make([]SafePineApple, len(pineapples))

	// Split the offers into chunks
	chunks := [][]Pineapple{pineapples[:len(pineapples)/2], pineapples[len(pineapples)/2:]}

	mutex := sync.Mutex{} // Create a mutex to lock the slice when writing to it

	var wg sync.WaitGroup // Create a WaitGroup to wait for all goroutines to finish
	wg.Add(1)           // Add 1 to the WaitGroup

	// Create a goroutine to convert the first half of the Pineapple objects
	go func(chunk []Pineapple) {
		defer wg.Done() // Decrement the WaitGroup when the goroutine is done
		for idx, pineapple := range chunk { // Loop through the chunk of Pineapple objects
			mutex.Lock() // Lock the mutex
			safePineApples[idx] = pineapple.ToSafePineApple() // Convert the Pineapple object to a SafePineApple object
			mutex.Unlock()  // Unlock the mutex
		}
	}(chunks[0]) // Pass the first half of the Pineapple objects to the goroutine

	// Convert the second half of the Pineapple objects in the main thread
	for idx, pineapple := range chunks[1] {
		mutex.Lock() // Lock the mutex
		safePineApples[idx+len(chunks[0])] = pineapple.ToSafePineApple() // Convert the Pineapple object to a SafePineApple object
		mutex.Unlock() // Unlock the mutex
	}

	// Wait for all goroutines to finish
	wg.Wait()

	return safePineApples
}

This function make use of the mutex to lock the slice when writing to it. This makes sure that the goroutine and the main thread don't write to the same index at the same time but instead wait for the other to finish.

Benchmark

Now that we have our functions we can benchmark them to see which one is the fastest. To benchmark our functions we run them with arrays of different sizes.

Our benchmark function looks like this:

func Benchmark_SimpleConvertPineApplesToSafety(b *testing.B) {
	for _, n := range []int{500, 1000, 2000, 5000, 10000} {
		b.Run(fmt.Sprintf("Benchmark_SimpleConvertPineApplesToSafety-%d", n), func(b *testing.B) {
			pineApples := make([]Pineapple, n)
			var pine Pineapple
			for i := 0; i < n; i++ {
				_ = faker.FakeData(&pine)
				pine.Created = time.Now().AddDate(0, 0, -i)
				pine.ID = uint(i)
				pine.IsAlive = true
				pineApples[i] = pine
			}
			for i := 0; i < b.N; i++ {
				SimpleConvertPineApplesToSafety(pineApples)
			}
		})
	}
}

To run the benchmark we use the following command:

go test -bench=. -benchtime 5s > benchmark.txt && benchstat benchmark.txt

Benchstat is a tool that can be used to compare the results of benchmarks.

The results of the benchmark are as follows:

name                                                                        time/op
Benchmark_SimpleConvertPineApplesToSafety-500-32                          15.8µs ± 0%
Benchmark_SimpleConvertPineApplesToSafety-1000-32                         32.0µs ± 0%
Benchmark_SimpleConvertPineApplesToSafety-2000-32                         66.5µs ± 0%
Benchmark_SimpleConvertPineApplesToSafety-5000-32                          193µs ± 0%
Benchmark_SimpleConvertPineApplesToSafety-10000-32                         465µs ± 0%
Benchmark_GoroutinesConvertPineApplesToSafety-500-32                      23.5µs ± 0%
Benchmark_GoroutinesConvertPineApplesToSafety-1000-32                     46.2µs ± 0%
Benchmark_GoroutinesConvertPineApplesToSafety-2000-32                     87.7µs ± 0%
Benchmark_GoroutinesConvertPineApplesToSafety-5000-32                      242µs ± 0%
Benchmark_GoroutinesConvertPineApplesToSafety-10000-32                     507µs ± 0%
Benchmark_NoMutexGoroutinesConvertPineApplesToSafety-500-32               28.3µs ± 0%
Benchmark_NoMutexGoroutinesConvertPineApplesToSafety-1000-32              48.7µs ± 0%
Benchmark_NoMutexGoroutinesConvertPineApplesToSafety-2000-32               105µs ± 0%
Benchmark_NoMutexGoroutinesConvertPineApplesToSafety-5000-32               257µs ± 0%
Benchmark_NoMutexGoroutinesConvertPineApplesToSafety-10000-32              533µs ± 0%

As you can see the simple function is the fastest.

Why ?

The reason why the simple function is the fastest is that the process of converting the Pineapple objects to SafePineApple objects is very fast. The time it takes to create the goroutines and wait for them to finish is longer than the time it takes to convert the Pineapple objects to SafePineApple objects.

Furthermore, in our goroutines implementation we have to convert the Pineapple objects then lock the mutex, write to the slice and unlock the mutex. This is a lot of overhead for a very simple task.

Conclusion

Don't use goroutines when you don't need them. That may seem obvious, but it's easy to forget when you're trying to optimize your code. In this example the overhead of creating the goroutines and waiting for them to finish is longer than the time it takes to convert the Pineapple objects to SafePineApple objects.

Goroutines are great for tasks that take a long time to complete and can be done in parallel. I would suggest to write the simplest code possible and then benchmark it to see if you can improve it using goroutines instead.

Moreover, simple code is easier to read and maintain than complex code, that's why writing complex code might not be necessary if performance is not an issue. I'd prefer to have a simple function that takes a few milliseconds longer to complete than a complex function.

Code

The code for this article can be found on <a href="https://github.com/CorentinGS/go-teaching/tree/main/goroutines_simple_vs_complex" style="font-weight: 700; text-decoration: underline; color: var(--primary);"> GitHub </a>

How to optimize a Go deployment with Docker

Tue, 08 Nov 2022 00:00:00 GMT

How to optimize a Go deployment with Docker ?

In this article, I will present the different steps that led me to optimize the deployment of services in golang using docker.

A simple Dockerfile

When I started Go and deployed a rest api service for my Memnix application, I used a basic Dockerfile that I had found on the internet.

FROM golang:1.19
RUN mkdir -p /go/src/myapp
WORKDIR /go/src/myapp
COPY archived /go/src/myapp

RUN go get -d -v
RUN go install -v

EXPOSE 8080
CMD ["/go/bin/myapp"]

The problem is that I ended up with a 2GB image while my project compiled locally without docker and optimized was only 10MB! So I decided to try to optimize the size of my Docker image as much as possible.

The first step: using a multi-stage build

It is possible to separate our Dockerfile in several parts and to use several images:

One image to build the project
A minimalist image to deploy it

This method allows us not to keep the sources and all the development tools provided with the golang image. Thus, we only keep the binaries. I decided to use the golang:1.19-alpine image as the builder.

FROM golang:1.19-alpine as builder

LABEL stage=gobuilder
ENV CGO_ENABLED=0
ENV GOOS linux
WORKDIR /build

COPY go.mod go.sum .
RUN go mod download

COPY . .
RUN go get -d -v
RUN go build -o /app/myapp .

This first part of the Dockerfile allows us to copy the sources, to synchronize the packages and to launch the command go build to generate the binary. Once the build is finished, we use a minimalist alpine image that will be used for deployment. We just need to copy the binary from our builder and run it.

FROM alpine:latest

WORKDIR /app

COPY --from=builder /app/myapp .

EXPOSE 8080

CMD ["/app/myapp"]

After testing this new Dockerfile, the final image was 34MB. It's far from the original 10MB but it's still much better than 2GB.

Improve the build process

It is possible in Golang to add flags to the build command in order to slightly optimize the binary size. After a little research on internet, I discovered the existence of "-s -w" flags. After testing them locally, I noticed an improvement of a few MB so I decided to add them to my Dockerfile. I also discovered Upx style="font-weight: 700; text-decoration: underline; color: var(--primary);"></a> which is a small program that allows to compress binary files to reduce their size. I tested this little software in a terminal and I noticed an improvement of almost <b>50%</b> on the size of the Memnix api. So I also added this step to my Dockerfile.

FROM golang:1.19-alpine as builder

LABEL stage=gobuilder
ENV CGO_ENABLED=0
ENV GOOS linux
RUN apk update --no-cache && apk add upx
WORKDIR /build

COPY go.mod go.sum .
RUN go mod download

COPY . .
RUN go get -d -v
RUN go build -ldflags="-s -w"  -o /app/myapp .
RUN upx /app/myapp

FROM alpine:3.14

WORKDIR /app

COPY --from=builder /app/myapp .

EXPOSE 8080

CMD ["/app/myapp"]

And here is my new Dockerfile! It may seem much longer and more complex than the first version but in the end, the result is very interesting. After testing this new Dockerfile, the final image is 16Mb so only 6Mb more than the version without Docker which is almost negligible.

Conclusion

Docker is a great tool to simplify application development and deployment, but it can be problematic if used incorrectly. This experience helped me understand how to better use Docker and how to optimize my images.

Solving the Sum of Squares Problem: Optimizing Performance This article is inspired by a <a href="https://twitter.com/GSCorentinDev/status/1564536030795075585?s=20&t=l2hgvKBPDMVs4pS6um2HeA" style="font-weight: 700; text-decoration: underline; color: var(--primary);">twitter thread</a> and the complete code of my Dockerfile is available on my <a href="https://github.com/memnix/memnix-rest/blob/e8e52b3d10731df5b2767f0d24bcdcb5d13d72e3/Dockerfile" style="font-weight: 700; text-decoration: underline; color: var(--primary);">github</a>. I hope this first article will interest you, don't hesitate to give me feedback so I can improve for the next articles and share your tips on Docker or Golang!

Corentin GS's Blog

Beyond PGN: Designing an Ultra-Efficient Chess Storage Format

The First Realization: A Chessboard Is Only 64 Squares

The Promotion Problem

Why This Is Already Better Than PGN

The Second Realization: Maybe We Don't Need Coordinates at All

The Beautiful Edge Case: Forced Moves Cost Almost Nothing

The Catch: Compression Is Not Free

Coordinates vs Legal Move Indexes

The Practical Design Choice

What About the Initial Position?

Conclusion: PGN Is the Beginning, Not the End

Beyond Minimalism: A Japanese-Inspired UX for the Web

Accumulated Density vs Designed Density

Ma Is the Interval Between Things

Bento as an Interface Model

Shogi, Go, and Relational Meaning

Layered Information UX

What Trustful Minimalism Looks Like in Practice

What Western Designers Can Borrow

Enough Context, Enough Room

References and Further Reading

Context and Cancellation in Go: A Practical Guide

The Problem: Orphaned Work

What Context Gives You

The Three Flavors

1. context.Background() — The Root

2. context.WithCancel() — Manual Control

3. context.WithTimeout() / WithDeadline() — Time-Bound

Listening for Cancellation

The Pattern: Per-Request Context

Propagation: The Golden Rule

Goroutines and Context

Common Mistakes

1. Storing Context in Structs

2. Ignoring ctx.Err()

3. Not Checking Cancellation in Tight Loops

4. Creating Background Contexts Deep in the Stack

Values: Use Sparingly

Graceful Shutdown

Summary

How to Merge PGN Files in F#: Streaming, Performance, and Discriminated Unions

Why F# for a CLI Tool

Streaming I/O: Merging PGN Files Without Loading Them into Memory

Discriminated Unions vs Exceptions for Error Handling

Install and Try PGN Merger

What Western UX Can Learn from Japanese Web Design

What Western UX Can Learn from Japanese Web Design

Context Is Part of Trust

The Problem Is Unmanaged Density

Structured Density: Bento, Ma, and Service Counters

From Funnel to Service Counter

What Western Designers Can Borrow

Toward Trustful Minimalism

References and further reading

Why Japanese Websites Look Overloaded: Density, Tokyo, and Trust

Why Japanese Websites Look Overloaded: Density, Tokyo, and Trust

Density Is Not One Thing

Websites as Service Counters

Tokyo Helps Explain the Pattern

What Research Suggests

Density Still Needs Hierarchy

Sources and Further Reading

TDD Isn't About Bugs — It's Your Permission to Refactor

TDD Isn't About Bugs — It's Your Permission to Refactor

The Moment It Clicked

The Trap: Testing Structure

The Fix: Test Behavior

Going Deeper: Value Objects

The Payment System

The "TDD Is Slow" Myth

When TDD Doesn't Help

Monday Morning

Docker 29 Broke Traefik — Here's the Fix (and Why It Happened)

Docker 29 Broke Traefik — Here's the Fix (and Why It Happened)

The "Everything Is Down" Moment

The 30-Second Fix

Verifying it worked

The Real Fix: Update Traefik

If you can't update Traefik directly

1. `context.Background()` — The Root

2. `context.WithCancel()` — Manual Control

3. `context.WithTimeout()` / `WithDeadline()` — Time-Bound

2. Ignoring `ctx.Err()`

The `List[E].Format[F]` Pattern