Time to put the pieces together. In this project you’ll build wordfreq, a small command-line tool that counts how often each word appears in text files or in its standard input, and prints the most frequent ones. Along the way you’ll use flags, bufio, maps, sorting with slices.SortFunc, error handling, tests, and finally go build and go install to turn it into a real command.
Here’s where we’re headed:
echo "Go is fun. Is Go fast? Go is fast." | wordfreq -iCode language: Bash (bash) 3 go
3 is
2 fast
1 funCode language: plaintext (plaintext)Nothing in this lesson is new on its own. If something looks unfamiliar, I link back to the lesson that introduced it.
Set up the project
Create a directory and a module, as in the packages and modules lesson:
mkdir wordfreq
cd wordfreq
go mod init example.com/wordfreqCode language: Bash (bash)go: creating new go.mod: module example.com/wordfreqCode language: plaintext (plaintext)The last part of the module path, wordfreq, will also be the name of the program when you install it. We’ll also need some text to count. Save this as sample.txt in the same directory:
The fox saw the dog. The dog saw the fox,
and the fox ran. "Run, dog, run!" said the cat.
A cat is not a dog, and a dog is not a fox.Code language: plaintext (plaintext)Step 1: count the words
The heart of the tool is a function that reads text and counts words in a map. Two decisions to make:
- What’s a word? A
bufio.Scanner(from the files lesson) can split its input on whitespace instead of lines:scanner.Split(bufio.ScanWords). That gives us chunks like"Run,andfox., so we trim everything that isn’t a letter or digit from both ends withstrings.TrimFunc. Punctuation inside a word stays, sodon'tremains one word. - Where does the text come from? The function takes an
io.Reader, so it doesn’t care whether it reads a file, standard input or a string. And it adds to a map you pass in, so counting several files is just calling it several times.
package main
import (
"bufio"
"fmt"
"io"
"strings"
"unicode"
)
func countWords(r io.Reader, counts map[string]int, ignoreCase bool) error {
scanner := bufio.NewScanner(r)
scanner.Split(bufio.ScanWords)
for scanner.Scan() {
word := strings.TrimFunc(scanner.Text(), notLetterOrDigit)
if word == "" {
continue
}
if ignoreCase {
word = strings.ToLower(word)
}
counts[word]++
}
return scanner.Err()
}
func notLetterOrDigit(r rune) bool {
return !unicode.IsLetter(r) && !unicode.IsDigit(r)
}
func main() {
text := `"Run, dog, run!" said the cat. The dog didn't run.`
counts := map[string]int{}
countWords(strings.NewReader(text), counts, false)
fmt.Println(counts)
folded := map[string]int{}
countWords(strings.NewReader(text), folded, true)
fmt.Println(folded)
}Code language: Go (go)map[Run:1 The:1 cat:1 didn't:1 dog:2 run:2 said:1 the:1]
map[cat:1 didn't:1 dog:2 run:3 said:1 the:2]Code language: plaintext (plaintext)counts[word]++ works even for a word that isn’t in the map yet, because a missing key gives the zero value 0 (see the maps lesson). notLetterOrDigit is an ordinary function passed as a value to TrimFunc, which calls it for each rune at the edges of the word. And with ignoreCase, Run and run become the same word.
(fmt.Println prints map keys sorted, which is handy here. Uppercase letters sort before lowercase ones, which is why Run and The come first.)
Step 2: sort by frequency
A map has no order, so to show the most frequent words first we copy the counts into a slice of structs and sort it. slices.SortFunc takes a comparison function that returns a negative number if a should come before b, a positive number if after, and 0 if they’re equal. cmp.Compare produces exactly that for two numbers or strings.
package main
import (
"cmp"
"fmt"
"slices"
)
type Entry struct {
Word string
Count int
}
func topN(counts map[string]int, n int) []Entry {
entries := make([]Entry, 0, len(counts))
for word, count := range counts {
entries = append(entries, Entry{Word: word, Count: count})
}
slices.SortFunc(entries, func(a, b Entry) int {
if c := cmp.Compare(b.Count, a.Count); c != 0 {
return c
}
return cmp.Compare(a.Word, b.Word)
})
if n > 0 && n < len(entries) {
entries = entries[:n]
}
return entries
}
func main() {
counts := map[string]int{"dog": 2, "run": 3, "cat": 1, "the": 2, "said": 1}
for _, e := range topN(counts, 3) {
fmt.Printf("%5d %s\n", e.Count, e.Word)
}
fmt.Println(len(topN(counts, 0)), "words in total")
}Code language: Go (go) 3 run
2 dog
2 the
5 words in totalCode language: plaintext (plaintext)Two details make the output predictable:
- Higher counts first. Comparing
b.Countwitha.Count(in that order) sorts from high to low. - Ties broken alphabetically.
dogandtheboth appear twice. Without the second comparison, their order would depend on the random order of the map loop, and the output could change from run to run. A tie-breaker makes the result the same every time, which also makes it testable.
n of 0 means “all words”, and an n larger than the number of words is fine too: the slice is only cut when it’s longer than n. %5d pads the count to five characters, so the words line up in a column.
Step 3: flags, files and standard input
Now the tool needs a command-line interface. The flag package parses options like -n 5 and -i:
n := flag.Int("n", 10, "show the `count` most frequent words (0 shows all)")
ignoreCase := flag.Bool("i", false, "ignore case (Go and go count as one word)")
flag.Parse()
files := flag.Args() // everything after the flagsCode language: Go (go)flag.Int defines a flag with a name, a default value and a help text, and returns a pointer to an int. The value is only filled in when you call flag.Parse(), so you read it afterwards as *n. A word in backticks in the help text becomes the placeholder name in the usage message (-n count). After parsing, flag.Args() returns the remaining arguments: our file names.
The rest of the plan:
- No file names means read from
os.Stdin, so the tool works in pipelines likecat *.txt | wordfreq. Unix tools have always worked this way. - Errors go to standard error (
os.Stderr), prefixed with the program name, and the program exits with status 1. That way an error never ends up mixed into output that’s being piped into another program, and scripts can check the exit status. mainonly handles flags and exiting. The real work happens in arunfunction that gets its input and output as anio.Readerand anio.Writerand returns an error. That keepsmaintiny and makesruntestable without real files or a terminal.
The complete program
Here’s the whole tool. Save it as main.go:
// Command wordfreq counts how often each word appears in text.
//
// Usage:
//
// wordfreq [-n count] [-i] [file ...]
//
// It reads the named files, or standard input if there are none.
package main
import (
"bufio"
"cmp"
"errors"
"flag"
"fmt"
"io"
"os"
"slices"
"strings"
"unicode"
)
// Entry is a word and the number of times it appears.
type Entry struct {
Word string
Count int
}
func main() {
n := flag.Int("n", 10, "show the `count` most frequent words (0 shows all)")
ignoreCase := flag.Bool("i", false, "ignore case (Go and go count as one word)")
flag.Usage = func() {
out := flag.CommandLine.Output()
fmt.Fprintln(out, "Usage: wordfreq [-n count] [-i] [file ...]")
fmt.Fprintln(out, "Counts words in the files, or in standard input if no files are given.")
flag.PrintDefaults()
}
flag.Parse()
err := run(flag.Args(), *n, *ignoreCase, os.Stdin, os.Stdout)
if err != nil {
fmt.Fprintln(os.Stderr, "wordfreq:", err)
os.Exit(1)
}
}
// run does the actual work. It takes its input and output as parameters
// so that tests can call it without real files or a terminal.
func run(files []string, n int, ignoreCase bool, stdin io.Reader, stdout io.Writer) error {
if n < 0 {
return errors.New("-n must be 0 or more")
}
counts := map[string]int{}
if len(files) == 0 {
if err := countWords(stdin, counts, ignoreCase); err != nil {
return err
}
}
for _, name := range files {
if err := countFile(name, counts, ignoreCase); err != nil {
return err
}
}
w := bufio.NewWriter(stdout)
for _, e := range topN(counts, n) {
fmt.Fprintf(w, "%5d %s\n", e.Count, e.Word)
}
return w.Flush()
}
func countFile(name string, counts map[string]int, ignoreCase bool) error {
f, err := os.Open(name)
if err != nil {
return err
}
defer f.Close()
return countWords(f, counts, ignoreCase)
}
// countWords reads text from r and adds the number of times each word
// appears to counts. Punctuation around words is removed.
func countWords(r io.Reader, counts map[string]int, ignoreCase bool) error {
scanner := bufio.NewScanner(r)
scanner.Split(bufio.ScanWords)
for scanner.Scan() {
word := strings.TrimFunc(scanner.Text(), notLetterOrDigit)
if word == "" {
continue
}
if ignoreCase {
word = strings.ToLower(word)
}
counts[word]++
}
return scanner.Err()
}
func notLetterOrDigit(r rune) bool {
return !unicode.IsLetter(r) && !unicode.IsDigit(r)
}
// topN returns the n most frequent words, most frequent first. Words with
// the same count are sorted alphabetically. If n is 0, it returns all words.
func topN(counts map[string]int, n int) []Entry {
entries := make([]Entry, 0, len(counts))
for word, count := range counts {
entries = append(entries, Entry{Word: word, Count: count})
}
slices.SortFunc(entries, func(a, b Entry) int {
if c := cmp.Compare(b.Count, a.Count); c != 0 {
return c
}
return cmp.Compare(a.Word, b.Word)
})
if n > 0 && n < len(entries) {
entries = entries[:n]
}
return entries
}Code language: Go (go)A few things worth pointing out:
- The comment above
package mainis the program’s documentation.go docshows it, and pkg.go.dev would too if you published the module. flag.Usagereplaces the default help message with one that explains the file arguments.flag.PrintDefaults()adds the list of flags.countFileexists so thatdefer f.Close()closes each file as soon as it’s counted. Adeferinside the loop inrunwould keep every file open untilrunreturns (the defer lesson explains why). Since we only read, ignoring the error fromCloseis fine.- The output goes through a
bufio.Writer, so printing thousands of words doesn’t mean thousands of separate writes.runreturns the error fromFlush, which covers every write before it. os.Exit(1)ends the program immediately with a non-zero status, the universal sign for “something went wrong”.
Step 4: test it
Before trying it by hand, let’s pin down the behavior with tests (see the testing lesson). Save this as main_test.go:
package main
import (
"bytes"
"maps"
"slices"
"strings"
"testing"
)
func TestCountWords(t *testing.T) {
tests := []struct {
name string
input string
ignoreCase bool
want map[string]int
}{
{"empty", "", false, map[string]int{}},
{"punctuation", `Hello, world! "Hello" (world)...`, false, map[string]int{"Hello": 2, "world": 2}},
{"case matters", "Go go GO", false, map[string]int{"Go": 1, "go": 1, "GO": 1}},
{"ignore case", "Go go GO", true, map[string]int{"go": 3}},
{"keeps inner apostrophes", "don't stop, don't", false, map[string]int{"don't": 2, "stop": 1}},
{"only punctuation", "-- ... !!", false, map[string]int{}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := map[string]int{}
err := countWords(strings.NewReader(tt.input), got, tt.ignoreCase)
if err != nil {
t.Fatal(err)
}
if !maps.Equal(got, tt.want) {
t.Errorf("got %v, want %v", got, tt.want)
}
})
}
}
func TestTopN(t *testing.T) {
counts := map[string]int{"b": 2, "a": 2, "c": 5, "d": 1}
want := []Entry{{"c", 5}, {"a", 2}, {"b", 2}}
if got := topN(counts, 3); !slices.Equal(got, want) {
t.Errorf("topN(counts, 3) = %v, want %v", got, want)
}
if got := topN(counts, 0); len(got) != 4 {
t.Errorf("topN(counts, 0) returned %d entries, want 4", len(got))
}
if got := topN(counts, 10); len(got) != 4 {
t.Errorf("topN(counts, 10) returned %d entries, want 4", len(got))
}
}
func TestRun(t *testing.T) {
var out bytes.Buffer
err := run(nil, 2, true, strings.NewReader("a b A c B a"), &out)
if err != nil {
t.Fatal(err)
}
want := " 3 a\n 2 b\n"
if out.String() != want {
t.Errorf("output = %q, want %q", out.String(), want)
}
}Code language: Go (go)go test -vCode language: Bash (bash)=== RUN TestCountWords
=== RUN TestCountWords/empty
=== RUN TestCountWords/punctuation
=== RUN TestCountWords/case_matters
=== RUN TestCountWords/ignore_case
=== RUN TestCountWords/keeps_inner_apostrophes
=== RUN TestCountWords/only_punctuation
--- PASS: TestCountWords (0.00s)
--- PASS: TestCountWords/empty (0.00s)
--- PASS: TestCountWords/punctuation (0.00s)
--- PASS: TestCountWords/case_matters (0.00s)
--- PASS: TestCountWords/ignore_case (0.00s)
--- PASS: TestCountWords/keeps_inner_apostrophes (0.00s)
--- PASS: TestCountWords/only_punctuation (0.00s)
=== RUN TestTopN
--- PASS: TestTopN (0.00s)
=== RUN TestRun
--- PASS: TestRun (0.00s)
PASS
ok example.com/wordfreq 0.238sCode language: plaintext (plaintext)TestRun shows the payoff of passing readers and writers around: it feeds the whole program a string instead of standard input and captures the output in a bytes.Buffer instead of the terminal. maps.Equal and slices.Equal compare a whole map or slice in one call.
Step 5: build and run it
You could use go run . for everything, but a real command is nicer. go build compiles the module into an executable named after it (wordfreq.exe on Windows), as you saw in Build and run a Go program:
go build
./wordfreq sample.txtCode language: Bash (bash) 5 dog
4 fox
4 the
3 a
2 The
2 and
2 cat
2 is
2 not
2 sawCode language: plaintext (plaintext)That’s the default top 10. Notice the and The on separate lines. That’s what -i is for:
./wordfreq -i -n 5 sample.txtCode language: Bash (bash) 6 the
5 dog
4 a
4 fox
2 andCode language: plaintext (plaintext)Several files are counted together. Create a second file and try it:
echo "Go is fun. Is Go fast? Go is fast." > notes.txt
./wordfreq -n 3 -i sample.txt notes.txtCode language: Bash (bash) 6 the
5 dog
5 isCode language: plaintext (plaintext)is appears twice in sample.txt and three times in notes.txt. Without file names, it reads standard input, so it works with redirection and pipes. (-n=2 is another accepted way to write a flag value.)
./wordfreq -n=2 -i < sample.txtCode language: Bash (bash) 6 the
5 dogCode language: plaintext (plaintext)Help and errors
The flag package gives you -h for free, using our flag.Usage:
./wordfreq -hCode language: Bash (bash)Usage: wordfreq [-n count] [-i] [file ...]
Counts words in the files, or in standard input if no files are given.
-i ignore case (Go and go count as one word)
-n count
show the count most frequent words (0 shows all) (default 10)Code language: plaintext (plaintext)An unknown flag prints flag provided but not defined: -x followed by the same help, and exits with status 2. Our own errors exit with status 1. $? holds the exit status of the last command in macOS and Linux shells:
./wordfreq -i sample.txt missing.txt
echo $?Code language: Bash (bash)wordfreq: open missing.txt: no such file or directory
1Code language: plaintext (plaintext)The flag package stops parsing at the first argument that isn’t a flag. ./wordfreq sample.txt -n 3 treats -n and 3 as file names and fails with open -n: no such file or directory. Put flags before the file names.
Step 6: install it
./wordfreq only works in the project directory. To use it anywhere, install it:
go installCode language: Bash (bash)go install builds the program and puts the executable in Go’s bin directory: the one in the GOBIN environment variable if it’s set, otherwise the bin folder inside go env GOPATH, which is $HOME/go/bin unless you changed it. If that directory is in your PATH (if not, add it the same way you added Go’s own bin directory when you installed Go), you can now run wordfreq from any directory, like at the start of this lesson:
echo "Go is fun. Is Go fast? Go is fast." | wordfreq -iCode language: Bash (bash) 3 go
3 is
2 fast
1 funCode language: plaintext (plaintext)The binary is self-contained: copy it to another computer with the same operating system and processor type and it runs, no Go installation needed. If you’re curious how small you can make it, read How to shrink a Go executable.
Step 7: give the counting its own package
One file is perfectly fine for a tool this size. But counting words is useful on its own, and if the tool grows (more output formats, more flags), keeping the command-line handling apart from the logic pays off. Let’s move countWords, topN and Entry into a package called count:
wordfreq/
├── go.mod
├── main.go
├── main_test.go
└── count/
├── count.go
└── count_test.goCode language: plaintext (plaintext)Functions used from another package must be exported, so they get capitalized names. Since the package is called count, callers will write count.Words and count.Top, which reads better than repeating the package name in the function name. Here’s count/count.go:
// Package count counts words in text.
package count
import (
"bufio"
"cmp"
"io"
"slices"
"strings"
"unicode"
)
// Entry is a word and the number of times it appears.
type Entry struct {
Word string
Count int
}
// Words reads text from r and adds the number of times each word
// appears to counts. Punctuation around words is removed.
func Words(r io.Reader, counts map[string]int, ignoreCase bool) error {
scanner := bufio.NewScanner(r)
scanner.Split(bufio.ScanWords)
for scanner.Scan() {
word := strings.TrimFunc(scanner.Text(), notLetterOrDigit)
if word == "" {
continue
}
if ignoreCase {
word = strings.ToLower(word)
}
counts[word]++
}
return scanner.Err()
}
func notLetterOrDigit(r rune) bool {
return !unicode.IsLetter(r) && !unicode.IsDigit(r)
}
// Top returns the n most frequent words, most frequent first. Words with
// the same count are sorted alphabetically. If n is 0, it returns all words.
func Top(counts map[string]int, n int) []Entry {
entries := make([]Entry, 0, len(counts))
for word, c := range counts {
entries = append(entries, Entry{Word: word, Count: c})
}
slices.SortFunc(entries, func(a, b Entry) int {
if c := cmp.Compare(b.Count, a.Count); c != 0 {
return c
}
return cmp.Compare(a.Word, b.Word)
})
if n > 0 && n < len(entries) {
entries = entries[:n]
}
return entries
}Code language: Go (go)notLetterOrDigit stays lowercase: it’s a detail the package doesn’t need to share. In main.go, delete Entry, countWords, notLetterOrDigit and topN, and import the new package by its module path plus directory:
import (
"bufio"
"errors"
"flag"
"fmt"
"io"
"os"
"example.com/wordfreq/count"
)Code language: Go (go)Then call it in the three places that used the old functions:
// in run:
if err := count.Words(stdin, counts, ignoreCase); err != nil {
return err
}
for _, e := range count.Top(counts, n) {
fmt.Fprintf(w, "%5d %s\n", e.Count, e.Word)
}
// in countFile:
return count.Words(f, counts, ignoreCase)Code language: Go (go)Move TestCountWords and TestTopN into count/count_test.go with package count at the top, renamed to TestWords and TestTop and calling Words and Top. TestRun stays in main_test.go. Then run all tests in the module with ./..., which means “this directory and everything below it”:
go test ./...Code language: Bash (bash)ok example.com/wordfreq 0.645s
ok example.com/wordfreq/count 0.909sCode language: plaintext (plaintext)The program behaves exactly as before; only the code is organized differently. That’s what the tests are for: they tell you the move didn’t break anything.
Practice
1. Skip boring words. In English text, “the”, “a” and “and” usually win, which isn’t very interesting. Write a function removeWords(counts map[string]int, words []string) that deletes the given words from the map. Try it on map[string]int{"the": 6, "dog": 5, "a": 4, "fox": 4, "and": 2}. As a bonus, add a -skip flag to wordfreq that takes a comma-separated list (strings.Split helps) and calls your function before topN.
Show a solution
package main
import "fmt"
func removeWords(counts map[string]int, words []string) {
for _, w := range words {
delete(counts, w)
}
}
func main() {
counts := map[string]int{"the": 6, "dog": 5, "a": 4, "fox": 4, "and": 2}
removeWords(counts, []string{"the", "a", "and", "cat"})
fmt.Println(counts)
}Code language: Go (go)map[dog:5 fox:4]Code language: plaintext (plaintext)delete on a key that isn’t there ("cat") does nothing, so no check is needed. In wordfreq, the flag would be skip := flag.String("skip", "", "comma-separated words to leave out"), and in run: if skip != "" { removeWords(counts, strings.Split(skip, ",")) }.
2. JSON output. Other programs would rather read JSON than columns. Write printJSON(w io.Writer, entries []Entry) error that writes the entries as an indented JSON array of objects with lowercase keys word and count (the JSON lesson has everything you need). Test it by printing two entries to os.Stdout.
Show a solution
package main
import (
"encoding/json"
"io"
"log"
"os"
)
type Entry struct {
Word string `json:"word"`
Count int `json:"count"`
}
func printJSON(w io.Writer, entries []Entry) error {
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
return enc.Encode(entries)
}
func main() {
entries := []Entry{{Word: "the", Count: 6}, {Word: "dog", Count: 5}}
if err := printJSON(os.Stdout, entries); err != nil {
log.Fatal(err)
}
}Code language: Go (go)[
{
"word": "the",
"count": 6
},
{
"word": "dog",
"count": 5
}
]Code language: plaintext (plaintext)To wire it in, add asJSON := flag.Bool("json", false, "print JSON") and, in run, call printJSON(stdout, topN(counts, n)) instead of the printing loop when it’s set.
Key takeaways
- The
flagpackage defines options with defaults and help text;flag.Args()holds the rest of the arguments, and-hcomes for free. - Read from files or
os.Stdinthrough the sameio.Readercode; write results to standard output and errors to standard error with a non-zero exit status. - Keep
maintiny and put the work in arunfunction that takes readers, writers and returns an error. It’s easy to test. - Sort with
slices.SortFuncandcmp.Compare, and always add a tie-breaker when sorting map data, so the output is the same every run. go buildmakes an executable in the current directory;go installputs it in Go’s bin directory so you can run it anywhere.