The CLI boilerplate you have been looking for

Using spf13/viper to gather parameters from various sources can be tricky. You will probably run into the problem of having to redefine default parameters for each means of input. This post shows how to to load parameters from different sources without having to define them multiple times. The common priority for input methods is this:

1. CLI parameters
2. Environment variables
3. Config file
4. Default values

The boilerplate outlined below is focused on various possibilities to configure your application. You can find a working example in this repository github.com/benchkram/cli-utils/base.

Why do I need different means of configuration?

Providing multiple ways to pass parameters to an application allows users to choose the most appropriate method for their use case, and provides the application with the flexibility to adapt to various scenarios. It is very common that your application is being run in various environments and by multiple users. Your app could be run in a local dev environment via go run with some command line arguments, it could be run inside a docker container with a config file, or with environment variables passed by some CI pipeline. Sometimes there is even a combination of all three options. E.g. sensitive data such as passwords and tokens are provided via environment variables, routing configuration though a config file and run-specific parameters via cli arguments.

Where to start?

When creating a new CLI application, it's helpful to start with a boilerplate that provides a foundation for the application. To be reusable across multiple projects, the boilerplate should be designed to be flexible and customizable and should include the most common features needed.

The CLI:  Just an interface to your application...

.. and configuration should not leak into your application. Everything needed to start and configure an application is bundled together in a package called cli . The application specific code goes in a directory called application . This makes it easy to configure an application from the cli package but also enables you to easily initialize an application for testing purposes.

The file structure created in this post is structured around this idea.

project
│   main.go
│
└───cli
│   │   cli.go
│   │   cmd_root.go
│   │   config.go
│
└───application // or 'app'
    │   // your app code

As the cli package does the wiring between an application and the actual config the main function acts as a simple entry point, like this:

func main() {
    // This main function is only responsible for calling the cli package
    // and forwarding errors to the user
    if err := cli.Execute(); err != nil {
        fmt.Println(err)
        os.Exit(1)
    }
}

Defining a config struct

Relying on one, explicit, config struct makes it easy to to pass configuration along to your application. The tricky part is to read the config from various sources into the config struct while keeping and reusing default values.

The config structs looks similar to this and is defined in cli/config.go:

type config struct {
    Param1 string `mapstructure:"param1" structs:"param1" env:"PARAM1"`
    Param2 int `mapstructure:"param2" structs:"param2" env:"PARAM2"`
}

As you can see each parameter has three tags:

• mapstructure assures to map the defaults to viper.
• structs is needed to create a map with default values
• env name of the associated environment variable

mapstructure and structs are required to contain the same value. The value of  env usually differs due to the the different style of flags vs. environment variables, e.g. --my-var vs. MY_VAR

Default values for the configuration can be defined through a instantiation of the config struct, like this:

var defaultConfig = config{
    Param1: "Hello World",
    Param2: 42,
}

Creating the root command

The root command is executed when our binary runs without additional commands. We'll be using github.com/spf13/cobra to define commands. For convenience I recommend creating a go file for each command. In our case we only have one command which is the root command. So we create a file called cli/root_cmd.go with this content:

var rootCmd = &cobra.Command{
    Use:   "our_app",
    Short: "cli to start example application
    Long:  "cli to start example application",
    Run: func(cmd *cobra.Command, args []string) {
        // here is where we start the actual application
    },
}

This will be the command we call from the cli.Execute function in the main.go file.

func Execute() error {
    return rootCmd.Execute()
}

Connecting the dots

Now that we have defined our config struct and command we need to establish a dependency between cli flags, environment variables and config read from a file. spf13/viper does a great job bringing everything together.

Defining cli flags on the root command

To define a flag on a cmd we are using the  PersistentFlags method. This is important because we want to make sure that the flags are available for all subcommands as well. The flags are defined like this:

func cliFlags() {
    rootCmd.PersistentFlags().String("param1", defaultConfig.Param1, "param1 description")
    rootCmd.PersistentFlags().Int("param2", defaultConfig.Param2, "param2 description")
}

Establish a dependency between cli flags, environment variables and config read from file

Each flag requires calls to viper.BindPFlag and viper.BindEnv to establish a dependency between the different config sources. To avoid code duplication we can create a combined call in one function called bindFlagsAndEnv . The tags we added previously to the config struct are used to define the name of the flag and the name of the environment variable. The following code snippet demonstrates how the binding can be accomplished:

func bindFlagsAndEnv() (err error) {
    for _, field := range structs.Fields(&config{}) {
        key := field.Tag("structs")
        env := field.Tag("env")
        err = viper.BindPFlag(key, rootCmd.PersistentFlags().Lookup(key))
        if err != nil {
            return err
        }
        err = viper.BindEnv(key, env)
        if err != nil {
            return err
        }
    }
    return nil
}

This reduces the number of places to modify when adding or removing parameters and save us some time.

Initialization

Now that we have a function for each of the three steps we can combine them into a single function that initializes the config. This function can then be called by the package's init function. If your cli package is not calling any other initialization functions you can also call the function directly from the init function.

func configInit() error {
    cliFlags()
    return bindFlagsAndEnv()
}

func init() {
    if err := configInit(); err != nil {
        panic(err.Error())
    }
    
    // additonal initialization
    // rootCmd.AddCommand(subCmd)
}

Loading the config

Now that we have defined and initialized the config we are ready to read it. These three steps need to be performed.

• Loading the defaults into viper:
  Relying on the tags added to the config struct we can call  viper.SetDefault in a loop for every parameter.
• Loading the config file with viper:
  We just need to tell viper the name of the config file and the path where it can be found. In our case the config file is called config and it is located in the current working directory. Of course, you could also create a cli flag to define the path to the config file.
• Unmarshaling and return the config struct:
  Binding the config to from viper to our config struct is done by using the Unmarshal method of viper. This method takes a pointer to a struct and will fill the struct with the values from the config and either return it or store it in a global config variable. In our case we want to return the config struct so we can pass it to our application.

// readConfig a helper to read default from a default config object.
func readConfig() (*config, error) {

    defaultsAsMap := structs.Map(defaultConfig)

    // Set defaults
    for key, value := range defaultsAsMap {
        viper.SetDefault(key, value)
    }
    
    // load the config
    viper.SetConfigName("config")
    viper.AddConfigPath(".")
    if err := viper.ReadInConfig(); err == nil {
        fmt.Println("Using config file:", viper.ConfigFileUsed())
    }
    
    // Unmarshal config into struct
    c := &config{}
    err := viper.Unmarshal(c)
    if err != nil {
        return nil, err
    }
    return c, nil
}

Et voilà, we have a config struct that is populated from multiple sources with a well-defined priority. It's even possible to read one parameter from a cli-flag another one from a different source.

It is now ready to be passed to our application.

How can we use it?

Now we have various ways on setting parameters for our application. Let's check how we can do this with some example parameters Param1,  Param2, Param3 and  Param4. Let's load each param from a different source.

config.yaml:

param1: "param from config"

defaultConfig:

var defaultConfig = config{
    Param2: "param with default value",
}

Then call the app like this:

PARAM3="param from env" ./our_app --param4="param from cli arg"

This call would result in the following config:

{
    "Param1": "param from config",
    "Param2": "param with default value",
    "Param3": "param from env",
    "Param4": "param from cli arg"
}

Redacting sensitive data when printing the config

It's often helpful for a DevOps team to print the entire config on application start for debugging purposes. Though, the config usually contains sensitive data such as passwords, API keys, and access tokens in configuration files. However, it is important to ensure that this sensitive data is not exposed to unauthorized users, as this could result in security breaches and data leaks. We can make use of the github.com/leebenson/conform package and simply add the tag conform:"redact"  to the sensitive variable in our config struct. It should then look like this:

SensitiveData string `mapstructure:"sensitive_data" structs:"sensitive_data" env:"SENSITIVE_DATA" conform:"redact"`

Additionally we need to define what the sensitive data should be replaced with. Therefor we add a sanitizer in the cli packages init function. The sanitizer takes our "redact" keyword and returns a redacted string. Simplest variant is to just return a few "*" characters.

func init(){
    // A custom sanitizer to redact sensitive data by defining a struct tag= named "redact".
    conform.AddSanitizer("redact", func(_ string) string { return "*****" })
}

Now when we want to print our config or return it in string format we need to call conform to redact all strings that are tagged before returning the output.

// Print the config object but remove sensitive data
func (c *config) Print() {
    cp := *c
    _ = conform.Strings(&cp)
    fmt.Printf("Config: %+v\n", cp)
}

// String of the config with sensitive data removed
func (c *config) String() string {
    cp := *c
    _ = conform.Strings(&cp)
    return fmt.Sprintf("%+v", cp)
}

Conclusion

We showed you how to create a cli package that can load configuration from different sources with a well-defined priority: cli-flags, environment variables, config files while providing the same default values.

The code for this post can be found at github.com/benchkram/cli-utils/base.

Do you need Golang Consulting?

We are a Software Agency from Stuttgart, Germany and can support you on your journey to deliver outstanding user experiences. Reach out to us.

Something Else?

Make sure to check out our build tool for large multi-language applications https://bob.build