Flagarize Your Go Configuration Struct!
TL;DR In my free time I wrote a simple but powerful, open source Go library flagarize to register your flags from Go struct tags! Please try it, share feedback and read below why it was created.
Flags FTW
You are probably familiar with the eternal battle among all engineers on what is the best IDE: vim or emacs (of course vim š).
In the SRE
/Developer world there is a similar dispute.
How we should configure our microservices? I know about three ways (I removed hardcoding from the list on purpose ^^):
- By passing
flagsusing process arguments. - By specifying
environment variablesfor the process. - Last but not the least by passing
configuration fileto your CLI in some format like JSON, YAML, Protobuf, TOML, INI, etcā¦
While all of those methods have its pros and cons, Through all of my experience as a Dev/SRE and maintainers of bigger projects like Prometheus or Thanos , I learned that in most cases flags are superior. Why?
- Flags are explicit. You always know exactly how your application was configured. You donāt have this with config files and environment variables.
One can think that they passed some envvars or file, but itās just guessing! They have no idea if some process changed the file or vars
just before starting the application process. Some endpoint like
/configwhich renders current configuration may help, but still, why not just passing this as a flag? No one can change passed arguments to CLI in runtime. - Flags are easier to operate. You donāt need to copy or provision files or play with a highly dynamic environment.
- Flags are highly discoverable. While you can try to generate docs from configuration like we do in Thanos
there is nothing better than good old
--helpflag. - There are a bit fewer formats for flags than for configuration formats. More or less there are two
-flagor--flagmultiplied by two with=or without. Good libraries support both. How many configuration formats we have? Probably every day someone creates new. (:
However, someone can argue that there are some use cases for environment variables and configuration files e.g:
Secrets
This is a sensitive topic, but if you dive deeper, for example, the environment is not a safe place for secrets either. And you can
totally pass secrets by a flag directly thanks to e.g Kubernetes from secret substitution (with --my-flag=$(SECRET_CONTENT) convention).
Complex structures
They say itās hard to pass map or custom structure via a flag and sometimes you have to (e.g. when dealing with configuring dynamic plugins). I disagree with this. You can totally pass multiline strings via flags with ease, so you can pass custom format (e.g YAML) without sacrificing flag benefits:
|
With Kubernetes itās even easier. You can do it either from secret or directly render content:
|
Dynamic reload of configuration
They say you cannot have a dynamic configuration with flags. That is also not true. Have you seen
amazing go-flagz
library? Itās used on production and I can definitely recommend it.
Problem: Defining flags in Go.
Hopefully, I convinced you that flags for your Go applications are what you need in most cases. Now letās look at what is the problem with old-school flag definition. You are probably familiar with how itās done in most of the Go libraries like:
- standard
flag kingpinspf13/cobra- Peterās ff library
For example kingpin/v2
|
This is great, so whatās the problem?
Well, the problem starts when you have more than 4 flags. See this configuration of Thanos querier component :
|
Not only defining them is hard, but also using all those flags after parsing looks extremely bad.
|
This works, but itās not readable, right? And it makes you cry if you need to add any more flags. This is why we attempted to fix this with some refactoring .
With end result refactoring we probably can get close to what we have in the Prometheus project with some kind of configuration struct that is filled from flags:
|
This is much better, but still not great, particularly:
- Itās not clear what fields from the Go struct are actually set from flags, which are manually set. Especially for nested configuration structs. Itās worth to note that overall, as mentioned by Peter Bourgon , mixing user intent with derived config is a bad design smell. Such mistake is quite hard to detect with flag definition like above.
- Itās easy to miss that some configuration variable is not used anymore, but some flag is still defined for it. This is
because Go will not detect it being used, as itās passed in for example
DurationVar(&cfg.var). In fact, during the move of Prometheus flags to my library, I found that one config field was exactly like this. - We just defined a field in configuration struct, why we need to create additional 2 complex lines for each of it?
- What if we would like to parse our configuration from JSON or YAML as well? It would require huge refactoring.
- What about complex flags like Thanosās
path or content? Registering is quite tedious.
Solution
Fortunately, there are better ways to do it. At Red Hat, we have regular days called Hack'n'Hustle where we can spend a whole day
on WHATEVER you always wanted to do to improve (or break š) something unrelated to your work in open-source. During the last
one I created flagarize
. A library that is meant to solve the problems mentioned earlier.
Flagarize your config structure!
The main issue is that you have two places of definition for your configuration in code. Struct and flag registration. Flagarize allows
to have just one: Struct. You can do that by adding a struct tag called flagarize:<key=value,>, in the same way, we do json:, yaml or protobuf tags!
For example:
type config struct {
Field1 string `flagarize:"name=flag1|help=Some help.|default=Some Value|envvar=FLAG1|short=f|placeholder=<put some string here>"`
Field2 YourCustomType `flagarize:"name=custom-type-flag|help=Custom Type always add prefix 'AlwaysAddingPrefix' to the given string value.|required=true|short=c|placeholder=<put some my custom type here>"`
}
Once you define such a flag itās as easy as passing instantiated configuration through Flagarize function. Donāt forget to pass
the kingpin.Application you want to register the flags in!
|
Then as usual, parse the arguments:
|
With this you can run your CLI with āhelp which will show you the flags:
|
And when you pass those flags:
|
See this example here
But waitā¦ how flagarize knew how to parse YourCustomType?
Great question, glad you asked!
Flagarize checks for if your type implements Set(s string) error method. This is exactly the same method as kingpin library is using,
so everything that worked with kingpin will work in flagarize as well.
So above example works because our type looks like this:
|
What if the type would NOT implement Set? Flagarize will return error with helpful message as we expect š¤:
|
But passing long help in struct tag is weird and what if I need to evaluate it in runtime?
True that. In this case Flagarize allows to specify <fieldName>FlagarizeHelp variable which will hold fieldās help in string.
See below example (also available here ):
|
Under the hood
There could be probably another blog post about this topic, but I will try to be brief. Everything works thanks to standard reflect library that
Go has. For each struct (including nested and embedded ones), we can reflect value and type of each field:
|
Then we can parse the struct tag thanks to fieldType.Tag struct:
|
Before moving forward we can detect if itās struct OR embedded type, so we recursively parse that as well:
|
After some extra checks before actually registering a flag for known types, we can check if the fieldās type supports any of our two interfaces. In this case, we favor custom parsing.
We do that by casting our interface from the fieldās value as interface() with a check. Then if make sure itās allocated.
Then we actually invoke it:
|
This is quite tricky as we can have some pointer to type *type and this will work for the above statement, but someone can put type as value type
so pointer for some type. Still, if the custom method is pointer receiver this will work for flagarize thanks to Addr() method.
Thatās why we need 3 more checks like the above:
|
The allocation is interesting as well, you can allocate nil field as follows (unless it is a map!).
|
And now the interesting part. How we register the flag for all the ānativeā types? We have a monster switch for all types we support here .
Letās take one case of it, for example:
|
This registers a kingpin flag to be set on our fieldās address and expects the flag to be our fieldās type. In this case for float32.
Since the field type is a value, not a pointer, we have to first take the pointer of our field so fieldValue.Addr().
Then we need to ask for unsafe.Pointer using Pointer() method. Only then we can convert this pointer to *float32
which is what kingpin's Float32Var expects!
And thatās it! The reflection might be difficult in the beginning, but after all, itās not that complex! (:
Summary
I think flagarize
fixed all problems mentioned earlier without introducing many new ones! (:
- With
flagarizeit is clear what fields from the Go struct are actually set from flags. You just look if they haveflagarizestruct tag! You can easily encapsulate deeply nested structs e.g. for components - this will work too! - Itās harder to miss that some configuration variable is not used anymore since the field will be just clearly unused outside of
reflectinside Flagarize. - We can define the field in a struct and register as a flag in a single place.
- What if we would like to parse our configuration from JSON or YAML as well? Just add
yaml:orjson:struct tag and usejson/yaml.Unmarshal. Done. - What about complex flags like Thanosās
path or content? Well ,one thing is thatflagarizesupports some complex types natively likeRegexp, AnchoredRegexp, TimeOrDuration, PathOrContent, secondly adding new custom ones is super easy with 2 interfaces. explained in README.md
Other projects
Truth is that there are some nice projects in the open source already existing with similar patter:
-
octago/sflag: This is quite epic as it is more generalized and works for most of the popular flag libraries. However, it does not support (?) custom types or custom flag registering. To me, the API and error messages could be better as well. Itās nice when the library avoids allowing more than one way of doing the same thing. -
alecthomas/kong: This is very similar toflagarize. In fact it, is actually written by the author ofkingpinlibrary! Even better, itās meant to replace it. Still, I think it does not support some complex typesflagarizesupports out of the box, and there are way too many ways to extend itā¦ and make a bug on the way. (: Also, I am not a great fan ofkongstruct tag format. It does not own just one struct tag, it kind of use.. all of them as a map. This is then tricky to provide friendly error message if someone makes a typo in the struct tag, etc.
Still, itās worth looking on those two if you considering moving to struct tag pattern. Great job by all those maintainers for providing such amazing tools to open source Go devs! ā¤ļø
The end
While writing flagarize
I definitely had lots of fun and learned a bit of reflect on the way.
However, this library was not purely for fun. I just released v0.9.0
(wanted 0.999.0 but letās be serious ;p)
I plan to maintain and use in production for our open source projects like Prometheus (PR is in review
)
and Thanos
thanks to Philip
.
1,0 release is planned once we adopt it in Thanos .
Comments