GoLibs is a collection of reusable Go packages for building robust, testable, and maintainable applications. It provides utilities for configuration, HTTP clients/servers, logging, graceful shutdown, key-value stores, Kubernetes testing, and more.
The initial open-source release of this project is provided as-is. That implies that the codebase is only a slightly modified version of what we at ING are using. In the future we would like to extend this Open Source repository with the appropriate pipelines and contribution mechanics. This does imply that your journey for using this project for your own purposes can use some improvement, and we will work on that.
This project is part of ING Neoria. Neoria contains parts of the ING Container Hosting Platform (ICHP) stack which is used to deliver Namespace-as-a-Service on top of OpenShift.
go get github.com/ing-bank/golibs| Package | Description |
|---|---|
access |
Subject Access Reviews and authorization helpers based on trust set in request contexts. Used for fine-grained access control in APIs and services. |
scope |
Scope definitions and utilities for expressing permissions and access levels. |
config |
Load, validate, and apply defaults to configuration structs from YAML/JSON files. Supports command-line flag binding. |
defaultmap |
map with key collision handling, merging, and update logic. |
errors |
HTTP status code error types, conversion utilities, and retry logic for API/transport errors. Includes helpers for common error patterns. |
fsnotify |
Polling-based file watcher for monitoring file changes and triggering callbacks, useful for hot-reload scenarios. |
ginresponse |
Experimental: Response helpers for Gin HTTP handlers, simplifying response formatting and error handling. |
ginserver |
Utilities for setting up Gin-based HTTP servers with middleware, metrics, graceful shutdown, and advanced configuration. |
manager |
Gin server manager for orchestrating server lifecycle and configuration. |
proxy |
Reverse proxy middleware for Gin servers, enabling request forwarding and load balancing. |
graceful |
Graceful shutdown and background task management with context cancellation, signal handling, and concurrency support. |
healthcheck |
Health check endpoints and checks for HTTP servers, background jobs, and dependencies. |
checks |
Built-in health checks for common protocols and services. |
http |
HTTP endpoint health checks. |
ok |
Always-OK health checks for basic liveness probes. |
telnet |
Telnet-based health checks for legacy systems. |
http |
HTTP client wrapper with middleware (tripperware), transport customization, and simplified request/response handling. |
response |
HTTP response helpers for parsing and error handling. |
tripperware |
HTTP client middleware for metrics, retries, logging, and more. |
httpserver |
HTTP server utilities with graceful shutdown, configuration, and middleware support. |
iofs |
Experimental: Generic utilities for reading and parsing files from various file systems, with format conversion helpers. |
kafka |
Kafka client utilities for producing, consuming, and monitoring Kafka messages. |
consumer |
Kafka consumer helpers and configuration. |
errors |
Kafka error helpers and error type definitions. |
producer |
Kafka producer helpers and configuration. |
stats |
Kafka statistics collection and monitoring utilities. |
kubemock |
Utilities for testing Kubernetes event-based applications with a fake client, supporting dry-run and error simulation. |
logging |
Structured logging with request tracing, logrus integration, and log data truncation. |
middleware |
HTTP middleware utilities for authorization, compression, logging, metrics, and more. |
authorization |
Authorization middleware for various authentication strategies (certificate, NPA, OAuth, trust, user). |
certificate |
Certificate-based authentication middleware. |
npa |
NPA-based authentication middleware. |
oauth |
OAuth-based authentication middleware. |
trust |
Trust-based authentication middleware. |
user |
User-based authentication middleware. |
gzip |
GZIP compression middleware for HTTP servers. |
logger |
Logging middleware for HTTP requests. |
metrics |
Prometheus metrics middleware for HTTP servers. |
requestid |
Middleware for injecting and propagating request IDs. |
timeout |
Middleware for enforcing request timeouts. |
opt |
Utilities for handling optional parameters with sensible defaults, emulating optional arguments in Go. |
orchestration |
Orchestration utilities for workflow-based resource management and automation. |
audit |
Audit logging for orchestration workflows. |
service |
Service interface for workflow-based resource orchestration, supporting validation, apply, and delete actions. |
status |
Status helpers for orchestration and workflow tracking. |
patch |
Utilities for applying JSON Patch and Merge Patch operations to Go structs, supporting RFC 6902/7386. |
reloader |
Watches files and reloads Kubernetes deployments on changes, supporting hot-reload of certificates and configs. |
retry |
Retry logic and backoff strategies for operations, with customizable policies. |
slices |
Generic utilities for working with slices, including unique, merge, and threadsafe operations. |
store |
Generic interfaces and implementations for key-value stores, with pluggable backends and middleware. |
backends |
Store backend implementations for various storage systems. |
configmap |
Kubernetes ConfigMap backend for key-value storage. |
fs |
Filesystem backend for persistent storage. |
http |
HTTP backend for remote key-value storage. |
client |
HTTP backend client implementation. |
server |
HTTP backend server implementation. |
kubernetes |
Kubernetes resource backend for key-value storage. |
labels |
Labels backend for key-value storage. |
memory |
In-memory backend for fast, ephemeral storage. |
replicate |
Replication backend for high-availability storage. |
s3 |
S3 backend for object storage. |
middleware |
Store middleware for caching, logging, metrics, and more. |
cache |
Cache middleware for stores. |
logger |
Logger middleware for stores. |
metrics |
Metrics middleware for stores. |
nameable |
Nameable middleware for stores. |
prefix |
Prefix middleware for stores. |
threadsafe |
Threadsafe middleware for stores. |
validatable |
Validatable middleware for stores. |
utilities |
Store utilities for advanced use cases. |
claimlock |
Claim lock utility for distributed locking. |
defaultmap |
DefaultMap utility for stores. |
timed |
Timed utility for time-based operations. |
task |
Workflow and job execution primitives for background and concurrent tasks, with support for chaining and retries. |
job |
Job execution primitives for background processing. |
runnable |
Runnable task primitives for concurrent execution. |
workflow |
Workflow execution primitives for chaining activities. |
tlsclient |
Utilities for creating and configuring TLS client configurations (mTLS, custom CAs, etc.), with certificate pool management. |
tlsserver |
Utilities for creating and configuring TLS server configurations (mTLS, client auth, etc.), with flexible validation. |
tlsutils |
TLS utility helpers for certificate and config management. |
trace |
Experimental: OpenTelemetry integration for distributed tracing, log correlation, and Gin instrumentation. |
utils |
Miscellaneous utility functions for context, networking, randomness, and signals. |
The access and middleware/authorization packages provide utilities for implementing fine-grained access control in
your applications using scopes. For this open source release we have tried to make the scopes as generic as possible,
but configuring them, e.g. via ginserver config, has room for improvements. We are working on improving the configuration
experience and documentation for these packages, but in the meantime feel free to explore the code and reach out if you
have any questions.
The orchestration package was the first initial open source offering. Below are details that were in the original readme.
The pkg/orchestration package is designed for concurrent synchronous Service calls. The package goes through several
stages for each Service:
- Check: Sanity check whether the
Servicerequest is likely to succeed - (Recover: Advanced usage to recover from a failing Check, used to recover from corrupted/illegal states)
- Run: Executes the
Servicerequest. AllServices must have passed theirCheckstage. - (Rollback: Is called for every
Servicewhen one or moreServicehas failed theirRunstage)
// Define your Service
var _ Service = &MyService{} // MyService implements Service
type MyService struct {
orchestration.Recoverable, // Implements the Recover func to satisfy Service interface
orchestration.Payload, // Implements the GenerateResponse func to satisfy Service interface
Datacenter string // For example, to generate some complexity
}
func (svc *MyService) Name() string { return "MyService" }
func (svc *MyService) Check(_ context.Context) error { ... }
func (svc *MyService) Run(_ context.Context) error { ... }
func (svc *MyService) Rollback(_ context.Context) error { ... }
func main() {
// Define the Services that you want to execute concurrently
services := []Service{ &MyService{Datacenter: "DC1"}, &MyService{Datacenter: "DC2"}}
// Call Check, Run, Rollback stages for each Service
// errs align with services, err[i] corresponds with service[i]. err[i] may be nil
// err is nil only when all errs are nil. Err contains information about which stage failed.
errs, err := CallServices(context.TODO(), services)
httpStatusCode, response := GenerateResponse(services, errs, err)
// 200, {"status":"ok","details":[{"name":"MyService","detail":"<your-response>"}}
}In the example above the services are executed in the following timeline:
-> | MyService{DC1}.Check | -> | MyService{DC1}.Run | -> Rollback Skipped
| MyService{DC2}.Check | | MyService{DC2}.Run |
The Service interface can be found in pkg/orchestration/api_service.go:
type Service interface {
Name() string
Check(ctx context.Context) error
Recover(ctx context.Context) error // Advanced usage, should return an error if not implemented
Run(ctx context.Context) error
Rollback(ctx context.Context) error
GetResponse(err error) interface{}
}Since the Service interface has no output, apart from the error, the output must be generated via your own
implementation. To help with this there is a Payload struct which has a response interface:
struct MyService {
Payload
}
func (svc *MyService) Run(_ context.Context) error {
svc.Payload.Response = "Good!"
return nil
}
func main() {
services := []Services{ &MyService{} }
errs, err := CallServices(context.TODO(), services)
httpStatusCode, response := GenerateResponse(services, errs, err)
// response: 200: {"status":"ok","details":[{"name":"MyService","detail":"Good!"}}
// or, on check stage error: 500: {"status":"one or more pre-run checks failed","details":[{"name":"MyService","detail":"some-error"}}
// or, on run stage error: 500: {"status":"one or more runs failed","details":[{"name":"MyService","detail":"some-error"}}
}A request payload should be contained in your Service implementation. Consider two types of payloads:
- A
Servicelocal payload - copied from the user's request, can be read/modified freely - A mutable payload shared across
Services (with an atomic lock)
type MyService struct {
LocalRequest MyServiceRequest // Can be read/modified freely
mutex.Lock // Owns SharedRequest
SharedRequest *MyServiceRequest // Shared state across Services, protected by Lock.
}In some scenarios it may be possible to recover from a failing Check. Recovery is only executed by CallServices when recover is set in the context:
type MyUpdateService struct {
Recoverable // Provides a MyUpdateService.Recovery function pointer
Request Spec
}
func (svc *MyUpdateService) Check(_ context.Context) error {
if !database.Has(svc.Request.Name) {
svc.Recovery = func(ctx context.Context) error {
return MyCreateService{Request: svc.Request}.Run(ctx)
}
return errors.new("cannot update " + svc.Request.Name + " because it is not found")
}
return nil
}When a Service has a rollback it is executed "in the background". Use the RollbackErrorReporter to handle rollback errors:
orchestration.RollbackErrorReporter = func(services []orchestration.Service, errs []error) {
for i := 0; i < len(services); i++ {
if errs[i] != nil {
log.Printf("Rollback failed for Service %s: %v", services[i].Name(), errs[i])
}
}
}When the dryRun flag is specified in the Context, CallServices only executes the Check stage:
ctx := context.WithValue(context.Background(), "dryRun", true)
_, _ = CallServices(ctx, []Services{ ... }) // Only calls Check stage for each ServiceREST APIs can be generically converted to a Service interface:
| Operation | Check | Run | Rollback |
|---|---|---|---|
| GET (Read/List) | None | Executes Get() | None |
| POST (Create) | Executes Get(), expects error | Executes Post() | Executes Delete() |
| PUT (Update) | Executes Get(), stores as backup |
Executes Put() | Executes Put() with backup |
| DELETE | Executes Get(), stores as backup |
Executes Delete() | Executes Create() with backup |
var request Nameable = &SomePostedPayload{}
name, err := request.Name()
var exampleApi RestApi = &MyExampleApi{}
svc := RestApiAsService(exampleApi, "Example Create", name, request)
errs, err := CallServices(context.TODO(), []Service{svc})When a Service depends on the output of another, use staged execution:
sharedState := &Example{mutex.Lock{}, State: SomeState}
stageNum, errs, err := CallStagedServices(context.TODO(), [][]Service{
{ // Stage 1
&DependencyServiceA{SharedState: sharedState},
&DependencyServiceB{SharedState: sharedState},
MakeDryRun(&OtherServiceA{SharedState: sharedState}),
MakeDryRun(&OtherServiceB{SharedState: sharedState}),
},
{ // Stage 2 - Executed after all Run's of stage 1 were successful
&OtherServiceA{SharedState: sharedState},
&OtherServiceB{SharedState: sharedState},
},
})This repository includes two example applications demonstrating the Create Memory Claim service:
- Service implementation: Fine-grained control over Check/Run/Rollback stages
- RestApi implementation: Automatically transformed to a Service with less code
# Successful create
$ curl http://localhost:8090/api/v1/memory -d @example-payloads/payload_ok.json
{"status":"ok","details":[
{"name":"MyService Create DC1_BLUE","detail":"ok"},
{"name":"MyService Create DC1_RED","detail":"ok"},
{"name":"MyService Create DC2_BLUE","detail":"ok"},
{"name":"MyService Create DC2_RED","detail":"ok"}
]}
# Duplicate create (Check stage catches conflict)
$ curl http://localhost:8090/api/v1/memory -d @example-payloads/payload_ok.json
{"status":"one or more pre-run checks failed","details":[
{"name":"MyService Create DC1_BLUE","detail":"already exists"},
...
]}
# Failed create with automatic Rollback
$ curl http://localhost:8090/api/v1/memory -d @example-payloads/payload_fail.json
{"status":"one or more runs failed","details":[
{"name":"MyService Create DC1_BLUE","detail":"ok"},
{"name":"MyService Create DC1_RED","detail":"not enough memory available"},
...
]}Contributions are welcome! Please open issues or pull requests. See CONTRIBUTING.md for guidelines.