Kivik

Kivik v4.0 is out!

2023-10-26T00:00:00+02:00

Three and a half years since the last major release of Kivik, I’m happy to announce the latest major update to Kivik, version 4.0.

Kivik 4.0 includes a number of breaking changes compared to the 3.x series, as well as a number of other improvements and organizational changes.

Here are some highlights:

The iterator interface has been improved, particularly when passing multiple queries to a view.
The various official Kivik drivers have been integrated in to the main module.
Options are now functional options, rather than a plain map[string]interface{} as before.
An effort has been made to make different functions with similar result types more consistent.
Kivik now implements the CouchDB replication protocol. This means you can manage your own replications (in contrast to simply asking CouchDB to trigger a replication for you). This is most useful for backups or when bootstrapping your CouchDB instance, as you can now replicate between CouchDB and a filesystem!
The kivik command line tool has been added, which allows you to interact with CouchDB more easily from the command line. Tired of curling to configure and test CouchDB? Try the kivik CLI tool instead!

See the project README for a more complete list of breaking changes, especially if you are upgrading from Kivik 3.

Of course, just because Kivik 4 is out, doesn’t mean I’m out of ideas for improving it. I still have big plans, and will continue to chip away at them as time permits. A few things I hope to make progress on in the coming months:

Support for full text search
Continue to flesh out the filesystem and in-memory drivers
Improve the replication protocol implementation. The current version works, but doesn’t use all of the efficiency gains made possible with a full implementation.

And of course, I’ll continue to respond to bug reports and feature requests, as I receive them.

Thanks for using Kivik and CouchDB!

Kivik v4.0.0-rc1 released

2023-09-26T00:00:00+02:00

I’m happy to announce a new major release of Kivik is on the way. v4.0.0-rc1 has been released. Anyone interested is encouraged to test this release, and report any bugs before a final release is published, within a week or two.

This is a major release, and does represent a breaking change, and includes several improvements as well. Here is a brief overview of the main highlights:

The multiple Kivik repositories have been integrated into the main repo
Options are now functional options, rather than a plain map[string]interface{}
The API for iterating over query results has been improved and made more consistent across various methods

See the project README for a more complete list of breaking changes.

New kivik CLI tool

2021-04-27T00:00:00+02:00

I’m happy to announce a new experimental CLI tool, kivik, which is designed to simplify interaction with CouchDB servers.

Feel free to grab the most recent binaries from the releases page and try it out, or read the documentation.

Motivation

While debugging, experimenting with, or administrating CouchDB, I found the repetitive use of long curl commands to be toilsome. I’ve also found the flexibliyt of tools like kubectl to be informative (i.e. supporting both JSON, YAML, and arbitrary output formats). And finally, I have often longed for a simple CouchDB analog to tools like pg_dump and pg_restore.

This tool aims to satisfy all of these itches.

Key features

Easier to use than curl for interacting with CouchDB
Supports both JSON and YAML inputs
Supports JSON, YAML, and custom template outpus
Allows replication between filesystem directories and CouchDB servers
Supports retrys, to easily detect when a server becomes available

Development status

While I believe the features most likely to be useful are implemented, some features are still completely unimplemented (see the TODO list). There are also likely bugs, as well as inconsistencies. Bug reports are very welcome!

All feedback is welcome!

Kivik v3.2.0 released with cluster membership support

2020-07-10T00:00:00+02:00

Today, Kivik 3.2.0 was tagged and released. This release adds a single minor feature: the Membership() method, which allows access to the /_membership endpoint on CouchDB 2.x and newer.

I needed this for a project at work, where we need to programmatically query all nodes in cluster.

I initially opted not to include this in Kivik, as it felt more like a system-administration task, but my mind was changed when I needed it in production.

The packages affected by this update:

kivik - v3.2.0
couchdb - v3.2.0
kivikmock - v3.2.0

Kivik v3.1.0 released with partitioned queries

2020-05-02T00:00:00+02:00

Today, Kivik 3.1.0 is finally tagged and released. This release adds two major features, and punts on a third:

Partitioned queries, as introduced with CouchDB 3.0 are now possible.

This did require a small breaking change, as Find(), Explain(), CreateIndex(), GetIndexes(), and DeleteIndex() now take optional (variadic) options. For normal usage, no change is necessary, but if you’ve defined an interface with these methods, or otherwise depend on the exact function signature, your code may need to be updated to use this new version.
Kivik and the CouchDB driver now support sending multiple queries to a view with a single request. Although this feature has existed in some form or another for ages (since at least version 1.7–the oldest I tested), I had never added support to Kivik.

And finally, I have decided to punt on one additional major feature added to CouchDB 3.0: full text search. The reason this isn’t covered in this release is simply that I don’t have a convenient way to test it. The official distribution cannot be deployed in a simple Docker container, so setting up a local test environment would be non-trivial, and automated testing would be very difficult.

Having said that, I would welcome a pull request, if you have the means to add and test this functionality.

The packages affected by this update:

kivik - v3.1.0
couchdb - v3.1.0
kivikmock - v3.1.0
kiviktest - v3.1.0

The pouchdb driver is unaffected by these changes, and remains at v3.0.x for now.

Upcoming Kivik changes for CouchDB 3.x

2020-02-29T00:00:00+01:00

Earlier this week CouchDB 3.0.0 was released! This represents a big step forward, with several exciting new features.

Good news and bad news

The good news for Kivik users is that the Kivik 3.0.0 integration test suite passes with flying colors against CouchDB 3.0.0, with no changes at all. This means that if you’re using Kivik 3.x (or 2.x) and choose to upgrade your CouchDB installation to 3.0.0, you won’t need to make any changes to your client code.

The bad news, though, is that none of the new API endpoints of CouchDB 3.0.0 are yet supported by Kivik. But I aim to remedy that quickly with a release of Kivik 3.1.0. Any features of CouchDB 3.0.0 that do not require API endpoint changes should work automatically with Kivik, though they have not been explicitly tested. If you discover something is not working, please report a bug.

Upcoming changes to Kivik

Here is a high-level outline of the new features in CouchDB, and plans for inclusion in Kivik. Your feedback is welcome, to help shape these new features of Kivik (please comment freely on the linked GitHub issues).

Multiple query requests – Kivik issue #447.

Kivik currently has four view methods: Query(), AllDocs(), LocalDocs() and DesignDocs(). Duplicating all of these to include multi-query variants seems like a heavy API change, for what is likely to be a rarely-used feature. For this reason, the two options I’m most seriously considering at the moment are:
- Deprecate AllDocs(), LocalDocs(), and DesignDocs(), making it possible to query these views via Query() (possibly by specifying an empty ddoc argument, and passing _all_docs, _local, or _design, respectively, as the view name). Then adding a single Queries() method will seem natural.
- Adding the same functionality as above, but keeping at least AllDocs(), and possibly LocalDocs(), and DesignDocs() as simple convenience functions. This seems attractive, given that AllDocs() is one of the most commonly used methods in Kivik code I have seen.
Please leave your opinion, or alternate suggestions, on the Kivik issue.
Partitioned queries – Kivik issue #448

The semantics of new partitioned queries are identical to the existing view queries, they just require one additional URL component: _partition/{partition}. Three possible approaches come to mind:
1. Create new Partition variants of each query metho. e.g. AllDocsPartition, QueryPartition, etc.
2. Accept the partition as part of the view ID. e.g. Query(ctx, "ddoc", "_partition/123/view") This will only work with Query(), but in conjunction with the proposed changes to support multiple query requests, this could work.
3. Accepting a partition option. e.g. AllDocs(kivik.Options{"_partition":"123})
I am leaning toward the third option. I believe it has two compelling advantages:
- It doesn’t clutter the Kivik API with any new methods.
- It would be 100% backward compatible with the current API.
And one minor disadvantage:
- Using an option to change the fundamental behavior of a query seems a bit icky. But it’s not entirely without precedent.
I believe the most natural way to implement this would be by adding a new constant to the couchdb driver package:
```
  // Partition specifies the name of a partition, and causes a partition-bound query.
  Partition = "kivik:partition"
```
Full Text Search - Kivik issue #450

I believe this will be pretty straight forward. I intend to add three new methods:
- Search(), the analog to Query() and Find(), will be used to perform search queries.
- SearchInfo() will be used for the _search_info endpoint
- SearchAnalyze will be the analog to Explain for mango queries.

And finally, a couple of new features I probably won’t be implementing in Kivik, because they are primarily used for maintenance, and thus the value of a Go SDK seems dubious. But if you feel otherwise, please let me know by commenting on the appropriate issue in GitHub, and I will be happy to re-consider.

That sums up the upcoming changes for Kivik 3.1.0. Depending on time, and the complexity of implementing the multi-query requests, I anticipate having a release ready by mid-April.

Opinions, comments, and of course, pull requests, are welcome from all!

Kivik v3.0.0 released

2020-02-16T00:00:00+01:00

Today, Kivik 3.0.0 is finally tagged and stable, with only one minor bug fix since the final pre-release announced in December.

kivik – v3.0.0
couchdb - v3.0.0
pouchdb – v3.0.0
kivikmock - v3.0.0
kiviktest - v3.0.0

Kivik v3.0.0 is functionally identical to v2.0.0. The only difference is that it has been configured to support Go modules. As such, Go 1.9 and 1.10 are no longer supported, as they do not support Go modules. See the v2 release notes for a list of changes from v1.

For any projects compiled with GopherJS, the new dependence on Go modules means a little extra effort is needed. I have detailed this in a brief tutorial on Using GopherJS with Go Modules.

Kivik v2.0.0 released

2020-02-09T00:00:00+01:00

Today, Kivik 2.0.0 is finally tagged and stable, with only one minor bug fix since the final pre-release announced in December.

kivik – v2.0.0
couchdb - v2.0.0
pouchdb – v2.0.0
kivikmock - v2.0.0
kiviktest - v2.0.0

Highlights for this release are:

An improved API.
Support for many newer features of CouchDB, such as the _local_docs, _design_docs,, and _ping endpoints.
Extended support for existing features of CouchDB, such as _bulk_get, multi-part get requests, improved support for replication features, etc.
A number of bug fixes.

A more extensive (but by no means exhaustive) list of changes follows.

Breaking changes:

New() no longer takes a context.Context argument.
Rows.Key() now returns the raw JSON for the key, rather than unquoting it.
The kivik.Method* and kivik.Status* constants were removed. Use http.Method* and http.Status* instead.
Client.DB() now defers errors until the next method call.
DB.Get() now defers errors until the next method call.
Client.DBUpdates() now takes a context.Context argument.
DB.BulkDocs() now takes a slice of documents, rather than an empty interface.
DB.GetAttachmentMeta() no longer takes a rev argument, which was redundant with options.
The Attachment type is completely re-designed.
Replaced the DB.Rev() method with DB.GetMeta().

Additions:

Added the DB.Err() method to explicitly check a deferred error.
The Attachment type now includes the RevPos and Digest fields, needed for replication.
Kivik-generated errors now provide the Unwrap method, introduced in Go 1.13.
Added the DB.RevsDiff() method, needed for replication.
Added the Client.Config(), Client.ConfigSection(), Client.ConfigValue(), Client.SetConfigValue(), and Client.DeleteConfigKey() methods, to manage server configuration.
Changes feed now reports the ETag.
Added the Client.Close() and DB.Close() methods.
Added the Client.ClusterSetup() and Client.ClusterStatus() methods.
Added the Client.Ping() method.
Added the DB.BulkGet() method.
Added the DB.LocalDocs() and DB.DesignDocs() methods.
Added the DB.Purge() method.
Added the Client.DBsStats() method.
Added the experimental AttachmentsIterator type, for use with multi-part get requests.

Kivik 3.0.0 should follow very shortly (within hours or days), and will be identical to Kivik 2.0.0, except that it will be configured for use with Go modules.

(Probably) final 2.0.0 pre-release, and v3 announcement

2019-12-16T00:00:00+01:00

Today I tagged what I expect to be the final pre-release of all core Kivik packages:

kivik – v2.0.0-pre3
couchdb - v2.0.0-pre4
pouchdb – v2.0.0-pre3
kivikmock - v2.0.0-pre3 (unchanged)
kiviktest - v2.0.0-pre4

I expect to tag the final v2.0.0 version within a couple of weeks, unless bug reports come in.

Here I would also like to announce that at roughly the same time, I intend to release v3.0.0. v3.0.0 will be identical to v2.0.0, except that it will support Go modules. I could make that change along with the v2.0.0 release, but this could stand to break anyone using a v2.0.0 pre-release, and although that wouldn’t strictly violate the compatibility guarantee, since these are, after all, pre-releases, I believe it will be the smoothest upgrade path. And version numbers are free, after all!

The Future of Kivik

2019-07-10T00:00:00+02:00

With a stable release of Kivik 2.0 coming soon, it seemed like a good time to share my vision of the Kivik future. This also comes after a brief conversation with a small number of would-be open-source contributors, looking for guidance in contributing to the project. I feel the first step toward that goal is to explain my vision of the future.

Some of what I will discuss here was already mentioned back in 2017 when Kivik was first announced. But as one would expect over 2.5 years, it has become more specific, and some things have changed–not least of all, the current state of Kivik.

Modular design

From the beginning, I’ve had the idea of making Kivik a modular project. This is probably most apparent in the way that drivers are interchangeable, but this modularity can extend to consumers of Kivik, as well.

✅ = Exists or possible today

In the end, my vision is that Kivik may provide a sort of swiss army knife for building CouchDB tools and applications. A few possible applications for Kivik, once this goal is reached, could include:

The ability to build apps in Go/GopherJS, which use CouchDB for database storage. This is possible today.
Easily bootstrap CouchDB instnaces. Historically we have had a number of ad-hoc tools to do this, including couchapp and others. None have felt very feature rich or well maintained to me (if I’m wrong, please let me know!)
Run an in-memory database for testing purposes. With the addition of an HTTP server for Kivik, the application being tested doesn’t even need to be written in Go.
Dump files from CouchDB to your local disk as JSON or YAML files
A nice command-line tool to interact with CouchDB from the console, or in shell scripts.
Build a custom HTTP proxy in Go, to handle authentication or other custom logic.
And maybe even eventually, with additional backend stores, maybe even put a CouchDB façade on top of MySQL, S3, or DynamoDB, for applications with low performance requirements.

From the chart above, I hope it is apparent how these outcomes, and likely many others, should be possible. But there is a lot to be done, so where to start?

Tooling

The priority for me lately, has been to focus on tooling. I recently finished the kivikmock library to facilitate testing, which has been a big boost to productivity (no longer relying on a Docker install of CouchDB for all tests).

I’ve also recently added the bare essentials to the fsdb driver to allow replicating from an on-disk collection of JSON/YAML files to a live CouchDB server. This has made bootstraping new databases many times easier for me. But there’s much room for improvement.

My next mid-term goal is to build a command-line tool which provides a simple interface to CouchDB and the Filesystem driver. I actually began an experimental version of this project here, but after initial learnings, intend to start from scratch.

Initial project goals include:

The ability to easily GET and PUT individual documents and attachments.
The ability to easily query views
The ability to read and set security objects
The ability to configure server parameters
The ability to dump a database via replication to the filesystem
The ability to bootstrap a database, via replication from the filesystem
The ability to consume and produce JSON and YAML, translating to JSON when sending to CouchDB

Ideally, the tool should make any manual interaction with CouchDB easier than it presently is with cURL, and also add the option to replicate to/from the filesystem. Naturally, this means the Filesystem driver needs to be fleshed out.

HTTP Server

A longer-term goal is to build a fully-capable HTTP server driven by Kivik. Some basic functionality along these lines already exists in the kivikd package, but it’s far from complete.

Once the Filesystem driver is complete, adding a full HTTP server in front will make it possible to run a stand-alone Kivik server, which could stand in for a CouchDB server for testing or light-weight loads. I find this prospect especially exciting once the Memory driver is complete, as it would allow any developer, in any language, to run a small, self-contained memory-only CouchDB instance for testing, either on their local machine, or in a CI/CD pipeline.

This also has long-term possibilities with the addition of future storage backends for Kivik. Suppose a DynamoDB backend is written, then certain small apps might be able to take advantage of CouchDB’s replication protocol, but using AWS-managed infrastructure, rather than managing their own CouchDB installation.

Other Possibilities

The logical implications of a modular system like Kivik I think are broad. No doubt there are possible applications I have not considered. Rather than documenting all possible use cases, I think the prudent course is to focus on the areas with the greatest possible benefit in the short-term.

Help Wanted

Up to now, this has been driven by my own needs for testing and tooling, and I have been the primary contributor.

But now I’d love to get some greater community involvement, particularly in choosing the area of focus.

If this project intrigues you, here are some areas where some help could go a long way:

Discussion No coding necessary! If you are a current user of Kivik, or find the future of Kivik promising, your input can help shape the direction. The CLI needs to be designed, and direction on the next features needs to be decided. You can weigh in!
Documentation I’ve tried to keep things fully documented, but there’s plenty of room for additional documentation, especially with regard to code samples on the wiki.
Coding If you’re a Go coder, and think this project sounds interesting, your help could be very valuable in bringing this vision to reality.

To get involved, you can submit an issue or stop by the #kivik channel on Gophers Slack.