Nunux Keeper

Nunux Keeper with Android

Tue, 03 Oct 2017 23:26:58 +0200

Content curation is a day to day activity. Due to that, you may expect an availability on your day to day device: your smartphone.

A good thing is that the Web App of Nunux Keeper try to be a Progressive Web App. Therefore you can appreciate a decent ergonomic on your smartphone. However the user experience is restricted to the browser capabilities. Even if the browser tend to be more and more integrated with the system (notifications, camera, etc), interactions between natives applications and web app are still limited.

With Android apps you can exchange data thanks to Intent mechanisms. So it’s possible to share a Tweet coming from the Twitter app with another app like Hangout.

How can we do that with Nunux Keeper? How can we keep a tweet without leaving the Twitter app?

The obvious solution is to create a new Android app that implements the share intent. Unfortunately I am not an Android developer. So I have to find out another solution.

By the way, if YOU are an Android developer and you enjoy Nunux Keeper… we have to talk :)

A solution came from a great Android app named: HTTP Shortcuts.

A simple Android app that allows you to create shortcuts that can be placed on your home screen. Each shortcut, when clicked, triggers an HTTP request.

It is very simple yet very powerful with many features among them: the ability to trigger an HTTP request on a share Intent. In other words, you can send a tweet to an HTTP endpoint. This is precisely what we want. We just have to create a shortcut to a Nunux Keeper endpoint and we will be able to create documents from any Android shared content.

Let’s do that!

The Nunux Keeper API is protected by OpenID Connect. That’s great but barely usable for a simple HTTP call. This is why some API endpoints are also exposed with a simple API key. It is the case for the document creation API endpoint. You can create a document with a simple HTTP call providing the API key as credentials.

You can obtain your API key from the setting panel of Nunux Keeper (or using the API). Think to copy paste your API key somewhere because this secret is stored hashed into the database. It is impossible to retrieve afterward.

Once your API key has been saved somewhere, you can call the API using:

$ curl https://api:c987ab4cf39a9d56cf8cb@api.nunux.org/keeper/v2/documents

Now it’s time to configure HTTP Shortcuts app. Let’s create a shortcut to send a URL to Nunux Keeper:

First we have to declare a Text Input variable called payload (Menu -> Variables). Don’t forget to check the Allow 'Share...' check box.

Once the variable has been created you can create a new shortcut:

Shortcut Name: Nunux Keeper (URL)
Method: POST
URL: https://api.nunux.org/keeper/v2/documents
Authentication Method: Basic Authentication
Username: api
Password: <YOUR API KEY>
Request Headers:
- Content-Type: application/json
Request Body: {"origin":"{{payload}}"}
Response Type: Simple Toast

The, you should be able to create a document from a URL shared by any Android app. This work very well when you share a page from your smartphone browser.

However, all Android apps don’t send URL as a sharing payload. For instance, the Twitter app send a short text with the link of the tweet. For this kind of app, we can directly send the payload as the document content. Let’s duplicate the shortcut and update it as following:

Shortcut Name: Nunux Keeper (content)
URL: https://api.nunux.org/keeper/v2/documents?title=Form my phone
Request Headers:
- Content-Type: text/html
Request Body: {{payload}}

And now we have two HTTP shortcuts able to create different type of content from our smartphone.

Of course it would be better to avoid having to know the intent payload type and be able to send whatever we get to the API endpoint. That’s possible, but you will need an intermediate tool like NodeRED, and this is another story…

Self-hosting Nunux Keeper

Wed, 17 May 2017 22:30:22 +0200

Before running any script, let’s have a quick overview of the software architecture. Many software are used to power the solution, and a first glance at the architecture may give you the feeling that the solution is a bit complex. You would be wrong.

A great approach to describe an architecture is a good drawing. An efficient schema model is the C4 model introduced by Simon Brown. This model is inspired by the geographic mapping. By analogy with Google Map you start at a global scale of the system: the context. Your system is summed up as a big box with short phrases explaining global features. All interactions with external systems or actors are also illustrated. Then you zoom in to an area of the map: the container level. It is the internal scaffolding of your systems. Which containers compounds the system and how they interact each other. Zoom in to the component level. It is the internal structure of a container. Finally you zoom again to see classes of a component. A class is a direct binding with the code.

Context, Container, Components and Class: Here come the C4 model.

By the way, if you enjoy this kind of diagram, feel free to download and adapt them. It is SVG format and it’s open source!

So let’s start with a overview of the system:

As you can see the system is straightforward: The system consumes web content of external systems and expose this content thru an REST API. This API handles main features that we can expect from a content curation system. Finally, the API can be consumed by other systems like a CLI, a Web App, etc.

Now, like using Google Map, we zoom in a bit to figure out how the system is structured.

Here the container diagram:

The core of the system is the API container. It is a Node.js app powered by Express. This container exposes a RESTFul API protected with JWT. The token creation is delegated to an external system. It can be forged by Auth0 but in our case it will be forged by a great open source IAM product: Keycloak.

The Core API stores web documents and meta data inside a NoSQL Data Store. This document data store can be MongoDB or ElasticSearch. The data store also persists other entity like labels, users and sharing informations.

In order to enable full text search, documents are indexed by a search engine: ElasticSearch.

Binary files attached to documents are stored inside an object storage container. This container can be a classic file system or S3 compatible object storage service. These files are downloaded by a job worker.

All asynchronous tasks are handled by Job Workers. This container implementation is based on Kue. This distributed job framework uses Redis as job queuing system. Job Workers are autonomous and can be deployed in parallel to handle the load. There are Job Workers to handle file downloads, import/export tasks and some administration task like database cleanup.

The Core API and some job workers produce metrics using the StatsD protocol. Any StatsD collector can be used to collect, aggregate and forward metrics to a Time Series Database (such as OpenTSDB, Prometheus or InfluxDB). For instance, the hosting platform of Nunux Keeper uses Telegraf and InfluxDB to handle those metrics.

Regarding the self-hosting need, we can stop here. You should now have a good understanding of Nunux keeper architecture.

Now, it is time to set up all of them. To handle this task, our best friend will be Docker and its new stack creation capability.

For this purpose we create a dedicated project: keeper-docker

Installation is simple:

$ git clone https://github.com/nunux-keeper/keeper-docker.git
$ cd keeper-docker
$ make deploy

With a bit of patience you will get the following services up and running:

Traefik: A dynamic reverse proxy used to route incoming requests to appropriate backend.
MongoDB: The database backend.
Elasticsearch: The search engine backend.
Redis: The in-memory database used as an event bus by the job scheduler.
Keycloak: The Identity and Access Management service. This service is auto configured by scripting.
Nunux Keeper Core API: The core API of Nunux Keeper.
Nunux Keeper job worker: A job worker for Nunux Keeper background tasks.
Nunux Keeper Web App: The Web App of Nunux Keeper.

Check the project repository README for more details about the installation and what could be missing to get a full and production ready installation.

Welcome in the wonderful world of the self hosting!

But if you don’t want to handle this you are welcome on our hosting platform.

Behind the portal

Tue, 28 Feb 2017 20:26:00 +0100

From static page to static portal

Any product should have at least a web page to describe briefly its purpose. Github or Gitlab help us with the README file of our repository. It is a convenient way to get quickly a public and referenced description of your product.

At the beginning, I wrote this cool README file mainly focus on technical aspects of the product. Write a good README file is not so easy. Fortunately there is some good articles to help you in the quest of writing README files.

However, README files are good for developers but not really useful for a non developer to figure out what does the product. That’s why we should not only focusing on the technical part and provide some product information to the end user. Therefore I added to the web application a welcome page:

Add a static page into the web app code is easy but I was not comfortable with. First I don’t like the idea to add static contents to the web app. I would like to get the tiniest deployable artifact in order to:

minimize the build time
minimize the global size of the app
minimize the user loading time
etc.

A second point is that the web app is powered by a dynamic web framework (React). As a result, first implementation of the welcome page was a React dynamic page. It means that the welcome page is loaded with javascript.

It is not annoying for the end user but it is not ideal to be correctly indexed by crawlers. In your quest of Search Engine Optimization (SEO) you have to make something else.

I could have made a dedicated fully static page but I would have complexified the build process. Nowadays, build a javascript web app is complex enough. Grunt/Gulp/Webpack are powerful tools, but I saw too many projects having a build configuration too complex and barely readable.

I think that the web app should be focused only on its own concerns: the app.

Therefore I need to delegate the marketing to a better tool: a static website generator.

Spoiled for choice

When you want to create a static web portal you have a huge choice! There are many good solutions to do the job:

Your choice can be influenced by your technological preferences. Ruby lovers will choose Jekyll, NodeJs lovers will choose Metalsmith, Java lovers will choose Maven Sites… I am kidding.

Personally I am doing almost everything with Docker therefore technology choice is not so important. What matter for me is simplicity. And maybe one of the most simple solution is Hugo.

Hugo is developed in Go. Ideal for installation. You have only one binary to put into your path. Obiously, you can still install it using your favorite distribution packaging system. Hugo is well distributed: Linux x86/amd64/ARM, OSX, Windows, FreeBSD… what is missing? Go is truly portable!

Maybe because it’s go, it’s fast… damn fast. In less than a second (47 ms to be precise) you have a development server up and running with live reload!

Another good point with Hugo is the layout. It is delegated to the theme and you have only one TOML file to configure Hugo and the Theme. No more kitchen sink to setup. Super simple.

The community is also very active and you can find a large variety of themes. For my needs I chose the universal theme that is very well designed. With few changes in the TOML configuration you quickly obtain a clean result.

Hugo is this kind of software that make you feel confident from the beginning.

The second objective of choosing a static website generator is to be able to add content over the time: To blog.

Some pages (such as the welcome page) are produced by the theme but one of the main feature of Hugo is to provide a static blog engine. Each articles are written in a markup language (Markdown) and converted into a static HTML page integrated and decorated by the theme. This article is a simple text file written in Markdown. It is very easy to create but furthermore to maintain, archive and work with. Your file system is your database and all your website can be managed through a version control system such as Git. It is super cool because it enables the ability of wrote a document in collaboration or make some review with merge/pull request.

This website is, as other keeper projects, an open source project hosted here. By the way, feel free to suggest an article or to contribute ;)

Unlike traditional CMS, static web site generators are great tools to create robust, secure, scalable, collaborative and simple websites.

What is content curation

Mon, 13 Feb 2017 23:16:34 +0100

It makes sense to start this product blog with a quick introduction of the concept behind this product. Nunux Keeper is introduced as a free content curation system. But what that means?

Definition

In a museum a curator (or a keeper) is the manager in charge of selecting the artworks and value their exhibits. Wikipedia provides us a far better definition.

In short it’s the action to select, edit and share content.

In the wonder world of Internet, this work is called content curation. It is no more related to art works but web content.

Internet produces billions of documents more or less relevant, more or less readable, more or less messed up with ads. Therefore, when you practice information watch, it can be difficult to read everything or organise yourself with all those informations. Content curation is a response to that problem. Manage your information watch. And Nunux Keeper is a tool to help you in that quest.

Nunux Keeper

I started this project in 2014 just after another project called Nunux Reader. A RSS feed aggregator born (like many other) just after the end of Google Reader. RSS takes a really big part into my daily technology watch. Still today. But, I quickly got to need to archive, organize, classify, retrieve and share the selected contents of this watch. At this time Pocket was launched and seemed to be perfect. And it was. But like many online products there are some drawbacks. Pocket don’t allow you to export your curation job. Pocket API is limited to retrieve only first lines of the document. It is obviously to protect Pocket. It’s seem to be fair for a cheap and powerful online service like Pocket but your are not free to dispose your data as you could expect. As usual you are the product and your data are precious not only for you but moreover for marketing. Most people don’t care about this kind of limitation, but not me. So I start this new project.

Yet another open source tool to do poorly the job that other product do with perfection \o/

Since 2014 Nunux Keeper have quietly evolve. And today comes the version 2.

Into the following articles I will try to present the product, explain technology choices, explains internals, and share ideas of improvement. I love to share my technology watch but I think it’s also time to share my nerdy work.