This project contains all the components needed to implement a Flask webserver providing the Dataflow APIs to a third party application.
NOTE: If you want to use the Keycloak-based topic autorization feature, please switch to the with-keycloack-authorization branch.
This modules interacts with:
- the Cloud Dataflow DB, to retrieve information about the active dataflows;
- the KSQLDB Server, to generate user specific topics;
- APISIX, to retrieve from a specific http header information about the user performing a request.
The yaml description of the APIs is provided in the api folder.
The endpoints in this section interact with the Cloud Dataflow DB to provide to the user information about the available dataflows.
The provided endpoints are:
- GET /dataflows/{dataflowId}, that returns the metadata of a specific dataflow.
- GET /dataflows/{dataType}/query, that returns the dataflowId of all the dataflows of a specific dataType; different query parameters are available in order to filter the dataflows.
- GET /dataflows/{dataType}/query/count, that returns the number of active dataflows with a specific dataType; different query parameters are available in order to filter the dataflows.
- GET /datatypes/{dataType}/properties, that return a JSON that, for each field of the dataflows metadata, shows the possible value that can be used as a filter.
- GET /datatypes/{quadkey}, returns the list of DataTypes available in a quadkey as a list of strings.
The endpoints in this section interact with the KSQLDB Server to allow to the third party application to manage its Kafka topics.
To identify the third party user, APISIX should add to the request an X-userinfo header, providing an "username" field.
The provided endpoints are:
- POST /topics/{dataType}/query, that generates a KAFKA topics and return the topicName to the user; different query parameters are available in order to filter the messages in the topic.
- GET /topics, that returns list of topicNames associated with the user.
- GET topics/{topicName}, that returns a JSON object containing the query that was used to generate the topic.
- DELETE topics/{topicName}, that deletes a specific topic.
- Federico Princiotto (federico.princiotto@linksfoundation.com)
#TODO