Using Swagger to tame HTTP/JSON
interfaces
John Billings
billings@yelp.com
Yelps Mission:
Connecting people with great
local businesses.
Yelp Stats:
As of Q3 2015
89M 90M 71% 32
HTTP/JSON is amazing!
HAProxy
requests Apache
curl
NGINX simplejson
Varnish
jq
httplib Pyramid
Dropwizard http://wallakitty.deviantart.com/art/Unicorn-attack-519106761
HTTP/JSON is amazing!
HAProxy
requests Apache
curl
NGINX simplejson
Varnish
jq
httplib Pyramid
Dropwizard
The website is
down again!
%&*#! I just
pushed the
petstore service!
The pet resource
takes a string,
right?
No, I told you,
its an int!
Curse you and
your strings!
If only there were a
better way to
specify our API?
Option 1: Write spec docs
Option 1: Write spec docs
Its easy to get started
People can comment if you use e.g. gdocs
Approachable by non-technical individuals
Implementation and spec can drift over time
Its easy to be imprecise
Option 2: Switch to Thrift / Protocol Buffers /
Avro / ...
namespace java ns
namespace py ns
typedef i32 int
service MultiplicationService
{
int multiply(1:int n1, 2:int n2),
}
Option 2: Switch to Thrift / Protocol Buffers /
Avro / ...
More efficient on the wire
More efficient to decode than JSON
Cannot use L7 technologies such as HTTP caching
Difficult to debug on the wire
Variable quality of support across languages?
Option 3: Write lots of integration tests
The tests become the de facto spec
As a client, if I send this request to the service, then I
should get back this response.
Option 3: Write lots of integration tests
You should already have (some) of these tests
Final testing phase; slow to correct bugs at this stage
Integration tests take a (relatively) long time to run
Overall, probably only want to have a few of these?
Option 4: Write client libraries
The client library API becomes the spec for consumers
Option 4: Write client libraries
Consumers dont need to worry about wire protocol
Weve used this approach at Yelp, and it can work
Lots of boilerplate
Manual validation
No spec for the wire protocol
Still need integration tests from clientlib / service ifc
Or...
Stick with our existing HTTP/JSON infrastructure
Invent a machine-readable specification language to
declaratively specify endpoints and return types
Create tooling to generate client libs from specs
Create tooling to perform server-side validation
against endpoint specifications
Create a vibrant open source community :)
http://swagger.io/specification
A brief history of Swagger
2011-08-10 Version 1
2012-08-22 Version 1.1
2014-03-14 Version 1.2
Formal swagger specification document
2014-09-08 Version 2
Combine Resource Listings and API Declarations
2016-01-01 OpenAPI Specification
Supported by Google, Microsoft, IBM and others
Petstore Try this out!
Top-level Swagger spec
Path
objects
http://petstore.swagger.io/v2/swagger.yaml
Definition
objects
Paths object
path
path parameter
parameter object
reference to a definition,
can split across files
if needed
Another parameter object
Used for /pet/findByStatus endpoint
Definition object
Swagger 2
More definition objects: Maps only
Datatypes and formats
Custom formats
Ignored by Swagger, but some tooling may
allow you to register your own validator
Where do Swagger specs live?
At Yelp we check them into the service codebase
Serve from a well-known endpoint of the service
This minimizes distance between spec and code
Could also store all specs in a central repo
Modifying specs
Theres no magic here
Swagger will not prevent you doing something bad
You-the-programmer need to make sure that all spec
changes are backwards compatible
If you like living safely, only add new endpoints
If you like living dangerously, change some existing
endpoints or remove some endpoints :)
A brief interlude
What's the best thing about UDP jokes?
A brief interlude
What's the best thing about UDP jokes?
I don't care if you get them
A brief interlude
What's the best thing about TCP jokes?
A brief interlude
What's the best thing about TCP jokes?
I get to keep telling them until you get them
A brief interlude
What's the best thing about TCP jokes?
What can I do with a spec?
Review an API
Browse other specs
Generate a client library
Perform server-side validation
Testing
API reviews
Browsing specs
API for
selected
service
Different
services
http://swagger.io/swagger-ui/
Perform a real query
Brief aside: Same-origin policy
swagger_ui service_1 service_2
Solution using Cross-Origin Resource
Sharing
swagger_ui service_1 service_2
Access-Control-Allow-Origin:
http://swagger_ui
Solution using a proxy
swagger_ui service_1 service_2
NGINX
Generating client libs
Try this out!
Using generated clientlibs
Try this out!
Bravado: dynamic clientlibs
for Python Try this out!
https://github.com/Yelp/bravado
pyramid_swagger
https://github.com/striglia/pyramid_swagger
pyramid_swagger: usage
Matched in swagger spec
pyramid_swagger: custom formats
Oops!
Oops!
Testing without Swagger
Client Clientlib Service
There could be inconsistencies across both of
these interfaces
Testing with Swagger
This interface is consistent
by construction (*)
Client Generated Swagger Service
Clientlib Spec
There could still be inconsistencies
across these interfaces
Testing with Swagger
Client Generated
Clientlib
This is a fairly standard testing problem
Your type-checker can help here (if you have one :)
Future work: add support for returning mock data
Testing with Swagger
Swagger Service
Spec
Validate your responses as part of your testing
Fairly easy if your service already contains a validator?
Could also use an external validator
SwaggerHub
Other spec langs: API Blueprint by Apiary
Other spec langs: I/O Docs by Mashery
Conclusions
Swagger provides an easy way to define JSON/HTTP
interfaces for new and existing services
Once you have an interface, you get lots of tooling
for free
Automatic generation of clientlibs for many
different languages
Automatic validation of requests and responses
Any questions?
