Practical operability techniques for teams - Matthew Skelton - Agil...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Practical Operability
Techniques for Teams
Matthew Skelton
Skelton Thatcher Consulting
skeltonthatcher.com / @SkeltonThatc...

Today
What is operability?
Modern logging
Run Book dialogue sheets
Endpoint healthchecks
Correlation IDs
User Personas for...

You
Software Developer
Tester / QA
DevOps Engineer
Team Leader
Head of Department

Operability:
use modern logging, Run Book
dialogue sheets, endpoint
healthchecks, correlation IDs,
and user personas as
te...

About me
Matthew Skelton
@matthewpskelton
Co-founder at
Skelton Thatcher Consulting
skeltonthatcher.com

Team Guide to
Software Operability
Matthew Skelton & Rob Thatcher
skeltonthatcher.com/publications
Download a free sample ...

Team Guide series
#Operability #BusinessMetrics #Testability #Releasability
skeltonthatcher.com/publications

Practical Operability
Techniques for Teams

What is operability?

Operability
making software work well
in Production

Logging with Event IDs

logging with Event IDs
reduce time to detect problems
increase team engagement
increase configurability
enhance collaborat...

search by
event
Event ID
{Delivered,
InTransit,
Arrived}

transaction
trace
Correlation ID
612999958…

How many distinct event
types (state transitions)
in your application?

represent distinct states

enum
Human-readable sets:
unique values, sparse,
immutable
C#, Java, Python, node
(Ruby, PHP, …)

Technical
Domain
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred...

BasketItemAdded = 60001
BasketItemRemoved = 60002

log using Event IDs
with a modern
‘structured logging’ library

example:
https://github.com/EqualExperts/opslogger
Sean Reilly
@seanjreilly

Example: video processing
On-demand processing of TV and
mobile streaming adverts
Ad-agency  TV broadcaster
High throughp...

Storage I/O
Worker Job
Queue
Upload

Example: video processing
Discover processing bottlenecks
Trigger alerts via LogEntries /
HostedGraphite
Report on KPIs
Ta...

http://honeycomb.tv/

Run Book dialogue sheets

Run Book dialogue sheets
Checklists for typical operational
considerations
Team-friendly exploration

runbooktemplate.infoRun Book dialogue sheets

System characteristics
Hours of operation
During what hours does the service or system actually need to operate? Can porti...

http://runbooktemplate.info/

runbooktemplate.infoRun Book dialogue sheets

Endpoint healthchecks

endpoint healthchecks
Every runnable app/service/daemon
exposes /status/health
An HTTP GET to the endpoint returns:
200 – ...

endpoint healthchecks
Each component is responsible for
determining its own health and
viability – this is very contextual

endpoint healthchecks
Use JSON as a response type –
parsable by both
machines and humans!

endpoint healthchecks
For databases and other non-HTTP
components, run a lightweight HTTP
service in front of the componen...

Helper service

https://github.com/Lugribossk/simple-dashboard

Correlation IDs

‘Unique-ish’ identifier for each request
Passed through downstream layers

Unique-ish ID

Synchronous HTTP:
X-HEADER e.g. X-trace-id
X-trace-id: 348e1cf8
If header is present, pass it on
(Yes, RFC6648, but this i...

Asynchonous (queues, etc.):
Message Attributes, name:value pair
e.g. "trace-id":"348e1cf8"
AWS SQS: SendMessage() / Receiv...

Example: electronic trading
High speed, low latency
Trading options & derivatives
Connected to stock exchanges
Sub-millise...

Correlations IDs for trading
Evidence for timely operation
Help identify bottlenecks
Target areas for perf tuning
Identify...

Example: OpenTracing / PCF
3 tracing elements:
TraceID, SpanID, ParentSpan
"X-B3-TraceId" "X-B3-SpanId"
"X-B3-ParentSpan"

Example: OpenTracing / PCF
Always log the TraceID as-is
Log calling SpanID as ParentSpan
Log new SpanID

Trace
Span
ParentSpan

Lightweight user personas

Lightweight user personas:
Ops Engineer
Test Engineer
Build & Deployment Engineer
Service Owner

Lightweight user personas:
Consider the User Experience (UX) of
engineers and team members using
and working with the soft...

http://www.keepitusable.com/blog/?tag=alan-cooper

https://www.geckoboard.com/blog/visualisation-upgrades-
progressing-towards-a-more-useful-and-beautiful-dashboard/

Lightweight user personas:
What data does the User Persona
need visible on a dashboard in order
to make decisions rapidly ...

Summary

Operability
making software work well
in Production

logging with Event IDs
use enum-based Event IDs to
explore runtime behaviour
and fault conditions

Run Book dialogue sheets
explore and establish
operational requirements as
a team, around a physical
table, together

endpoint healthchecks
HTTP 200 / 500 responses to
/status/health call with
JSON details – good for
tools and humans

Correlation IDs
trace execution using
correlation IDs:
synchronous (HTTP X-trace-id)
async (SQS MessageAttribute)

lightweight user personas
explore the UX and needs of
different roles for rapid
decisions via dashboards

Operability
use modern logging, Run Book
dialogue sheets, endpoint
healthchecks, correlation IDs,
and user personas as
tea...

Team Guide to
Software Operability
Matthew Skelton & Rob Thatcher
skeltonthatcher.com/publications
Download a free sample ...

Team Guide series
#Operability #BusinessMetrics #Testability #Releasability
skeltonthatcher.com/publications

Resources
• Team Guide to Software Operability by Matthew Skelton and Rob
Thatcher (Skelton Thatcher Publications, 2016)
h...

Questions?
Twitter: @SkeltonThatcher | #operability
email: questions@skeltonthatcher.com

thank you
@SkeltonThatcher
skeltonthatcher.com

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Continuous Delivery for databases -... by Skelton Thatcher ... 10120 views

View on Slideshare

1 of 83 Ad

View on Slideshare

1 of 83 Ad