anti-fragile infrastructure

DEADBEEF_SOCIETY.

who?

  • dlg@uq.edu.au
  • dlg@openbsd.org

what?

fragile
easily broken or damaged
infrastructure
the basic physical and organizational structures and facilities ... needed for the operation of a society or enterprise

how?

  • prepare for change
  • prepare for failure
  • prepare for attack

preparing for change

  • arrange stacks in layers
  • avoid loops
  • use standards between layers
  • always have a way to roll back
  • understand the consistency/availability of each service

Semantic Versioning

http://semver.org/

Summary

Given a version number MAJOR.MINOR.PATCH, increment the:

  1. MAJOR version when you make incompatible API changes,
  2. MINOR version when you add functionality in a backwards-compatible, and
  3. PATCH version when you make backwards-compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

EAIT PostgreSQL

  • Single write master
  • Read-only slaves on each web server
  • No automatic failover or promotion
  • Read access is highly available
  • Writes can fail while the master is unavailable

EAIT PostgreSQL Upgrade

  1. Disable access to the master
  2. Dump master database
  3. Upgrade master software
  4. Disable first slave
  5. Upgrade first slave
  6. Reconfigure first slave replication
  7. Repeat slave disable, upgrade, reconfigure
  8. Enable access to the master

preparing for failure

  • fragility is inherited from the weakest component
  • ... research the bits you use
  • ... be aware of argumentum ad populum
  • ... be conscientious about the bits you make
  • understand CAP
  • partial failures suck

OpenBSD firewalls

  • config is synced between firewalls
  • routing is stateless
  • ... can fail over or split traffic
  • carp lets two firewalls act as a single gw
  • ... active/backup
  • pf is stateful
  • ... pfsync supports failover

OpenBSD firewall failure

  • passive fw failure - who cares
  • active fw failure - passive peer becomes active
  • carp becomes master
  • pfsync keeps states going
  • partial failures suck

Being Conscientious

  • every line of code has consequences
  • you must handle failure
  • complete failure is straightforward
  • partial failure is subtle

preparing for attack

  • simple things are easier to defend
  • you cant lose what you dont have
  • default deny
  • restrict privilege as much as possible
  • compartmentalise
  • crypto is a tool, not magic

questions?

i'm on irc

programming in C

  • the easy answer is: don't
  • the real answer is: it's complicated
  • programming in C is a lot of responsibility
  • ... more than just getting it to compile
  • ... more than just getting it to run once
  • ... the programmer has to do a lot of work

c is a sharp knife

  • most OSes are written in C
  • so if you tweak the OS you do it in C
  • C is the best interface to what they provide
  • everything else is a lo fi translation to the C interface
  • no abstraction away from what the machine is doing
  • ... effectively complete control over failure handling
  • understanding C lets you understand your system
  • pity about all the bleeding though

working in C

  • c is a very simple language, but...
  • a lot of cognitive load
  • ... explicit memory management
  • ... concurrency is hard
  • ... portability isn't easy

explicit memory management

  • memory accesses are unbounded
  • ... buffers must have a terminator
  • ... or you have to know how big it is at all times
  • strlcpy vs strncpy vs strcpy vs memcpy
  • you have to know what the best interface is

explicit memory management

  • memory allocation can fail
  • ... always check for NULL
  • ... or have a wrapper that exits/aborts
  • ... realloc is subtle
  • ... calloc is subtle
  • memory must be manually freed
  • ... free() can take NULL

concurrency is hard

  • c programs are a single execution context
  • locks are hard
  • event driven programming isn't natural
  • ... but in my experience, the less of two evils
  • maybe Threads without Locks

portability isnt easy

  • an int isn't safe to put on the wire
  • types are different between platforms
  • ... 32bit vs 64bit, ILP32 vs I32LP64
  • ... endiannes varies
  • ... different alignment requirements
  • ... char is signed on powerpc
  • ... #ifdefs in code are error prone

portability is easy

  • use stdint.h
  • pick a single platform to develop on
  • emulate or provide compat to it everywhere else
  • use gcc packed (and aligned) attributes
  • ... or memcpy bits in/out of buffers

bonus tips

  • never allow user controlled format strings
  • privsep
  • priv revocation
  • use process boundaries
  • re-exec to get alsr benefits
  • a consistent style makes code easier to maintain
  • use const pointers as much as possible
  • read good code
  • copy good code

OpenBSD

  • a unix-like operating system written mostly in C
  • a focus on correctness and security
  • generally uses libevent
  • a good platform to develop on
  • ... tweaked toolchain
  • ... aggressive memory protections
  • ... aggressive randomness
  • ... existing libopenbsd compat

OpenBSD

  • allocator and fread prevent int overflow
  • hardened malloc, mmap, guard pages, pools
  • propolice, stackgap, stackghost, -fstack-shuffle
  • W^X, kernel W^X
  • random shlibs, PIE, ELF
  • enabled by default

Further Reading