There are a bunch of panic statements scattered around right now. These should probably mostly be replaced with error return values, so that we only panic when it is really appropriate for the program to crash.