• Managing large Linux installations. Performance tuning and database analysis. Releasing things without breaking other things.
• bash, python, SQL, sed/awk, perl, dpkg, and all the usual things.
• I am looking to work with a fun group of people in a socially responsible company with a strong sense of mission. I like scalability problems.

Responsible for several sections of the backend spam analytics engine, as well as taking the lead on converting our production clusters to use puppet and other automated deployment systems.

• Deployed and managed spamnet backend systems. Wrote tools in perl for deploying and managing configs.
• Replaced the entire high-volume inbound mail feedback apparatus with a Cloudmark Gateway cluster, with zero downtime or service degradation. Performed load testing and capacity planning.
• Puppetized and automated large swaths of production servers and internal products. Replaced several 50-page setup documents with a puppet module and deploy script apiece.
• Replaced inbound sendmail relay with Cloudmark Gateway cluster. Served as in-house customer and QA for Gateway development team. Set up a MySQL user database for user management, wrote user management scripts for IT.
• On call once a month, for servers and networks in four datacenters, two continents, and two CDN providers.
• Used VMware, iptables, apache, etc

systems engineer, Shopkick (3/2010 - 4/2011)

I joined shopkick as a hybrid systems engineer, release manager, and dba. Over the course of the year, I handled all the systems work through the initial product launch, in July 2010, and explosive growth through the end of the year. We experienced over 500% traffic growth in a single week alone leading up to Black Friday.

• Wrote tools for rapidly provisioning new cloudservers and deploying role-based services. Set up and configured puppet for deploying system-level services. Scaled up from a handful of rackspace instances to a few hundred (250-400 depending on current resource needs). Engineered datacenter move from DFW -> Chicago.
• Deployed and administered a logging/MapReduce framework using local scribe instances, scribe aggregators with autodiscovery using zookeeper, and Hadoop/HDFS, complete with automated archiving to cloudfiles.
• Managed the weekly release cycle and all emergency cherrypicks. Wrote tools to automate the push process and rollbacks. Wrote services to bring up new continuous build instances (Hudson).
• Wrote tools to autogenerate DNS and firewall configurations from cloud provider data. Performed rudimentary SOX compliance audit to satisfy partner security needs. Wrote tools to do automatic port scans and package update checks. Set up demilitarized hosts for partner files access.
• Engineered a number of mysql scaling strategies, such as spinning off tables for offline access, flags to temporarily redirect read load, relay-only slaves, etc. Performed regular query analysis and traffic audits.
• Configured nginx pairs with heartbeat IPs as an HTTP proxy. Used apache and pylons to serve web traffic, backed by python web services and mysql databases. Built custom RPMs, set up a local yum repository. Set up nagios and ganglia, wrote a number of custom checks and gmetrics. Deployed memcache. Set up postfix and outbound mail relays.

systems infrastructure developer, Linden Lab (11/2008 - 3/2010)

In late 2008, I moved into an architecture team and assumed primary responsibility for improving the stability and performance of our central databases. In 2008 we'd had six major database outages and over 30 episodes of severely degraded service; since May of 2009, we've had no major outages and just half a dozen minor episodes. To get to this point, I delivered the following projects:

• Query profiler. I worked with data warehousing to develop a Streambase app for realtime analysis and aggregation of every query as it's hitting every single database (~100 total). We use tcpdump to sniff all the queries, then python to clean the queries of nonessential info and feed it back into streambase.
• Full code audit. I spent a lot of time sifting through the query profiler data for problems, and then working with the various developer teams to fix them. For example, I found that a third of our write time was being spent writing to the registration table, so we moved it to its own cluster. I also discovered that nearly 2/3 of our central db tables were legacy tables that were no longer in use, so we were able to get rid of that technical debt.
• Mysql benchmarking. Developed (with two other people) a protocol-agnostic tool called Apiary for distributed load testing using Python and RabbitMQ. This allowed us to sniff hours or days worth of mysql queries from the central database, and replay them over and over under various hardware/software/configuration environments. (This tool is being released to the open source community.)
• Mysql upgrade. Successfully upgraded our central database from 4.1.11 to 5.0.84 with zero downtime. Preparing for this was a mammoth task, which involved verifying thousands of queries for query syntax changes and data type changes, replaying traffic and checksumming table and replication integrity, and extensive benchmarking and performance tuning on the new version.

systems engineer/operations technical lead, Linden Lab (11/2004 - 11/2008)

When I started out at Linden Lab, we had two ops people and a handful of machines. When I left the group, four years later, we had about 15 people and 8,000 machines in four colo facilities around the world. I was an individual contributor for about half that time; later I designed and built and staffed the international NOC team, and then managed one of the two ops teams (the one devoted to infrastructure and scaling issues). Some key projects included:

• Designing a homogenous system image. We used systemimager and PXE for the base system, and cvsup for per-host/per-group configuration files. We focused on creating a sustainable model that would work for all host group types with zero administrative overhead.
• Upgrading the operating system. In 2007 we wrote tools to help us upgrade all the thousands of Debian boxes from sarge to etch as nondestructively as possible.
• Asset system. We moved all the assets out from the database to a hashed filesystem storage on a cluster of Isilon boxes. Later we added a second cluster of Isilons in a second colo facility, and wrote our own mirroring setup (since theirs didn't work for us).
• Monitoring system. We ran up against the limits of Nagios active checks early on, and switched to a hybrid ganglia/Nagios model that should scale indefinitely.
• DNS views. Set up DNS views to distribute load for nightly image deploys and utility hosts, as well as accomplishing primitive geolocation.
• We also insourced DNS, got rid of NFS on all our production systems, unified login data via LDAP and Kerberos, offloaded transient storage traffic using memcached, and did lots of other toolsmithing and scaling work.

• Provided third-tier support for a small web hosting company (about 800 Redhat boxes). I handled all the backend systems (mysql, apache, postgres, squid, tomcat, etc) with a couple other people.
• Wrote a sendmail milter plugin to quarantine outgoing spam from our customers. We had no host management tools at all, so I set up our first CVS root, centralized syslog facility, load-balanced DNS, and RANCID to track changes to switches and routers. Set up an NTP master, built a lot of custom RPM packages, and admined some Oracle as well.

• Managed the computer science department network. Set up authoritative primary/secondary DNS nameservers, set up the department's first package update system for their unique mix of FreeBSD, Linux, Solaris, HP/UX, and Irix. Set up Bugzilla to buffer student and faculty requests.

• My team of half-dozen sysadmins was responsible for 24x7 uptime as the company grew from 2 Sun UE4k's to over 300 machines in heterogenous clusters worldwide. I was in charge of the CVS root and deployed the first generation of monitoring tools (Big Brother and homegrown perl)
• Later, I moved into development, and performed all the bug fixes and feature additions to the MTA, which was based on qmail. I wrote the second known implementation of the sieve mail filtering protocol (RFC 3028), writing the compiler in C and a grammar parser in lex/yacc. I ported the MTA from Solaris to FreeBSD and built releases in several formats.

• I wrote an SNMP MIB to monitor key points of their network. Then I wrote ACLs for their Cisco switches. I traveled to hospitals in California and Washington to install or troubleshoot their proprietary video imaging systems, used in radiology departments.

• Ran a small production network of Sparc 5&10's, configured and maintained the Oracle 8i server, used samba to connect the Solaris/FreeBSD/NT environments, and automated lots of daily chores. Also served as the build and release engineer, using their in-house Lisp-like language.