Filesystem choices

Bitpusher has made a strategic decision to begin recommending and using XFS in lieu of JFS in most cases. This is a tentative decision but also a significant shift, not one we take lightly.

We have had a pretty good but not great experience with JFS over the past 18+ months.

Now RedHat has shown movement towards official adoption of XFS (and also ext4) but not for JFS.

Our strategy must support the major Linux distros (most of our clients run CentOS, RHEL, Ubuntu or Debian), and since XFS and JFS are both equal “cousins” on the Debian/Ubuntu side of things it just makes sense to prefer XFS. ext4 is still a bit immature for our taste.

Furthermore, various performance benchmarks including this one generally give both high rankings
We will implement this strategy through attrition rather than by any kind of forced retrofit.

Custom graphing with Cacti

Cacti is a great tool for time-based visualization of data. Out-of-box functionality can leave something to be desired. Here is a stepwise tutorial for creating custom graphs of web server requests. Most of the instructions can be applied to other scenarios.

From a high-level you need to:

  1. Create a source of the (time-based) data which exists independent of Cacti
  2. Create a means of obtaining the data (Data Input Method)
  3. Assemble outputs of (2) into a Data Template
  4. Create Graph Template based on the values available in the Data Template
  5. Apply those to a Device, this creates a Data Source and set of graphs

Then after some time passes you will have some nice graphs to ogle at.

A real world example would be capturing requests-per-second statistics from Nginx. This technique would apply to any web server logs with a little massaging I guess. In this case, our Nginx server was not compiled with stub_status ability which might otherwise be used, so instead we fashion a shell script that can run on (each) web server and determine rps from the access log.

statsnginx is a rudimentary script that captures average rps in ranges of 10s, 1m and 5m ago.

Example of running and the output:

$ /usr/local/bin/statsnginx
/usr/local/nginx/logs/nginx.vhost.access.log
c10s:389 c1m:409 c5m:312

Next step, make this data available to cacti. I choose snmp MIB since snmpd is already running on our Nginx servers.
Just by adding this line to snmpd.conf and restarting can we see the data remotely (wrapped for clarity).
exec 1.3.6.1.4.1.5001.3 statsnginx
/usr/local/bin/statsnginx
/usr/local/nginx/logs/nginx.vhost.access.log

(note this exec syntax is broken after net-snmp-5.3.1 on FreeBSD anyway, so YMMV)


$ snmpget -v 1 -On -c public
192.168.1.117:161 .1.3.6.1.4.1.5001.3.101.1
.1.3.6.1.4.1.5001.3.101.1 = STRING: "c10s:275 c1m:274 c5m:230"

This part can be tricky to figure out the exact OID. Use snmpwalk if necessary, e.g.
$ snmpwalk -v 1 -On -c public
192.168.1.117:161 .1.3.6.1.4.1.5001.3

Create the cacti script in /opt/cacti/scripts/statsnginx.sh

#!/bin/sh
OUTPUT=`snmpget -Ov -v 2c -c bitpushsnmp $1
1.3.6.1.4.1.5001.3.101.1`
echo $OUTPUT | sed -e 's/STRING: //' | sed -e 's/"//g'

Test it from the cacti server.

$ ./statsnginx.sh 192.168.1.117
c10s:190 c1m:192 c5m:152

Notice how the output has three fields of name:value pairs separated by spaces. These are what cacti likes.
Don’t make the mistake like I did of using name=value because it will make cacti think it’s a PARTIAL result..

Now we can move into cacti to make use of the script and data it provides.

Please see this helpful link to Cacti docs.

Go to the Cacti console and create a Data Input Method to tell Cacti how to call the script …

  1. Data Input methods
  2. Add
  3. Name: statsnginx
  4. Input Type: Script/Command
  5. Input String: <path_cacti>/scripts/statsnginx.sh <hostname>
  6. …Save…

Now you have Input Fields and Output Fields

Cacti wants you to provide these.

Since we are gathering data from a remote (to cacti server) host, need to give hostname as input

  1. Add an Input Field
  2. Name: hostname
  3. Field Order: 1
  4. Friendly Name: Hostname

Add an Output Field for each of the name:value pairs above (c10s, c1m, c5m)
Example:

  1. Field [Output]: c10s
  2. Friendly Name: Average requests/sec over last 10 seconds
  3. Update RRD File: checked
  4. Do this for the other two fields.
  5. Then Save once more

Now create the Data Template

  1. Add
  2. Data Template
  3. – Name: nginx – Requests
  4. Data Source
    • Name: |host_description| – nginx – Requests
    • Data Input Method: statsnginx (chosen from list)
  5. Data Source Items
    • Internal Data Source Name: for each data point above (c60s, c1m, c5m) add all (output fields items from above)

This is a tricky part. You want all three data source items listed with appropriate min/max values (left at 0 in this case) and using GAUGE as the Data Source Type. Also make sure to select the appropriate Output Field from the list for each one.

Now create the Graph Template

  1. Add
  2. Template Name: nginx – Requests
  3. Title: |host_description| – nginx – Requests
  4. I changed Upper Limit to 10000 just to be sure.
  5. ..Create…

Now add Graph Template Items one by one
e.g Data Source nginx – requests (c10s)
and so on. Give each one an appropriate type like area, stacked or line1.
You can use the Graph Template for ucd/net – Load Average as a reference since it has similar measures (1, 5 and 15m load average).

Pusher of Bits Needed (Senior Linux Administrator)

Pusher of Bits Needed (Senior Linux Administrator)

BitPusher is a small, rapidly growing web infrastructure company with offices in Seattle and San Francisco. We are seeking a creative, dynamic and highly responsible senior systems administrator (SAGE level IV).

BitPusher manages web servers, networks and other IT infrastructure, both owned by BitPusher and hosted elsewhere. We build long-term relationships with our customers and take the time to understand their applications, their technical needs and their business needs. We customize things as required to meet those needs, but we also keep things as standard as we can for reasons that we’re sure you already understand.

Do you like making order out of chaos? Are you effective when managing lots of little projects, but also have a tendency to make improvements that you didn’t originally have in mind? Are you always looking for ways to manage servers more efficiently? If so, let’s talk.