end of an era

In May of 1999, BitPusher acquired our first customer-facing server and signed our first colocation contract.  Over the next several years we increased our physical server footprint and colo infrastructure, focusing most of the next several years on providing a combined hosting-plus-management offering.

This morning, we shut down our colo and sold off the remaining equipment in it.  While the number of servers we manage continues to grow, and in some cases we still sell hosting (from third parties), maintaining our own physical infrastructure is no longer a fit for us.  The advantages of larger hosting environments have increased, from being able to provision or de-provision many servers quickly to having large enough networks to withstand large DDoS attacks to having geographical diversity with unified management.  Much of what makes a modern hosting company effective requires larger-scale infrastructure than it makes sense to tie to a primarily services-oriented business.

Also, the quality of hosting available, whether dedicated servers or cloud, has improved greatly while the costs have plummeted.  And we’ve realized that while understanding the performance and reliability characteristics of server hardware and networks is indeed central to the full application stack management that we do, most aspects of running a data center are really distractions from the application-centric focus that defines BitPusher.


under construction

As you’ve probably noticed, we’re in the process of changing the look and feel of our web site and marketing materials.  At the moment, we’ve updated the main web site, but we haven’t redone the blog yet.  We’ll also be making further improvements to the web site over the next few weeks.

We’d like to know what you think!  Please comment on this post or e-mail us at info@bitpusher.com with your feedback.

dal-noc hardware issues (no production impact)

Our NOC server in Dallas is currently having some problems and this may cause some spurious alerts to go out. Softlayer is attempting a chassis swap to resolve the hardware issue. In the meantime, nagios and munin graphs are offline in Dallas, affecting any customers whose systems are monitored from there.

Update: The server is back online now and service alerts are recovering.

Update2: Several queued alerts went out about 1:30am Pacific, they were false-positives. The situation is now stable.

Filesystem choices

Bitpusher has made a strategic decision to begin recommending and using XFS in lieu of JFS in most cases. This is a tentative decision but also a significant shift, not one we take lightly.

We have had a pretty good but not great experience with JFS over the past 18+ months.

Now RedHat has shown movement towards official adoption of XFS (and also ext4) but not for JFS.

Our strategy must support the major Linux distros (most of our clients run CentOS, RHEL, Ubuntu or Debian), and since XFS and JFS are both equal “cousins” on the Debian/Ubuntu side of things it just makes sense to prefer XFS. ext4 is still a bit immature for our taste.

Furthermore, various performance benchmarks including this one generally give both high rankings
We will implement this strategy through attrition rather than by any kind of forced retrofit.

Transparent ssh tunnels

One of our clients tipped me off to this awesome ssh configuration to create dynamic tunnels to servers which would otherwise be hidden behind NAT or a firewall. The mechanism uses a bastion host as a proxy combined with netcat.

Example snippet from .ssh/config or /etc/ssh/ssh_config…

Host example-gw
Hostname <ip-address>

Host  *.example.com
  ProxyCommand ssh example-gw exec 'nc %h %p' 2>/dev/null

Combine this with ssh keys and (something like) keychain/pageant/ssh-agent and accessing the systems at a remote site becomes oh so easy!

32-bit package cleanup

Red Hat and it’s variants have a nasty habit of installing 32-bit packages on a 64-bit platform like x86_64. See the output from uname -a to be sure, before proceeding.

Here is a one-liner that can be used (AT YOUR OWN RISK) to remove the offending packages. It has been my experience that this is generally safe to do.

rpm -qa --qf '%{name}-%{version}-%{release}.%{arch}n' | grep i.86$ | xargs rpm -e --nodeps

Also, adding this to /etc/yum.conf will prevent 32-bit packages from creeping back in.

exclude=kernel* *.i.86

BitPusher support policy

To our valued customers, colleagues, friends and fans.

This post should help to clarify our support policy. There often seems to be some misunderstanding about how responsive we (BitPusher) will be during the weekend or evening hours.

BitPusher staff is normally in-office from about 7am to 6pm US/Pacific time, Monday through Friday. We don’t monitor the incoming tickets on a 24×7 basis which means that normal requests (e-mail or phone) generated outside of those “normal business” hours (and sometimes within depending on how busy we are which as of late is VERY) will be handled the next business day. By handled I mean seen, assigned, scheduled and/or started.

If you need urgent support (24×7 day or night) call the support hotline (1-888-9PUSHER) and choose URGENT support. You will either be connected directly to a technician or be prompted to leave a voicemail. If the latter, a pager alert will go to the on-call technician who  should respond within 1 hour. Most of the time we’ll respond much quicker, within just a few minutes.

Any questions can be posted to comments here.

Mark Foster
Sr. Systems Engineer

HostingCon 2009 Photos

Hostingcon was an absolute success this year. Daniel and I had a great time overall and met a lot of great people. We had a chance to catch up with old friends and partners and met some new ones.

Hostingcon was definitely an eye-opener this year, there are a number of exciting changes about to transform the hosting industry. Over the next two years are definitely going to be a whirlwind of change for the industry. We’re in for a barrage of new cloud hosting and storage services coming at us from mid-sized dedicated-server & hosting companies.