I just finished with Crafting Rails Applications by José Valim. The book claims to teach Expert Practices for Everyday Rails Development. I didn’t find it to be exactly that, however I did enjoy it and I learned a ton.

The Good

The book flows pretty well. Each chapter contains a challenge or set of related challenges and it’s pretty easy to follow along with the code. Each example contains automated tests, and he creates a gem for almost each solution, so that it is easy to see how to modularize things.
The coverage of Rails internals is really enlightening. Sometimes it can be a bit dry, but you’ll learn quite a few things about Renderers, Responders, Engines, Railties, Routing, Rack, Instruments, and more.
The book also contains a wide range of technology topics. José doesn’t just utilize Rails & ActiveRecord. Other examples include MongoMapper, Redis, and Sinatra.

The Bad

Even though each of the examples contains unit tests, they are written in a style that I find to be problematic in real systems. The concise nature of the tests is probably an artifact of being printed, but most of the testing examples utilize few test classes with many assertions per test method.
In addition, though he “writes” the tests before implementation, this actually hurts understandability in some cases. The reason is most of the tests read as if he already knew the implementation and knew exactly how to test it. But it isn’t apparent to the reader.
I tend to prefer RSpec with 1 assertion per test method and many fine-grained methods that test various behaviors in each context my system is in. The book’s Test::Unit style wasn’t something I’d suggest as a good way to test real applications.

The Bottom Line

Ultimately I think the book is excellent, and certainly gives plenty of information about Rails 3 internals. I think newcomers would probably be lost, but if you’re past the newbie stage with Rails then this book is recommended.

I spoke at iPhone Dev Con in Boston last week and I attended Mike Lee’s excellent keynote presentation: “Mariachi Product Engineering”

At iPhone Dev Con in San Diego last year, Mike came out dressed like a Pirate. No kidding.

This year he came with a Mariachi outfit on. It’s pretty hilarious as nobody expects a keynote speaker to be dressed this way.

Over beers some of the guys at the conference decided it would be awesomely-epic to hire a real mariachi band to play during his keynote. So how does one find a mariachi band with after midnight on a Monday? Apparently, AMEX. One of the guys called his AMEX service and they delivered. I swear this could be the makings of a new AMEX commercial.

It made for a memorable keynote to say the least. The talk was excellent as well, as Mike certainly has a lot to say about Product Engineering.

Here is the video of the first 30 or so minutes of the talk (I ran out of battery before it was over).

Mike Lee - Mariachi Product Engineering from Ben Scheirman on Vimeo.

I’ll have links to my presentation material soon, but I wanted to share this awesome event. Certainly one for the books!

My blog was feeling a little stale to me, hence my utter lack of content in the past few months. I wanted to move it off of Windows hosting which really didn’t make sense for me anymore. Orcsweb graciously provided me with hosting through my MVP, and I would recommend their service to anyone.

But for me, it doesn’t make sense to run a .NET blog anymore. The days of “oh, maybe I’ll be able to extend it somehow” have just not ever mattered much. In the end, it was a pretty easy decision to move to Wordpress because it’s easy to host, easy to extend, there’s a gajillion themes & plugins, and the entire thing is fairly polished. Sure, there’s the nasty PHP underneath, but hey it lends itself well to simple hacking of the site to get things up & running.

So this is my attempt to freshen things up & encourage me to start posting more. Let me know what you think!

Side note: My blog has now had a total of 4 homes. First was blogger, then dasBlog, over to GraffitiCMS, and now finally WordPress.

This is way hard than it should be, but once you go “off the rails” not everything is candy & roses and chocolate covered bunnies. Of course, there is still a great community of talented folks producing stuff that makes all of this work.

The trick is to know what to look for. This post is for you, dear reader, to hopefully avoid all of the trial and error that I went through and hopefully get it to just work.

So, in my quest to connect a little ruby app on Linux to a SQL Server 2008 instance on Windows, I stumbled across the magical combination that allowed me to perform this task.

All of these commands were run on Ubuntu 10.4.

Step 1: Get unixODBC.

You’ll be connecting via ODBC, so let’s grab this one. There’s also iODBC, but I didn’t give that one a try.

1

sudo apt-get install unixodbc-dev

Step 2: Get FreeTDS

TDS stands for “Table Data Stream” and is the communication protocol that Sybase & SQL Server use. There are plenty of commercial drivers out there for just about every operating system, but one stood out to me… FreeTDS. I’ll give you 3 guesses why I liked this one (but you’ll only need 1).

We’ll also need the ODBC connector for this.

1

sudo apt-get install freetds tdsodbc

Step 3: Configure-shit

There are 3 INI files that we need to modify. When I did this on my Mac, they were in /usr/local/etc, but on Ubuntu it was all in the /etc directory.

Confirm that the FreeDTS & TDSODBC drivers are in place by looking in `/usr/lib/odbc` for the files `libtdsodbc.so` and `libtdsS.so`. We’ll be using those in the next step.

Using your favorite editor, open up `/etc/odbcinst.ini`. This file might not exist yet, that’s okay.

1

sudo vim /etc/odbcinst.ini

Here’s where we tell unixODBC which ODBC drivers the system supports. We’ll add the following text:

1
2
3
4
5
6
7

[FreeTDS]
Description = FreeTDS Driver
Driver = /usr/lib/odbc/libtdsodbc.so
Setup = /usr/lib/odbc/libtdsS.so
FileUsage = 1
CPTimeout = 5
CPReuse = 5

Save that file. Next up we should configure FreeTDS to define our data sources.

1

sudo vim /etc/odbc/freetds/freetds.conf

There should be some example configurations in here. Also note that here is where you can un-comment a line to enable debug output if you’re having trouble (I needed this). Here is my configuration:

1
2
3
4
5

#my sql server (SQL Server 2008)
[my_database]
host = 11.22.33.44 #no this isn't a real IP. you think I'm crazy?
port = 1433 #change this if you're running your db on a non-standard port
tds version = 8.0

One last piece of configuration is needed. This is the general system datasources configuration.

1

sudo vim /etc/odbc.ini

The following should be fairly obvious configuration:

1
2
3
4

[my_database] #note this doesn't have to match what's in freetds.conf
Servername = my_database #this matches what's in freetds.conf
Driver = FreeTDS #this matches the driver you configured in odbcinst.ini
Database = YOUR_DATABASE_HERE

Step 4: Try it out

Using the unixODBC tool, test out your connection:

1

isql my_database uid password

If you had any errors, you’ll get something like “Cannot SQLConnect”. If this happens, check your debug log (if you enabled it in freetds.conf). In my case I forgot to enable Mixed Mode authentication on the SQL Server side.

Step 5: Install the gem

We’ll be using a nice gem called tiny_tds.

1

gem install tiny_tds

With that installed, let’s test it out!
Launch irb:

1
2
3
4

require 'tiny_tds'
client = TinyTds::Client.new(:dataserver => 'my_database', :username=>'user', :password=>'password')
client.closed? # should be false, if it's true you probably had an error connecting
client.execute("SELECT getdate()").each {|row| puts row}

If you got a date back, then your connection is working properly. Huzzah!

I’m setting up a new Linode box for a client. (Speaking of Linode, it’s freaking awesome and you should sign up. If you do, use this link to give me some bonus credit).

For the Linux distro I chose Ubuntu 10.4 (32-bit). This is just the one I know the best.

One of the first things I like to do is set up a personal user account so that I don’t use the root. This is so less-than-intuitive I have to look it up every time.

Here are my notes so that I don’t forget.

As root, run the useradd command:

sudo useradd -d /home/testuser -m -s /bin/bash testuser

Reset their password:

sudo passwd testuser

Then add them to the sudoers file:

visudo

Create a user alias for the company accounts:

User_Alias CHAIONE = user1, user2

Then add this line lower down:

CHAIONE ALL=ALL

For future users, just add them to the CHAIONE alias

Having my hands in .NET, Rails, and iPhone for the past year I’ve come to appreciate and loathe the various ways that projects incorporate code from 3rd parties.

Objective-C & Xcode get the award for the least-awesome way of reusing libraries. I’ll cut them a tiny bit of slack since this technology is decades old, however it’s 2010 and I have work to do. Let’s just say there’s no “iPhoneGems” project and many projects (such as the excellent ASIHTTPRequest) simply suggest copying over source files into your project. This sucks for a number of reasons, but at the core it’s simply copy-paste code reuse.

No Dynamic Linking

The reason is mostly because there is no dynamic linking on the iPhone. You can’t simply “reference a dll” and have it work. You are limited to static linking, however that means that you need to pre-compile for each SDK version & architecture. Not fun, nor do I even completely understand how all this works. This is how I originally used the JSON-Framework, however I’ve since relegated to copying over source files into my project, after realizing that they hadn’t updated their static library for the 4.0 SDK.

What do the big boys do?

As a larger example, Facebook’s Three20 project itself has many projects, and utilizes Xcode’s project dependencies. The setup is slightly more involved as you need to setup Header Search Paths, Linker Flags, and other (seemingly arcane) configurations, but if you follow the directions it is not too terrible.

This method sets up project dependencies, where the output of the sub-projects (in this case the suite of Three20’s projects) are static libraries for the SDK & Architecture you’ve specified for your project.

This is a pain to setup initially but does offer some additional benefits, such as having the other projects remain in a separate source tree. In our case, we’re pulling in Three20 from a git repository, and we are leveraging git submodules to make this happen. (Though git submodules are a bit of a pain to deal with).

What do you do?

Do you utilize this technique for your internal shared code? Or do you resort to copying around source files from project to project?

New Relic is seriously awesome. It’s pretty much the standard for performance monitoring in Rails. Recently New Relic introduced support for .NET apps. I think this is HUGE.

When I show off New Relic RPM to .NET developers, most are flabbergasted and ask…

Why can’t we have this in .NET?!?!?!

Well, now you can!

I made a comment (jokingly) on Twitter:

…The more I think about this the more I think I’m right. They have a proven track record in the Rails community and are a seriously valuable tool for troubleshooting bottlenecks and perf issues, all the way from the web tier to the database and back again.

I’m not sure how much of this is present in the current .NET version of their plugin, but I suspect that it will get more robust over time and be a standard in the .NET community as well.

I’m curious to hear from anyone who is currently using this. What is the setup experience like? If it’s as “drop-in” easy as it is in Rails then it’s a no-brainer. Let me know in the comments.

Improving is putting on an AgileDotNet conference in Houston (at the Microsoft Offices) on Friday, November 12th )Looks like the conference got pushed to February. Bummer!). Word on the street is that it will be a mix of technical and project management related content! I heard great things about this conference’s past incarnations and I look forward to attending this one.

Oh yeah, and I’ll be giving a talk on Git and why you should love it as much as I do.

At a price of only $49, it’s really a no-brainer. Hope to see you there!

More registration and info can be found here.

I went to (and spoke at) my first Ruby conference this past weekend. Lone Star Ruby Conference was held in Austin and consisted of 1 training day & 2 days of conference sessions. There were 2 tracks, and even still I was often conflicted on which session to attend. The hallway conversations were also quite engaging.

Day 1: Ruby Intrigue

I went to the Ruby Intrigue full-day training. I think by about 10:00 I was already thinking the trip to Austin had been worth it. I learned some ruby tricks that I just didn’t know before. The agenda was to create a Web Crawler, Asteroids Clone, and SMS Gateway all in Ruby. We explored areas of the language that we might not otherwise see writing web applications. The Web Crawler was refactored multiple times to attempt to improve performance by using threads, processes, queues, and even jruby.

Next up, we learned how to leverage the Gosu library to write games in Ruby. We wrote a complete asteroids clone that is quite impressive. You can see the code for this on github.

Lastly we dug into some SMS integration using the sms_fu gem. Unfortunately there were some gem dependency problems and not everyone was able to get it working. I spent that time improving the asteroids game. :)

I really want to thank the guys from Intridea who put it on, specifically Pradeep Elankumaran, Adam Bair, and Brendan Lim. They did a great job and I learned a lot. You can see some of the photos from this day on Brendan’s Flickr.

The material that we went over can be found here.

Day 2

The day kicked off with an excellent keynote by Glen Vanderburg about the history of (Software) Engineering, studying the original papers that kicked off 40 years of Waterfall software projects. Truly interesting.

We learned how to Decipher Yehuda from Greg Pollack, get a glimpse of how a Vim Ninja works, a little bit of Hashrocket culture, and a fascinating tour of languages by Bruce Tate. Jesse Wolgamott (my co-hort at ChaiONE) gave a talk about NoSQL, giving some good comparisons on a few of the tools.

The highlight of Day 2, was hearing Tom Preston-Werner (co-founder of Github) give a closing keynote on what made him successful. I really enjoy using Github, and it was interesting to hear the story behind it. Afterwards he bought beer for everyone at the Draught House. Unfortunately I couldn’t make it because I needed to go rest up, as my voice was getting worse & worse.

Day 3

I enjoyed a number of sessions this day. First was Nick Gauthier explaining how he took a 13m-long test run down to 18 seconds by using a few gems, some file system tweaks, and leveraging all cores of his machine by using Hydra. Quite impressive, and a little bit intimidating. I clearly need to know more about how to tune my machine!

Next was Rogelio J. Samour from Hashrocket (and author of Hitch) talking about how to effectively use rvm and gemsets. I don’t know what I was waiting for, but my new philosophy will be: rvm & gemset for every new project I work on. Combined with a .rvmrc there’s really no reason not to.

In the afternoon I gave a talk on Heroku, and I think it went well. There were probably 200+ people in the room and my nasty cold made my voice a little scratchy, but luckily it held out, and I was able to do all the material planned in the 35 minutes allotted. I was even given a Heroku t-shirt to sport up on the stage, which I’ll wear proudly.

The slides are available here:

The ChaiONE crew, during a break:

I also had to go check out the Austin Man vs Food venue, Round Rock Donuts:

All in all, I had a fantastic time at Lone Star Ruby Conference and I absolutely will be back next year.

I’ve said many times before how much I love Heroku. Though they’ve had an unusually large amount of downtime this week (~1 hour), I’m still loving their service. One of the applications that we’re building is expecting a large wave of traffic shortly following a blitz PR campaign. The golden question of the hour is: How much traffic can we handle?

Currently we’re on the Koi 1 plan, which gives us a 20GB database (shared) and 1 dyno. A “dyno” is basically 1 concurrent request. It’s very similar to a super market. You get in line and you are served in the order in which you came. If some jackass has a cartload of Natural Light & Cheeze Whiz in front of you, you’re gonna have to wait a while. Adding another Dyno is like opening a 2nd lane.

The key to good performance on Heroku is to watch the Queue Depth. If you ever reach too many people in the queue at one time your app will stop serving up requests past that mark, you’ll get a Backlog Too Deep error. I’m not sure what the limit is, but I’m guessing it’s around 100 (which is HUGE).

For more information on Heroku performance, see their excellent docs.

To perform my tests I’m using 2 tools: Apache Bench & New Relic RPM. Both are stellar.

Test 1: Serving up the landing page (1 Dyno, 5000 requests, 20 at a time)

This is a simple GET request that the Heroku routing mesh likely caches heavily. There’s no database queries on the homepage and no logic.

Requests: 5000Concurrency: 20# of Requests/Sec: 85

This is pretty darn good. 50% of the requests are served in under 300ms and the max request took less than 1 second. No requests failed.

Test 2: A simple API read operation (1 Dyno, 5000 requests, 20 at a time)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

Concurrency Level: 20
Time taken for tests:   68.067 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      4276315 bytes
HTML transferred:       2665000 bytes
<strong>Requests per second:    73.46 [#/sec] (mean)</strong>
Time per request:       272.267 [ms] (mean)
Time per request:       13.613 [ms] (mean, across all concurrent requests)
Transfer rate:          61.35 [Kbytes/sec] received
Connection Times (ms)min  mean[+/-sd] median
maxConnect:       68   75   4.2     74      97
Processing:    85  197  73.9    180     607
Waiting:       85  197  73.9    180     607
Total:        157  272  73.8    255     685
Percentage of the requests served within a certain time (ms)
50%    255
66%    291
75%    313
80%    325
90%    365
95%    407
98%    477
99%    535
100%    685 (longest request)

Again, great results. This is due to caching of course, but at least we know we can handle lots of requests on this API.

Test 3: A simple API write operation (1 Dyno, 5000 requests, 20 at a time)

This API operation inserts 2 records in the database. Again, 5000 requests, 20 at a time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

Concurrency Level:      20
Time taken for tests:   320.981 seconds
Complete requests:      5000
<strong>Failed requests:        4165
</strong>   (Connect: 0, Receive: 0, Length: 4165, Exceptions: 0)
Write errors:           0
Total transferred:      4022365 bytes
Total POSTed:           1935387
HTML transferred:       2651618 bytes
Requests per second:    15.58 [#/sec] (mean)
Time per request:       1283.923 [ms] (mean)
Time per request:       64.196 [ms] (mean, across all concurrent requests)
Transfer rate:          12.24 [Kbytes/sec] received 5.89 kb/s sent  18.13 kb/s totalConnection 

Times     (ms)min  mean[+/-sd] median
maxConnect:   68   71   3.9     70     104
Processing:   134 1210 865.8    975    5559
Waiting:      134 1210 865.8    975    5558
Total:        202 1281 865.7   1048    5627

Percentage of the requests served within a certain time (ms)
50%   1048
66%   1364
75%   1638
80%   1819
90%   2430
95%   3176
98%   3898
99%   4248
100%   5627 (longest request)

You can see here that we failed 4165 requests. That’s not good! Let’s try adding a dyno and do it again.

Test 4: A simple API write operation (2 Dynos, 5000 requests, 20 at a time)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Concurrency Level: 20
Time taken for tests:   135.882 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      4045448 bytes
Total POSTed:           1935774
HTML transferred:       2675000 bytes
Requests per second:    36.80 [#/sec] (mean)
Time per request:       543.530 [ms] (mean)
Time per request:       27.176 [ms] (mean, across all concurrent requests)
Transfer rate:          29.07 [Kbytes/sec] received
13.91 kb/s sent
42.99 kb/s totalConnection Times (ms)min  mean[+/-sd] median
maxConnect:       67   95  32.7     72     979
Processing:   121  448 162.4    420    1481
Waiting:      121  448 162.5    419    1481
Total:        199  543 154.4    515    1549
Percentage of the requests served within a certain time (ms)
50%    515
66%    580
75%    624
80%    654
90%    751
95%    840
98%    946
99%   1034
100%   1549 (longest request)

We’re getting a pretty healthy 36 requests per second now, and look, no errors!

Test 5: A simple API write operation (2 Dynos, 10000 requests, 20 at a time)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Concurrency Level: 20
Time taken for tests:   326.317 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      8091062 bytes
Total POSTed:           3870000
HTML transferred:       5350000 bytes
Requests per second:    30.65 [#/sec] (mean)
Time per request:       652.634 [ms] (mean)
Time per request:       32.632 [ms] (mean, across all concurrent requests)
Transfer rate:          24.21 [Kbytes/sec] received
11.58 kb/s sent
35.80 kb/s totalConnection Times (ms)min  mean[+/-sd] median
maxConnect:       68   72  13.8     70    1004
Processing:   113  580 233.1    539    2145
Waiting:      113  580 233.1    539    2145
Total:        183  652 233.5    610    2213
Percentage of the requests served within a certain time (ms)
50%    610
66%    701
75%    771
80%    822
90%    971
95%   1103
98%   1254
99%   1374
100%   2213 (longest request)

We slipped a tad on overall throughput (down to 30 reqs/sec) but we still served up all of the requests in a reasonable time without any failures. Let’s check out New Relic & See what it says. In this graph you can see the overall HTTP Throughput (higher is better) against the Heroku backlog depth. The more dynos you have, the quicker your backlog is cleared out, equating to more throughput. You can clearly see that we are now hitting a limit on the throughput we can handle because the queue depth is increasing. We might choose to optimize the site or add another dyno to squeeze some more perf out of this. Also in the image if you were to mouseover on the queue stacks on the website, you’d see that my peak queue depth was 29. That’s pretty high, and you can definitely see a cap in the total # of requests that we can serve. In the end, I’m able to pretty reliably say that we can handle a 10k request spike when running on 2 dynos. I’d suggest more dynos if the client wanted to handle more than that. Gotta love tools that make investigation like this easy. Props to Heroku & New Relic.