One of our teammates will get back to you soon.
As developers, we agree that we want to spend our time developing features that deliver value to our projects
As developers, we agree that we want to spend our time developing features that deliver value to our projects. Testing should be a tool that helps us to focus on the features that matter, while giving us the confidence to do important things like refactoring existing code. They should help us to focus on essential complexity in the problems that we're trying to solve rather than being another obstacle that we have to overcome in order to complete our task.
I've seen a number of cases, however, where novices in particular get caught up in sporadically failing tests or teams get dragged down by slow test suites. Below I've tried to identify some of the most common mistakes I've seen in Rails test suites, along with best practices and solutions to common errors.
In most Rails applications, much of our time testing is spent writing and running model tests. Following the traditional model of "skinny controller, fat model," this makes sense. What many people don't realize until their test suites start taking more than 10 minutes to run is the cost of persisting these records.
Writing to ACID-compliant databases is an expensive thing to do. This isn't the fault of the database implementations, it's because they care about not losing your data. The down-side of this is that your tests will start to drag if you always create persisted instances of your objects whenever you have to test a method. Instead, just instantiate a model with the attributes you need, and save the record only when absolutely necessary. If you're using FactoryGirl, this means using Factory.build instead of Factory.create whenever possible.
If you're reading a code base or a test, and see a place where a record is being persisted, do yourself and your team a favor and replace it with a test on a non-persisted object.
Following on the above point about minimizing writes to the database in your tests, if you find yourself writing a loop to create 30 records in the database, or even a few, stop. Think about how you can test your function with a couple of mock or stub objects instead.
In the past two years, I've spent some amount of time on the first day after January 1st fixing tests that depended on the year. In many large applications where a mixture of experienced and senior developers are working on code, chances are, someone will forget that time has an effect on the tests that you write. It's your responsibility when writing tests to make sure that they're not going to break due to something external changing. If you're not thinking about how time may affect your tests, start now. TimeCop is a useful gem for stubbing out time in your application where necessary.
A more insidious manifestation of this problem is the following:
user_a = Factory.create(:user)
user_b = Factory.create(:user)
User.all(order: 'created_at').should == [user_a, user_b]
Although this seems like it should work, the created_at
timestamp is recorded with precision only to the
second. Since you can probably persist many records per second on your machine, the two users likely have
the same created_at
timestamp, and you've just written code that contains a strange sort of time dependency
which will cause sporadic failure.
See the section below, "Depending on an order that isn't guaranteed," for more common order-related issues
This is a common beginner mistake in testing, and it even occasionally throws off people who have been
running tests for a while. You'll waste less time if you understand the way that tests interact with the database
schema.First, when you run rake,
rspec
will run the db:test:prepare
task. This loads the current database
from the schema.rb
file. If you run a single spec
example without running the full test suite with rake,
your
test schema won't necessarily be in sync with what you expect, and you'll get weird failures. Make sure you
run rake db:test:prepare
before you run rspec
on a single file. Also, if you just created a migration, but haven't
run the migrations yet, don't expect those changes to be visible to your test. You have to migrate first, then
rake db:test:prepare
before you see those schema changes in your tests.
Javascript-enabled tests in Capybara run in another thread.This is stated clearly in the README,but I've seen a lot of tests showing that developers haven't yet internalized this concept. This has a few implications:
If you're using transactional fixtures (this is an rspec default, and usually the right thing to do if you're not using js tests in your suite), you need to understand what this means. A transaction is a set of database statements that are executed as an atomic action. The classic example is moving money from one bank account to another – if anything happens in between, you should completely roll back the transaction to its initial state, rather than only completing part of the transaction.
Rspec leverages transactions to clean out your database tables between examples. A transaction block is started at the beginning of the test, and rolled back at the end, thus clearing out the records that you persisted during the scope of the test. There are a couple of ways that I've seen developers waste time troubleshooting tests when they don't understand transactions well:
Rspec
and DatabaseCleaner
are generally configured to roll things back that are created in the context of a
test example. This also applies to code that is run in a before(:each) block. Developers expect the database
state to be clean
between examples. I have seen a number of cases where stray data at the start of the test
breaks things, and causes sporadic test failures because of inter-spec dependency issues. Here are some
places to look if you think you have this problem:
While rspec
and mocha do a good job of limiting the scope of stubs and mocks to a particular example, there
are certain things that won't be reset between test examples. For example, I found one really awful spec that
did this:
ActionController::Base.asset_host = nil
The result is that after this particular line was executed, every subsequent test that was run would have a
slightly different state than those that ran before the asset_host
was set. In this case, links to assets like the
application javascript were broken for every example after the above line, and depending on the order that
rspec evaluated things, sometimes the integration tests would be broken, and sometimes they wouldn't.
Be particularly careful about setting class-level variables or other configuration options during a test, as these will affect every subsequent test that you run. Often, this is a sign that your code should be refactored so that you don't have to change this kind of global state. In the few cases where this is the most pragmatic decision, make sure that you use an 'ensure' block to roll back the state after your test.
This is one of the most frequent causes of failure that I see when developers break the CI build and then say,
"it passes on my workstation!" Most database engines don't guarantee the order in which results are returned
when an explicit ORDER clause is absent. Just because you asked for Person,where(hair_color: 'blue'
) and
got back [person1, person2, person3]
doesn't mean that the CI server will return results in the same order – it
may not randomize the results locally, so it may seem it's consistent, but the CI server could just as well have
a database that decides to return [person2, person1, person3]
. Your test fails, your team gets mad, and
everyone becomes less productive for twenty minutes while you push a change to fix the test.
There are two fixes for this scenario:
Sets
instead of Arrays
. Two ruby Sets are equal regardless of the order of contents, so if
your test example doesn't care about order, just compare your expected to actual results as two sets.
Rspec
provides syntactic sugar for this case with the =~
operator. For example,
Person.where(hair_color: 'blue') =~ [person1, person2, person3]
will pass even if the returned results are [person3, person2, person1].
Prevention is the best cure for most of the cases above. Learning Ruby, and frequently reading the source
code for rspec
and Capybara
when you don't understand how something works will do wonders for your
testing and application development skills in general. There are some things that you can do to detect where
bad things are happening in code, though:
Tests are supposed to provide a quick and reliable way to see if you've changed unintended application behavior by adding features or refactoring. It's annoying, and it makes you feel like you're wasting time if tests don't fulfill this basic purpose. I hope that these tips help you to waste less time in testing so that you can create tests that add value to the product that you're creating, rather than senselessly reduce your development velocity.
Published on May. 6, 2012
Join our community of avid readers and stay informed with the latest articles, tips, and insights delivered straight to your inbox. Don't miss out on valuable content – subscribe now and be part of the conversation!
We care about your data. Check out our Privacy Policy.