Running Laravel Seeders before tests

Posted on by — No Comments ↓

Photo by Christian Joudrey on Unsplash

Overview

In this post, we’ll break down a custom Laravel service provider I wrote to solve a common but tricky problem: ensuring a fixed, reliable seeding pattern across all test scenarios, especially with parallelization.

We all know that deploying database schema changes is a standard part of the deployment process. But one of the big debates we’ve had within our own team was how to deploy stuff that wasn’t schema but also wasn’t code.

The thing is, you could put data inside of the migration. In fact, there are people that do and it does work for them. But it has limitations:

  1. If you’re putting your data changes directly in the migration, then you’ve got data changes scattered in your migrations.
  2. Those changes aren’t applied if you run php artisan schema:dump (because it only writes out the schema). And they are gone if you tack on --prune.
  3. You can write a seeder and centralize the data changes, but that, too, comes with its own problems:
    • Seeders can run multiple times.
    • Running schema:dump will remove calls to those seeders.

What we are looking for is a solution where:

  1. Seeders will run before tests (e.g. the code can depend on the data in the seeders).
  2. Seeders will run during our deployments.

Seeding for success

To us, a successful seeding strategy looks like this:

  1. Seeds only code-dependent information. (e.g. Permissions and roles.)
  2. Runs once before the tests. (Don’t re-run the seeder before every test.)
  3. Doesn’t require additional setup. (Don’t require devs to run the seeders manually before running tests.)

To accomplish this, we created a new Seeder that run 2 sets of seeders: the administrative permissions and our user-focused permissions: 

class RequiredPermissionsSeeder extends Seeder
{
    public function run(): void
    {
        $this->call(AdminPermissionsAndRoles::class);
        $this->call(SeedUserSubscriptionPermissions::class);
    }
}

Getting this to run as part of our deployments was pretty straightforward. In our deploy.php, just add a task to run the seeder: 

task('artisan:production-seed', artisan('db:seed --force --class=RequiredPermissionsSeeder', ['skipIfNoEnv']))->once();

And then just throw it into the deploy task: 

task('deploy', [
    // <snip>
    'artisan:migrate',
    'artisan:production-seed',
    'deploy:publish',
]);

Now, we can ensure that our deployments will always deploy our permissions as long as they are in our seeders. We write pretty comprehensive tests so that’s not going to be a problem: so long as the seeders run in the tests we’ll know we added them because we’re going to ensure that they are enforced.

Running seeders before tests

It’s pretty easy to run a seeder before you run a test. Just add this to a setUp() method: 

$this->seed(SomeSeederRequiredByYourTest::class);

The problem is that this seeder will run before every test method. Doing this before a handful of tests isn’t a huge deal, but with over 2000 tests in our suite, running the seeders before each test added a substantial amount of time.

We need a way to just run the tests before all the tests.

What didn’t work (for me)

I initially asked ChatGPT for help on this and it gave me a number of interesting paths:

  • PHPunit Extension: Apparently you can run code once before all tests. But… there is still the problem bootstrapping the application. I need something that sits between Laravel and PHPUnit.
  • PHPunit bootstap: Same issue. I’ll still need to bootstrap the application.
  • Put it into the base class setUp(): Had this worked it would have been easy. But it didn’t for reasons I’ll get into in a moment.

The letDown() of setUp()

Our applications base TestCase looks like this: 

abstract class TestCase extends BaseTestCase
{
    use DatabaseTransactions,
        MakesJsonApiRequests;

    protected function setUp(): void
    {
        parent::setUp();
        LaunchDarkly::fake();
        $this->withoutMix();

        // Other application specific setup.
    }

That parent::setUp() call must be first because it bootstraps the application. We can’t Artisan::call(...) before it because there won’t be a Facade root set.

However, putting the Artisan::call() after parent::setUp() will wrap the seeders in a transaction. That’s not gonna work either because it’ll run the seeders before every test.

Let’s open up setUp()

protected function setUp(): void
{
    $this->setUpTheTestEnvironment();
}

Er, we gotta go deeper… 

protected function setUpTheTestEnvironment(): void
{
    Facade::clearResolvedInstances();
    if (! $this->app) {
        $this->refreshApplication();
        ParallelTesting::callSetUpTestCaseCallbacks($this);
    }
    $this->setUpTraits();
    foreach ($this->afterApplicationCreatedCallbacks as $callback) {
        $callback();
    }
    Model::setEventDispatcher($this->app['events']);
    $this->setUpHasRun = true;
}

Much better. OK. That first line doesn’t look that interesting but… refreshApplication() and ParallelTesting::callSetUpTestCaseCallbacks() are both interesting.

But what’s setupTraits()?

Dig into setupTraits() and you’ll find the real reason your seeders in setUp() run in a transaction. Everything after setupTraits() will be wrapped in a transaction: 

protected function setUpTraits()
{
    $uses = array_flip(class_uses_recursive(static::class));
    // <snipped>
    if (isset($uses[DatabaseTransactions::class])) {
        $this->beginDatabaseTransaction();
    }

So now the question becomes: how do we get it to run during the application “refresh”. Well, how do you do anything when the application boots? A Service Provider!

The Solution: TestingServiceProvider

Let’s walk through the code and see how we tackled this: 

namespace AppProviders;

use DatabaseSeedersRequiredPermissionsSeeder;
use IlluminateSupportFacadesArtisan;
use IlluminateSupportFacadesParallelTesting;
use IlluminateSupportServiceProvider;

class TestingServiceProvider extends ServiceProvider
{
    protected static bool $hasRun = false;

    public function boot(): void
    {
        if (! app()->runningUnitTests()) {
            return;
        }

        if (static::$hasRun) {
            return;
        }

        static::$hasRun = true;

        if (env('LARAVEL_PARALLEL_TESTING') === '1') {
            ParallelTesting::setUpTestDatabase(function ($database, $token) {
                Artisan::call('db:seed', ['--class' => RequiredPermissionsSeeder::class]);
            });
        } else {
            Artisan::call('db:seed', ['--class' => RequiredPermissionsSeeder::class]);
        }
    }
}

Let’s break this down: 

if (! app()->runningUnitTests()) {
    return;
}

Don’t do anything if we aren’t in a unit test. 

if (static::$hasRun) {
    return;
}

static::$hasRun = true;

Because the application will get reset between runs, our provider will get initialized multiple times. However, because we are running before the transactions, we only need to run once. This check just ensures that we only ever run the provider once. 

if (env('LARAVEL_PARALLEL_TESTING') === '1') {
    ParallelTesting::setUpTestDatabase(function ($database, $token) {
        Artisan::call('db:seed', ['--class' => RequiredPermissionsSeeder::class]);
    });
} else {

Parallel Testing in Laravel uses a new database for each process. This database setup happens during bootstrapping. Thankfully, ParallelTesting gives us a callback that we can use to hook into its setup process. Using ParallelTesting::setUpTestDatabase() hooks into that so we can run the seeder once per parallel process!

The key part is this: env('LARAVEL_PARALLEL_TESTING') === '1'. You might think that, “Well, hey, if we can run the seeder once per process, then just running it in the service provider would be enough, right?” Well, it turns out the answer is “no”. I cannot tell you why exactly but for some reason, the seeders would somehow manage to produce database errors and would throw odd exceptions.

Because the test helper sets this env variable, we can know if we are running in a parallel invocation. If we are, then we register our callback with the ParallelTesting facade. Otherwise… just run it regularly: 

} else {
    // Make sure we run our required seeder once per test suite run.
    Artisan::call('db:seed', ['--class' => RequiredPermissionsSeeder::class]);
}

Now, the parallel test runner is happy and our regular invocations will also work properly.

In Summary

This was a real head scratcher. However, the cool part is that now our test suite now can have information we depend on in the seeder. The CI test suite runs as expected, and local devs can do their normal work without having to worry about making sure that they seed their single test DB or their parallel testing DBs. The Service Provider takes care of everything.

Leave a Reply