Integration testing our Laravel package with a real server and queue

Flare hooks into a lot of Laravel's internals: HTTP requests, views, queries, cache operations, queued jobs, Livewire components, and more. For each of these, we collect spans and send them as traces. We also catch exceptions and send error reports.

We're currently building new versions of the Flare packages that will also collect logs next from your application. To verify that everything works end to end in a Laravel application, we needed integration tests that go beyond mocking. We needed a real HTTP server, a real queue worker, and real payloads being sent.

Unit testing individual recorders of spans, logs and errors is straightforward. Laravel even provides facades for testing Http and Queue calls from the framework. The problem with them, they all run in the application currently booted.

We also wanted to verify that a full HTTP request through the Laravel kernel produces exactly the payloads we expect. That a queued job dispatched from a route creates a trace linked back to the originating request. That a Livewire component mount exception generates both a trace and an error report.

You can't really mock your way through that using facades. You need an actual Laravel application running.

Setting up a workbench application

When building a Laravel package, you cannot go around the Orchestra Testbench package, it's the de facto standard for testing Laravel packages. While it does a great job for unit testing, it doesn't really support the kind of integration testing we needed out of the box.

Luckily testbench has this great feature called workbench. It lets you scaffold a full Laravel application inside your package that you can use for testing.

You can install it like this:

composer require --dev orchestra/testbench

It will set up a workbench/ directory in your package. You can customize this skeleton application and use it to run real HTTP requests against it.

With workbench, we basically run two commands before we run our tests: vendor/bin/testbench workbench:build and vendor/bin/testbench serve. The first one prepares the skeleton, the second one starts the HTTP server.

The configuration for workbench lives in testbench.yaml:

laravel: '@testbench'

providers:
  - Workbench\App\Providers\WorkbenchServiceProvider

migrations:
  - workbench/database/migrations

workbench:
  start: '/'
  discovers:
    web: true
    api: true
    commands: true
    components: true
    factories: true
    views: true
    config: true
  build:
    - asset-publish
    - create-sqlite-db
    - db-wipe
    - migrate-fresh
  sync:
    - from: storage
      to: workbench/storage
      reverse: true

The build section is important. It tells testbench what to do when you build the application. In our case: publish assets, create a SQLite database, wipe it, and run migrations fresh.

The sync section creates a symlink from the testbench skeleton's storage directory to workbench/storage. This becomes relevant later.

Now we can create a real Laravel application inside workbench/ and customize it as we like. We can add routes, controllers, jobs, views, and more.

Our tests will make real HTTP requests to this server, which will trigger all the normal Laravel lifecycle events and Flare instrumentation.

The next challenge is how to capture these payloads in our tests so we can make assertions on them?

Writing payloads to disk instead of sending over HTTP

Normally, Flare sends data to our API over HTTP. That's not great for testing. We don't want to hit a real API.

Flare has a concept of a "sender" which is responsible for sending payloads to the API. By default, it uses an HTTP sender. But we can swap this out when testing.

So we created a FileSender:

class FileSender implements Sender
{
    public function post(
        string $endpoint,
        string $apiToken,
        array $payload,
        FlareEntityType $type,
        bool $test,
        Closure $callback,
    ): void {
        $id = uniqid();
        
        file_put_contents(
            storage_path("{$type->value}_{$id}.json"),
            json_encode($payload)
        );
    }
}

Instead of sending payloads over HTTP, it writes them as JSON files to the storage directory. The filename encodes the type (errors, traces, or logs) and a unique ID.

The ExpectSentPayloads helper

Now for the fun part. We built a test helper called ExpectSentPayloads that ties everything together:

$workspace = ExpectSentPayloads::get('/exception');

$workspace->assertSent(reports: 1, traces: 1);

$trace = $workspace->lastTrace()
    ->expectLaravelRequestLifecycle();

$trace->expectSpan(SpanType::Request)
    ->expectAttribute('http.response.status_code', 500);

$workspace->lastReport()
    ->expectExceptionClass(Exception::class)
    ->expectMessage('Test exception');

Let's take a look at how it works internally.

When you call ExpectSentPayloads::get('/exception'), the constructor does three things:

1. Cleans up any leftover payload files from a previous test
2. Makes an HTTP request to http://127.0.0.1:8000/exception using Laravel's HTTP client
3. Reads the payload files that the server wrote via FileSender

After the request, we read all files from workbench/storage and categorize them by type:

foreach (File::files($this->workSpacePath) as $file) {
    $entityType = match (true) {
        str_starts_with($file->getFilename(), FlareEntityType::Errors->value) => FlareEntityType::Errors,
        str_starts_with($file->getFilename(), FlareEntityType::Traces->value) => FlareEntityType::Traces,
        str_starts_with($file->getFilename(), FlareEntityType::Logs->value) => FlareEntityType::Logs,
        default => null,
    };

    // Parse and store each payload...
}

The files then get parsed and stored in the ExpectSentPayloads instance, which provides helper methods to make assertions on them.

Waiting for async work

Some tests dispatch queued jobs. The HTTP request returns immediately, but the job is processed asynchronously by the queue worker. We need to wait for the job to finish before reading the payload files.

In order to do that we only need to observe the jobs table in the SQLite database. When it's empty, we know all jobs have been processed.

We use a backoff strategy which eventually gives up after about 10 seconds, if something went wrong and the jobs are not processed:

$backoff = [
    500_000,   // 500ms
    750_000,   // 750ms
    1_000_000, // 1s
    1_500_000, // 1.5s
    2_500_000, // 2.5s
    4_000_000, // 4s
];

while (true) {
    usleep($backoff[$currentBackoffIndex]);

    if (DB::table('jobs')->count() === 0) {
        return;
    }

    $currentBackoffIndex++;
}

This behavior needs to be enabled on test case base since not all tests dispatch jobs and waiting for jobs to be processed when you don't need to is just extra time added to your tests.

What an actual test looks like

Here's a test that verifies a queued job creates the right traces with proper parent-child relationships:

it('can handle a queued job', function () {
    $workspace = ExpectSentPayloads::get(
        '/trigger-job',
        waitUntilAllJobsAreProcessed: true,
    );

    $workspace->assertSent(traces: 2);

    // First trace: the HTTP request that dispatched the job
    $httpTrace = $workspace->trace(0)
        ->expectLaravelRequestLifecycle();

    $requestSpan = $httpTrace->expectSpan(SpanType::Request)
        ->expectAttribute('http.response.status_code', 200);

    $queuingSpan = $httpTrace->expectSpan(LaravelSpanType::Queueing)
        ->expectParentId($requestSpan)
        ->expectAttribute('laravel.job.queue.connection_name', 'database')
        ->expectAttribute('laravel.job.name', SuccesJob::class);

    // Second trace: the job itself, linked back to the HTTP trace
    $jobTrace = $workspace->trace(1);

    $jobTrace->expectSpan(LaravelSpanType::Job)
        ->expectParentId($queuingSpan)
        ->expectTraceId($queuingSpan->span['traceId'])
        ->expectAttribute('laravel.job.success', true);
});

Two traces come out of this: one for the HTTP request, one for the job execution. The job trace's parent span ID points back to the queueing span in the HTTP trace. That's distributed tracing working end to end.

When a test dispatches a queued job, we end up with two trace files: one for the HTTP request and one for the job. Our tests need to know which is which. trace(0) should be the HTTP trace, trace(1) should be the job trace.

We can't rely on filenames for ordering since the IDs are random. Instead, we sort by inode number. On most filesystems, files created earlier get lower inode numbers. Since the HTTP trace is always written before the job trace, this gives us a consistent ordering.

In total we have about 65 integration tests like this, covering HTTP requests, exceptions, views, queries, transactions, cache operations, queued jobs (including chains, batches, retries, and failures), Livewire components, log messages, and context.

Auto-restarting a crashed server

Sometimes a test crashes the PHP built-in server or queue. That's actually something we want to test, especially at the moment when Laravel is terminating itself.

We tried getting the setup working where we started a fresh server and queue for each test. While this worked perfectly locally, it was very flaky in CI. Killing the server process between tests turned out to be tricky, and sometimes a server would linger in the background and cause port conflicts.

We instead opted for a single long-lived server that auto-restarts when it crashes. If a request fails to connect, we attempt to restart the server and retry the request:

try {
    $response = $client->get($this->endpoint);
} catch (ConnectException|ConnectionException $e) {
    if (! $this->restartServer()) {
        throw new Exception('Workbench server is not running.');
    }

    // Retry the request
    $response = $client->get($this->endpoint);
}

While not the cleanest solution, it turned out to be the most stable in CI.

Conclusion

This setup lets us test the full Flare instrumentation pipeline end to end. Just a real server, a real queue worker, and real payloads written to disk.

By using this approach we were able to catch a lot of edge cases and verify that all the different pieces of Flare work together as expected in a real Laravel application.

We're working on the latest changes to these new Flare packages, and we're excited to share them with the world soon.

For now, you can check out the code in the pull request of the spatie/laravel-flare package.