Migrating our billing portal to the latest version of Laravel Spark

Monday 2nd of August 20212nd of August 2021

Deep Dives

When Flare was launched, we used Laravel Spark classic to take care of all user, team and billing functionality. We recently migrated to the latest version of Laravel Spark. It offers a beautiful billing management screen. Here's how it looks like.

screenshot

Our new Spark-powered billing portal allows you to see all available plans easily, subscribe, download invoices, etc...

By migrating to the latest version of Spark, we could vastly simplify our codebase. In this blog post, we'd like to share how we performed this migration and how we customised Laravel Spark.

How we used Laravel Spark classic

When we started work on Flare, we decided very early on that we didn't want to code everything around team management and billing ourselves. Using Laravel Spark was a given since we already had some experience with it by building Oh Dear.

At Spatie, we very much prefer working with React. We're also very detail focused when creating a UI. Because Spark classic was built with Vue components, that didn't work how we wanted, we made a drastic decision: we started using Spark in a headless way. We use everything that Spark classic offered on the backend but used nothing of the front end components. We created our own React components that hooked in the Spark classic back end.

For us, this worked out pretty well. Even though we had some work creating our own React billing components, Spark classic still saved us quite some time.

Moving to the newest version of Spark

Earlier this year, the Laravel team introduced a new version of Spark. The significant difference between the classic and the new Spark is that the new Spark is a separate billing portal instead of a Laravel starter template project on steroids. The user and team management features have been removed; they now reside in a separate free package called JetStream.

We started migrating to the newest version of Spark for two reasons:

The React components we built to handle billing were quite complex. We could remove this complexity from our app by moving to the new version of Spark while still providing a feature-rich billing portal.
The old Spark version still gets security updates, but no new features will be added. By moving to the latest version, we can take advantage of new additions that will be added to Spark in the future. In general, we like using the latest versions of everything.

Taking care of user and team management

We relied on the user and team management of the Spark classic. The new Spark doesn't offer this functionality. We could try to migrate to JetStream, but we opted for a much simpler solution.

In the old Spark, you had to let User and Team model of your app extend the base User and Team classes of Spark.

namespace App\Domain\Team\Models;

class Team extends \Laravel\Spark\Team
{
		// your code
}

Before removing the old Spark dependency, we copied over all the code from \Laravel\Spark\Team to our own App\Domain\Team\Models\Team model (and did the same with the User model).

Apart from a few other minor changes, this refactor did work. Our custom users/team UI just kept functioning normally.

We did the same refactor with the other code that extended stuff from \Laravel\Spark: team invitations, API token generation, auth controllers.

Because we used old Spark in a headless way and did not rely on any of its UI, just pulling the back end to our own codebase was easy.

After everything was copied, we did a round of refactoring where we simplified the imported Spark code. Any Spark features that Spark offered that we didn't use were removed.

Migrating the database

By comparing the tables and stored values of the production version of Flare with the local version of Flare with new Spark installed in it, we could write migrations that modify the table structure towards what the latest version of spark expects. We've also written migrations to copy the old data before any tables or columns are dropped.

You'll find the migrations we've used in this public gist on GitHub. We tested this migration by copying over parts of production data to our local environment to verify if the migrations work as expected.

Customising VAT handling

For the most part, we're using the Spark billing portal as is. The only significant customisation we've done revolves around VAT handling.

To verify European VAT ids, Spark uses the European VIES API. We've noticed that this API is down a lot, resulting in people that cannot subscribe. Of course, this is not good for business. It would be better to just accept the VAT number when the API is down and verify it later when the API is back up.

If you follow the rules strictly, you'll argue that, in theory, this isn't allowed as you are obliged to verify a VAT number before you invoice it. But in practice, we see that all people that sign up are using valid VAT numbers.

Luckily, it's easy to override how Spark handles VAT so we can implement this new behaviour. Spark handles VAT calculation in an action class which can be overridden in de container. In our SparkServiceProvider, we've added this binding.

// in app/Providers/SparkServiceProvider.php

$this->app->singleton(
   Spark\Contracts\Actions\CalculatesVatRate::class,
   App\Domain\Subscription\Actions\Spark\CalculateVatRate::class
);

Inside of the CalculateVatRate we can our own custom \App\Domain\Subscription\Support\VatCalculation\VatCalculator class. Here's the code for that action.

namespace App\Domain\Subscription\Actions\Spark;

use App\Domain\Subscription\Support\VatCalculation\VatCalculator;
use Spark\Actions\CalculateVatRate as SparkCalculateVatRate;

class CalculateVatRate extends SparkCalculateVatRate
{
    public function calculate($homeCountry, $country, $postalCode, $vatNumber)
    {
        return app(VatCalculator::class)->getTaxRateForLocation(
            $country,
            $postalCode,
            $vatNumber
        );
    }
}

Let's now take a look at the App\Domain\Subscription\Support\VatCalculation\VATCalculator itself. We're not going over all of this code in detail. The extra features that this code has over the default VATCalculator that ships with Spark are:

if the VIES API is down, we'll optimistically determine that the given VAT number is valid (we'll verify it later in a scheduled command)
responses from VIES will be cached
Greece, Norway and Turkey should not be charged VAT

namespace App\Domain\Subscription\Support\VatCalculation;

use Carbon\CarbonInterval;
use Illuminate\Config\Repository;
use Illuminate\Support\Facades\Cache;
use Mpociot\VatCalculator\Exceptions\VATCheckUnavailableException;
use Mpociot\VatCalculator\VatCalculator as BaseVatCalculator;

class VatCalculator
{
    protected BaseVatCalculator $baseVatCalculator;

    protected array $countryCodesThatNotShouldBeChargedVat = ['GB', 'NO', 'TR'];

    public function __construct(Repository $config, ?BaseVatCalculator $baseVatCalculator = null)
    {
        $this->baseVatCalculator = $baseVatCalculator ?? new BaseVatCalculator($config);
        $this->baseVatCalculator->setBusinessCountryCode('BE');
    }

    public function isValid(string $country, ?string $vatId): bool
    {
        if (! $vatId) {
            return false;
        }

        try {
            if (! $this->isValidVatNumber($vatId)) {
                return false;
            }

            $details = $this->getVATDetails($vatId);

            if (! $this->areEqualCountryCodes($details->countryCode, $country)) {
                return false;
            }

            return $details->valid;
        } catch (InvalidVatIdOrCountryCodeException) {
            return false;
        } catch (ViesUnavailableException) {
            // VIES is unavailable. Consider the VAT ID valid anyways.
            return true;
        }
    }

    public function isValidVatNumber(string $vatId): bool
    {
        try {
            $details = $this->getVatDetails($vatId);

            return $details->valid;
        } catch (InvalidVatIdOrCountryCodeException) {
            return false;
        }
    }

    public function getTaxRateForLocation(string $country, ?string $zip, ?string $vatId): float
    {
        if (! $this->shouldCollectVAT($country)) {
            return 0.0;
        }

        if (in_array($country, $this->countryCodesThatNotShouldBeChargedVat)) {
            // Edge case for GB after Brexit & other countries
            return 0.0;
        }

        if (! $vatId) {
            $vatPercent = $this->baseVatCalculator
                ->getTaxRateForLocation($country, $zip, company: false);

            return $vatPercent * 100;
        }

        $isValidVAT = $this->isValid($country, $vatId);

        try {
            $vatDetails = $this->getVatDetails($vatId);
        } catch (ViesUnavailableException) {
            $isValidVAT = true; // API did not respond
        }

        $vatPercent = $this
            ->baseVatCalculator
            ->getTaxRateForLocation($country, $zip, $isValidVAT);

        return $vatPercent * 100;
    }

    public function getVatDetails(string $vatId): object
    {
        $cacheKey = "vat_details.{$vatId}";

        $details = Cache::remember(
            $cacheKey,
            CarbonInterval::week(),
            function () use ($vatId) {
                try {
                    return $this->baseVatCalculator->getVATDetails($vatId);
                } catch (VATCheckUnavailableException $exception) {
                    if ($exception->getMessage() === 'INVALID_INPUT') {
                        throw InvalidVatIdOrCountryCodeException::make($vatId);
                    }

                    // Anything else (might) mean VIES service is actually just unavailable
                    throw ViesUnavailableException::make($exception->getMessage());
                }
            }
        );

        if (! is_object($details)) {
            // Base VAT calculator caught a VatServiceUnavailableException. Consider VAT ID invalid and invalidate cache.

            Cache::forget($cacheKey);

            throw InvalidVatIdOrCountryCodeException::make($vatId);
        }

        return $details;
    }

    public function shouldCollectVAT(string $country): bool
    {
        if (in_array($country, $this->countryCodesThatNotShouldBeChargedVat)) {
            return false;
        }

        return $this->baseVatCalculator->shouldCollectVAT($country);
    }
}

And finally, here is the command the will validate VAT numbers that have not been validated yet. We'll also validate VAT numbers again every 30 days after they validate. This way, when a VAT number gets retracted, we'll also know it.

If a VAT number wasn't valid, we'll send a notification to our Slack channel.

namespace App\Domain\Subscription\Commands;

use App\Domain\Subscription\Mails\VatIdCouldNotBeVerifiedOrIsInvalidMail;
use App\Domain\Subscription\Support\VatCalculation\InvalidVatIdOrCountryCodeException;
use App\Domain\Subscription\Support\VatCalculation\VatCalculator;
use App\Domain\Subscription\Support\VatCalculation\ViesUnavailableException;
use App\Domain\Team\Models\Team;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Mail;

class RetryValidatingVatIdCommand extends Command
{
    protected $signature = 'flare:retry-validating-vat-id';

    public function handle(): void
    {
        $teamsWithUnvalidatedVatIds = Team::query()
            ->whereNotNull('vat_id')
            ->get()
            ->filter(fn (Team $team) => $team->vatIdNeedsToBeRevalidated())
            ->filter(fn (Team $team) => app(VatCalculator::class)->shouldCollectVat($team->billing_country))
            ->map(fn (Team $team) => $this->retryValidation($team))
            ->filter(fn (Team $team) => ! $team->vatIdHasBeenValidated())
            ->each(fn (Team $team) => Log::channel('slack')->critical("😬 VAT ID `{$team->vat_id}` of team `{$team->name}` has not been validated"));

        if ($teamsWithUnvalidatedVatIds->count()) {
            Mail::queue(new VatIdCouldNotBeVerifiedOrIsInvalidMail($teamsWithUnvalidatedVatIds));
        }

        $this->comment('All done!');
    }

    protected function retryValidation(Team $team): Team
    {
        $vatCalculator = app(VatCalculator::class);

        if ($team->billing_country && ! $vatCalculator->shouldCollectVAT($team->billing_country)) {
            $team->markAsVatIdNotValidated();
        }

        try {
            $validationResult = $vatCalculator->isValidVATNumber($team->vat_id);

            if ($validationResult) {
                $team->markAsVatIdValidated();
            }
        } catch (ViesUnavailableException) {
            // Vies is unavailable. Ignore for now.
        } catch (InvalidVatIdOrCountryCodeException) {
            $team->markAsVatIdNotValidated();
        }

        return $team;
    }
}

It sure is quite a lot of code to get VAT handling right. We considered creating our own VAT calculator package. We held off that plan as Stripe recently announced a new product, called Stripe Tax, that can calculate VAT. We hope that Spark will offer support for Stripe Tax as soon as Stripe makes its new production publicly available.

In closing

By removing our custom UI for billing, we could make our codebase about 12,5K lines lighter.

✨ https://t.co/IEFWJgl4Fq now uses the latest version of Spark ✨

🥰 This allowed us to remove 12,5K lines of our code base pic.twitter.com/VJOkvycLAJ— Freek Van der Herten 🔭 (@freekmurze) July 8, 2021

Thanks to Spark's latest version, we now have a beautiful billing portal that we don't have to maintain ourselves. We're very grateful for the effort that the Laravel team put into this product.

In this blog post, we only touched upon a few aspects of the migration. If you want to read more details about upgrading Spark and JetStream, head over to this blog post on how our sibling service Oh Dear was upgraded.