Building a flexible yet abstract external integrations Structure

Here we are. Flare is better than ever! We've been adding features week after week. Yet, one significant request on our roadmap remains: Jira! It is the most upvoted request on our roadmap, which is something we can't ignore.

We already had a GitHub integration. It allows you to link GitHub issues and PRs to Flare errors. What's cool is that you can automatically close an issue when an error is resolved. The architecture behind this was built over two years ago, and we thought we had built it so that we could plug in another issue-tracking system like Jira without any problems. Oh boy, were we wrong about that.

So, building the Jira integration opened up an opportunity to do it better, and that's just what we did. We built a flexible structure that abstracts much of the code shared among integrations, allowing us to add new integrations quickly. We'll go through three examples where we've abstracted a lot of code while still making it possible to implement completely different integrations.

Requirements

At its simplest, an issue tracker is an external service with a few groups (projects might be a better term here, but this clashed too much within our code base with Flare projects). For GitHub, a group is a repository; for Jira, this is a project. Each group has issues within them. These issues have a title and a body where the issue is summarized. Most of the time, these issues also have comments with a body.

Issue Data

We want to link a Flare error with an issue in an integration. So, somewhere within our database table, we need to store this link. A possible schema will probably look like this:

id
error_id
integration_type
integration_id

Because we want to show some information about linked issues on an error, we need a data structure for them:

For Jira, the structure would look like this:

class JiraIntegrationIssueData
{
    public function __construct(
        public int $id,
        public string $key,
        public string $summary,
        public ?string $description,
        public string $status,
        public ?string $reporter,
        public ?string $assignee,
        public ?string $creator,
    ) {
    }
}    

For GitHub, it would look like this:

class GitHubIntegrationIssueData
{
    public function __construct(
        public int $id,
        public int $number,
        public string $title,
        public ?string $body,
        public GitHubIssueType $type,
        public GitHubIssueStatus $status,
        /** @var string[] */
        public array $assignees,

    ) {
    }
}    

Apart from the id, these structures are entirely different. We could create separate tables for them and then use a polymorphic relation to these structures. But that means we need to create two more extra tables + two more extra models just for this metadata. At the same time, these data objects contain all the structured data.

Even worse, because we need to link a Flare project to an integration group, we'll need some extra metadata here, so add two tables and two models. To finish it off, a team is linked with an integration instance. For example, you link your Flare team with a GitHub organization or account. Again, we need to store data about this; we need to store some configuration (like the name of the GitHub organization user) and secrets (API keys). This means we'll need to add another four tables and four models to make the Jira and GitHub integrations work.

Adding an extra integration would require four tables and models when we use this approach. We plan to add more integrations, so we would need twenty tables and models to store data when we have five integrations. That's an awful lot of extra models and tables!

Luckily, there's another way. Let's update our schema and add a JSON field called data:

id
error_id
integration_type
integration_id
data

We could try to cast the JiraIntegrationIssueData and GitHubIntegrationIssueData objects ourselves because we need the functionality to cast them to a JSON payload and back into an object again. Luckily, we don't need to write code for this because we can use the excellent laravel-data package to do this for us.

Letting our classes JiraIntegrationIssueData and GitHubIntegrationIssueData extend from data adds a lot of functionality. What's most useful for us is that we can call $someDataObject->toJson() on our data objects, and they will be automatically cast into a JSON string. Using SomeIntegrationIssueData::from($json) then allows us to create a data object again from JSON, neat!

But we're not done here. We still need a cast for our ErrorIntegrationIssue model representing the link between an error and an integration issue. The data field of the model can be JiraIntegrationIssueData, GitHubIntegrationIssueData, or even some completely new object type that we'll build for a new integration in the future.

Luckily, laravel-data saves us again here. We create an abstract IntegrationIssueData class like this:

abstract class IntegrationIssueData extends Data
{
    public function getId(): int;
}

The data classes then would look like this:

class JiraIntegrationIssueData extends IntegrationIssueData
{
    public function __construct(
        // Properties here
    ) {
    }

    public function getId(): int
    {
        return $this->id;
    }
}

// and

class GitHubIntegrationIssueData extends IntegrationIssueData
{
    public function __construct(
        // Properties here
    ) {
    }

    public function getId(): int
    {
        return $this->id;
    }
}

When we create a new link as such:

$gitHubIntegrationIssueData = new GitHubIntegrationIssueData(....);

$errorIntegrationIssue = ErrorIntegrationIssue::create([
    'error_id' => $error->id,
    'integration_type' => IntegrationType::GitHub,
    'integration_id' => $gitHubIntegrationIssueData->id,
    'data' => $gitHubIntegrationIssueData,
]);

And retrieve somewhere later in our code, we can do the following:

$errorIntegrationIssue->data; // a GitHubIntegrationIssueData instance

How does this magic work? We've added the following cast to ErrorIntegrationIssue:

class ErrorIntegrationIssue extends Model
{
    protected $casts = [
        'data' => IntegrationIssueData::class,
    ];

    // other model stuff
}

Since IntegrationIssueData is abstract, laravel-data knows that each object being saved to the database extends from that abstract data object. So, we also store the class name next to the data from the data object. When retrieving the model and thus data object again from the database, laravel-data will create a new object based on that stored class name and with the stored data.

Considerations

The database people are probably already screaming; these data structures all deserve their own table with their own fields. And most of the time, I would agree with them, but not in this case. We don't need to query these data objects; they are only shown on an error page. We can fetch them by looking for issues linked to the error by checking the error_id field.

In one case, we need to query a field from the data object where a webhook comes in, for example, because the issue's title changed. In this case, we need to know that the issue is from a specific integration (this would most certainly mean that we need to check the class name of the data object to determine if it's Jira or GitHub) and its id so that we can update the issue data object.

That's why we have the integration_type and integration_id fields on ErrorIntegrationIssue. When a webhook comes in, we can use these (indexed) fields to quickly look up the issue and then update the data object.

Communicating with APIs

Let's look at another part of our application where we tried to create a flexible abstract structure. A new feature we're adding with these integrations is linking a Flare error with an issue using a URL. On an error, you'll have a text field in which you can put a URL to a GitHub or Jira issue, and by clicking a "link" button, these issues will be magically linked.

The process behind this has three steps:

1. Find the issue based on a URL

2. Add a new entry to ErrorIntegrationIssue with the data from the integration issue

3. Add a comment to the issue that we've linked to it

We chose a driver-based design for this. We created an interface called IssueIntegrationDriver, which looks like this:

interface IssueIntegrationDriver
{
    public function getType(): IntegrationType;

    public function getIssueByUrl(Project $project, string $url): IntegrationIssueData;
    
    public function createIssueComment(
        Project $project,
        int $issueId,
        string $body,
    ): void;
}

Each integration will have its driver, so we have a GitHubIntegrationDriver and a JiraIntegrationDriver. The getIssueByUrl method will first parse the string URL given and try to find the issue id (for a GitHub URL like https://github.com/spatie/flareapp.io/issues/481, this would be 481). Then, we'll use the provided Flare project and the Flare team linked to it to fetch the required data for the API (the repository, GitHub owner of the repository, and API tokens) and call the GitHub API to fetch the issue.

Next, a GitHubIntegrationIssueData object is created from the API issue, which saves an ErrorIntegrationIssue model. Lastly, we call createIssueComment and use the URL to the Flare error as a body.

All of this looks like this action:

class LinkIntegrationIssueWithErrorAction
{
    public function execute(
        Project $project,
        Error $error,
        string $url,
    ): ErrorIntegrationIssue {
        $driver = $project->getIssueDriver();

        $data = $driver->getIssueByUrl($project, $url);
        
        $errorIntegrationIssue = ErrorIntegrationIssue::create([
            'error_id' => $error->id,
            'integration_type' => $data->getType(),
            'integration_id' => $data->getId(),
            'data' => $data,
        ]);

        $flareUrl = ErrorLinksData::from($error)->show;

        $driver->createIssueComment(
            $project,
            $errorIntegrationIssue->integration_id,
            "Issue was linked with Flare error: [{$error->exception_message}]({$flareUrl})"
        );

        return $errorIntegrationIssue;
    }
}

This action is tied to a controller, which first checks if there's an issue in the integration for the URL; otherwise, a validation error is shown. The beauty of this approach is that we're done here. This code should not be changed when we want to add another integration. We simply implement the interface, and that's it. The feature works!

Considerations

Again, there are a few considerations to be made. First, the drivers will become large when we count line numbers. Our GitHub driver is 360 lines, and our Jira driver is 380. They will grow bigger in the future when we add even more features.

Secondly, we briefly discussed the abstract data structures beside IntegrationIssueData. These are ProjectIntegrationConfigData on a project and two structures on a Flare team called: IntegrationConnectionSecretsData and IntegrationConnectionConfigData. A Project comes in with ProjectIntegrationConfigData in the interface methods. We don't know if it's GitHubProjectIntegrationConfigData or JiraProjectIntegrationConfigData.

This means that for a lot of methods, we'll have the following code:

public function createIssueComment(
    Project $project,
    int $issueId,
    string $body,
): void {
    if (! $project->config instanceof JiraProjectIntegrationConfigData) {
        throw new Exception('Invalid config');
    }
    
    // the API calling stuff
}

This code is required to ensure that the correct instances are passed to the methods and adds IDE completion, which is the most significant benefit. It gets even worse when we also need to check IntegrationConnectionSecretsData and IntegrationConnectionConfigData, sometimes adding a whopping three extra checks making our method less readable.

To make the code a bit more readable, we've added some helper methods like this:

private function ensureProjectConfigData(Project $project): JiraProjectIntegrationConfigData
{
    if (! $project->config instanceof JiraProjectIntegrationConfigData) {
        throw new Exception('Invalid config');
    }

    return $project->config;
}

Making the method a bit more readable:

public function createIssueComment(
    Project $project,
    int $issueId,
    string $body,
): void {
    $config = $this->ensureProjectConfigData($project);

    // the API calling stuff
}

I don't think these two disadvantages weigh up against all the "free" functionality we get by using this abstraction. Adding a new integration is as easy as implementing the interface for the integration, and we're done.

Processing Webhooks

Lastly, webhooks. These differ significantly between integrations; they have other methods for verifying the webhook is sent from the integration itself and not some bad actor. The payload of an issue created webhook looks different between each integration. Also, when a webhook is sent, it can differ depending on the integration.

But is this really true? In the end, we can see that there is a common ground for all the integrations; each of these integrations sends an event in these cases:

• group created/updated/deleted

• issue created/updated/deleted

• issue comment created/updated/deleted

Can we also make abstractions here?

We create a specific webhook controller for each integration. Since these integrations all have specific verification systems, we can't go around this. What we can do is create a generalized IntegrationWebhook model. It keeps track of a few things:

• Which connection to an integration on a Flare team does the webhook belong to

• Possibly the project the webhook is connected to (not relevant to groups)

• A unique webhook id so that a webhook is only processed once

• The payload of the webhook request

• The headers of the webhook request

A webhook controller for GitHub now looks like this:

class GitHubWebhookController
{
    public function __invoke(
        Request $request,
        IngestIntegrationWebhookAction $ingestIntegrationWebhookAction
    ) {
        if ($this->isSignatureValid($request) === false) {
            return response()->json(['message' => 'Invalid signature'], Response::HTTP_FORBIDDEN);
        }

        $payload = $request->all();

        $connection = IntegrationConnection::query()
            ->where('type', IntegrationConnectionType::GitHub)
            ->where('identifier', $payload['installation']['id'])
            ->first();

        if (! $connection) {
            return response()->json(['message' => 'Connection not found'], 200);
        }

        $message = $ingestIntegrationWebhookAction->execute(
            $connection,
            $request,
            $request->headers->get('X-GitHub-Delivery'),
            $payload['repository']['name'] ?? null,
        );

        return response()->json(['message' => $message]);
    }

    private function isSignatureValid(
        Request $request
    ): bool {
        // Some code to check the request
    }
}

The IngestIntegrationWebhookAction has four parameters:

• connection the connection on the team with the integration (stores all API secrets and config)

• request a Laravel request object

• webhookId a unique id provided by GitHub so that we don't process the webhook twice

• integrationGroupId is a nullable id to a group we use to find the project. In the case of GitHub, this is the repository name

The IngestIntegrationWebhookAction will now create the IntegrationWebhook model or stop if we already have an IntegrationWebhook model with the webhookId due to an earlier webhook being sent.

It tries to find a related project when integrationGroupId is provided and will trigger a job where the webhook will be handled so that the whole request to this controller is quick so that we can send a 200 response to the integration in a matter of milliseconds.

Within the job, the specific IssueIntegrationDriver for the integration based upon the connection is retrieved, and getWebhookHandler is called. This method will return an IntegrationWebhookHandler again an interface so that we'll have a JiraIntegrationWebhookHandler and GitHubIntegrationWebhookHandler and even more when we implement more integrations. These handlers have two tasks: trigger events that will interact with ErrorIntegrationIssue and ensure our system stays in sync with the integration.

When do we need to keep staying in sync? For example, we save the repository's name on the Flare project with the GitHub integration. But, a repository name can be changed in the GitHub UI. So when that happens, all the API requests using the old repository name will fail. That's why we're reacting to the GitHub repository renamed event and update the project. These actions are not required with Jira, where projects are identified by an id that cannot change.

On the other hand, we can still abstract away some code. When an issue is created or updated, it doesn't matter if it happened within GitHub or Jira. We only need to know when the issue was created or updated and with which data. That's why we can often send out an IntegrationEvent, which will be handled by our IntegrationEventHandler. If an issue is created or updated, we will trigger an IntegrationIssueUpdatedOrCreated event.

A simplified version of the GitHubIntegrationWebhookHandler would look like this:

class GitHubIntegrationWebhookHandler implements IntegrationWebhookHandler
{
    public function handle(
        IntegrationConnection $integrationConnection,
        ?Project $project,
        IntegrationWebhook $webhook,
        array $payload,
        HeaderBag $headers,
    ): void {
        $event = $headers->get('X-GitHub-Event');
        $action = $payload['action'];


        if ($event === 'repository' && in_array($action, ['opened', 'edited', 'closed', 'reopened', 'assigned', 'unassigned'])) {
            $this-> handleRepositoryRenamed($integrationConnection, $payload, $headers, $action);

            return;
        }

        if ($project === null) {
            throw WebhookNotMatched::noProjectFound($payload, $headers);
        }

        if (
            in_array($event, ['issues', 'pull_request']
            && in_array($action, ['opened', 'edited', 'closed', 'reopened', 'assigned', 'unassigned'])
        )) {
            $this-> handleIssueUpdatedOrCreated($integrationConnection, $project, $payload, $headers, $action);

            return;
        }

        throw WebhookNotMatched::invalidEvent($payload, $headers);
    }

    private function handleRepositoryRenamed(
        IntegrationConnection $integrationConnection,
        array $payload
    ): IntegrationGroupUpdated {
        Project::query()
            ->where('integration_connection_id', $integrationConnection->id)
            ->where('integration_group_id', $payload['changes']['repository']['name']['from'])
            ->each(function (Project $project) use ($payload) {
                /** @var GitHubProjectIntegrationConfigData $config */
                $config = $project->config;

                $config->repository = $payload['repository']['name'];
                $project->config = $config;
                $project->integration_group_id = $payload['repository']['name'];

                $project->save();
            });

        return new IntegrationGroupUpdated(
            $integrationConnection,
            GitHubIntegrationGroupData::createFromWebhook($payload)
        );
    }

    private function handleIssueUpdatedOrCreated(
        IntegrationConnection $integrationConnection,
        Project $project,
        array $payload
    ): IntegrationIssueUpdatedOrCreated {
        /** @var GitHubProjectIntegrationConfigData $config */
        $config = $project->config;

        return new IntegrationIssueUpdatedOrCreated(
            $integrationConnection,
            $project,
            GitHubIntegrationIssueData::createFromWebhook($payload),
        );
    }
}

Lastly, let's quickly look at the IntegrationEventHandler. It will listen to events and react accordingly. For example, when a handleIntegrationGroupDeleted, we'll delete all ErrorIssueIntegration models for that group since it no longer exists. When an IntegrationIssueUpdatedOrCreated is triggered, we start looking for existing ErrorIssueIntegration models and update them for that issue. If there is a link to a Flare error in the created or updated issue title or body, we'll link them together.

We react to even more events in the IntegrationEventHandler, which means we don't have to write this code for each integration, which is excellent! Fixing issues in this handler will also fix issues with each integration instead of fixing each individually.

Considerations

Interfaces are great but can be restrictive, becoming a real pain. Our approach works perfectly for Jira and GitHub, but another future integration can require some extra properties, which the current interfaces don't provide. This can be a problem, but we're probably reasonably safe because we're working with a specific set of events and operations (all based on issues). And if changes need to be made in the future, we'll find a way to make it work for all our integrations.

Closing

We're almost ready to release this. We've already implemented the Jira integration into this new system and moved the existing GitHub integration. Now we're in the phase of documenting it all and making sure the fronted code works and looks great!

As a bonus, I'm delighted to announce another new feature to Flare. We wanted to add another integration since we wanted to test how waterproof this whole system was. We managed to add GitLab within two days to Flare. So, we'll launch our complete integration revamp soon with Jira, GitHub, and GitLab.

Are there any other integrations with issue trackers you're interested in? Make an issue on our roadmap.