How we built our GitLab integration

A few days ago, we added a GitLab integration. In this blog post, we'll talk about the road to that integration and our struggles.

Frequent readers of our blog might have read Building a flexible yet abstract external integrations Structure. In that technical blog post, we describe how we revamped our internal structures to allow us creating integrations like GitHub, Jira, and GitLab in days rather than weeks. If you haven't read it, be sure to read it first.

We started building these new structures with the Jira integration and used them again when we converted our existing GitHub integration. To ensure these structures were as flexible as we wanted them to be. It was required to battle-test them!

Could we set up a new Flare integration from scratch in two days? That's where GitLab comes into the picture. Could we add a GitLab integration like we did on GitHub and Jira without breaking a sweat?

At Spatie, we're using GitHub. We had no experience with GitLab, but they kinda follow the same conventions as GitHub. Issues are issues, and a repository is called a project, a team is a group, and, oh yeah, you can nest groups.

The most complex part of the GitLab integration was that GitLab exists in two flavors, like GitHub, you can use the cloud version, where you create an account, and you're ready to go. GitLab also offers the ability to be self-hosted; before you can create an account, you'll need to set up a GitLab instance on your server, and after that, you can create an account.

This self-hosted option added extra challenges for us. Because we cannot, like with GitHub and Jira, create an application for Flare on GitLab with which we can connect. Instead, a user should create its own application with which Flare can connect when using a self-hosted GitLab instance.

We provide an application for the cloud version, which makes connecting Flare with the cloud version of GitLab as complex as clicking one button, which is nice.

When installing a new GitLab connection into Flare, you will be asked to use the Cloud or self-hosted version, allowing you to choose your personal favorite.

Implementing

We spend most of our time finding out how GitLab works, reading the GitLab API documentation, and writing front-end code.

Implementing the backend code was straightforward; the integration structures we discussed earlier worked like a charm. Let's go through a few of them:

IntegrationIssueData

The IntegrationIssueData object which represents an individual issue :

class GitLabIntegrationIssueData extends IntegrationIssueData
{
    public function __construct(
        int $id,
        public int $number,
        public string $title,
        public ?string $description,
        public GitLabIssueState $state,
        public GitLabIssueType $type,

    ) {
    }
    
    public function getId(): int
    {
        return $this->id;
    }

    public static function createFromApiPayload(array $payload): self
    {
        return new self(
            id: $payload['iid'],
            number: $payload['iid'],
            title: $payload['title'],
            description: $payload['description'],
            state: GitLabIssueState::from($payload['state']),
            type: GitLabIssueType::from($payload['issue_type']),
        );
    }
}

It is still looking great. We can keep track of all the issue properties a GitLab issue has.

Thanks to the createFromApiPayload method, a GitLab API object is automatically mapped to our structure (no more iid's), keeping it simple for other developers in our team to work with the data without any knowledge of the GitLab internal structure.

IntegrationIssueDriver

The IntegrationIssueDriver is the heart of our integration system. It looks (redacted) like this:

class GitLabInstalledIntegrationDriver implements IssueIntegrationDriver
{
    public function getIssue(ProjectIntegration $projectIntegration, int $id): GitLabIntegrationIssueData
    {
        $config = $this->ensureProjectIntegrationConfigData($projectIntegration);

        $response = $this->resolveConnector($projectIntegration->integrationConnection)->send(
            new GetGitLabIssueRequest($config->project_id, $id),
        );

        return GitLabIntegrationIssueData::createFromApiPayload(
            $response->json(),
        );
    }

    public function getIssueByUrl(ProjectIntegration $projectIntegration, string $url): IntegrationIssueData
    {
        preg_match_all('/\/issues\/(?<id>\d+)$/m', $url, $matches, PREG_SET_ORDER);

        $id = $matches[0]['id'] ?? null;

        if (! $id) {
            throw CouldNotMatchUrlToIntegrationIssue::create($url);
        }

        try {
            return $this->getIssue($projectIntegration, $id);
        } catch (NotFoundException) {
            throw CouldNotMatchUrlToIntegrationIssue::create($url);
        }
    }
}

There are many more methods in the connector than this, but to keep things more readable, we will show these two methods.

The getIssue method returns an IntegrationIssueData object for an issue tied to a Flare project. It makes an API call to GitLab and uses laravel-data to map the response to a data object.

See that resolveConnector method? We use the excellent Saloon package to have an API abstraction layer between Flare and GitLab. It handles everything from authorization with OAuth, sending requests, receiving responses, and allowing us to use fakes for testing the API. Saloon is excellent. If you haven't, you should start using it now!

The getIssueByUrl method we saw in the previous blog post tries to find an issue by URL and returns an IntegrationIssueData object. After using regex magic, we use the getIssue method to fetch the issue.

We can even abstract more here. In the GitHub, Jira, and thus GitLab connectors, this method always looks the same: use a regex to extract an id and then call the getIssue method with that id returning a data object.

It would require us to transform IssueIntegrationDriver from an interface into an abstract class and add the requirement to implement resolveIssueUrlRegex. Coding this will take little time.

So why didn't we do it? Could our subsequent integration need more complex code to resolve the ID? We don't know it at the moment, but trying to abstract everything away will, in the end, make things a lot harder to implement. A bit of code duplication is not bad in these cases.

When we've implemented more than five integrations, and the code is the same, then that's the moment we'll refactor.

Webhooks

In the previous post, we had a long section about Webhooks. External platforms like Jira and GitHub send them to us, allowing Flare to react to changes on these platforms.

At the moment, we do not ingest webhooks from GitLab. That's because, unlike those other platforms, there's no option to subscribe to them automatically.

Webhooks do exist in GitLab, but a user must manually register them. For our GitLab self-hosted customers, this could be an extra step in the wizard when an app is created in their GitLab instance.

For our GitLab cloud customers, the installation process has become much more complex than clicking a button; they now should register webhooks themselves.

We could add the webhooks ourselves since we have access to the API, but adding things to a GitLab instance that isn't ours feels intrusive.

Even worse, somebody could remove those webhooks (intentionally or unintentionally), and the two-way sync would stop working. To do it "right" we need to check around certain intervals if the webhooks still exist and if they are configured correctly.

This complicated webhook setup is why we're not implementing a two-way GitLab sync for now. We do love to listen to our users. Let us know if you want to see a two-way sync in GitLab, which might make the integration a bit more intrusive.

Luckily, we've built the IssueIntegrationDriver so that all the code related to webhooks is defined in another interface called WebhookIntegrationDriver, making it possible to add a new integration to Flare without webhooks support.

Conclusion

Our GitLab integration started as an experiment to see if our code was flexible enough to support upcoming integrations. I can tell you it is! We implemented the whole integration in two days!

We love working on Flare and have a lot of new features in the pipeline, heck, maybe even a GitLab two-way sync 😉