Lightning-fast search using Symfony Algolia Bundle
Part of the "Under the hood" series of posts, detailing how stackselect.tech is built, and giving you a better idea how some tools listed on stackselect.tech work.
Algolia helps businesses across industries quickly create relevant, scalable, and lightning fast Search and Discovery experiences.
I'd like to share a little overview of how the search works on stackselect.tech. You can see it in action just above!
Without the search, if you're searching for a tool on the site, you have to know which tag it might be in. That's a big ask, especially if you're just beginning a side-project and don't know where to begin. The search just quickens this whole experience.
Algolia is pretty well-known and widely used. It has a free tier and a Symfony Bundle that looked well maintained. I'm not 100% sure yet at what point I'll need more than the free tier offers, but with relatively little traffic right now, it's the easiest and quickest option with a good developer and user experience.
Defining the goal
Simple as it seems, it's good to describe what we want to achieve. In this case we want something to match a term if that term appears in either the name of the tool, one of its tags, or its description (in code that's called the summaryLine).
For every listing that's returned, I want to display the name and the description, along with its image.
First attempt
It's pretty common in Symfony to use annotations for the configuration of bundles or integrations. The Doctrine ORM uses them, API platform and EasyAdmin too. The AlgoliaBundle supports the same. I don't love them as a configuration method, but they're fast to set up.
This involves importing a namespace and configuring a Symfony Serializer group on each of the name, slug, summaryLine and tags properties.
use Symfony\Component\Serializer\Annotation\Groups;
/**
* @Groups({"searchable"})
* @ORM\Column(type="string")
*/
private string $name;
Next I configured the bundle to recognise this entity.
# All available configuration can be found here:
# https://www.algolia.com/doc/api-client/symfony/configuration/
algolia_search:
# ...
indices:
- name: tools
class: App\Entity\Tool
enable_serializer_groups: true
The bundle comes with a few commands to manage the search index. One of these is search:import. This iterates through the database and synchronises the index in Algolia:
$ ./bin/console search:import
Indexed 64 / 64 App\Entity\Tool entities into stackselect_dev_tools index
Done!
That's great, but there's a little problem here. Not all 64 items should be public. Some don't have enough details, or haven't met the criteria to be selected for showing on the site. They're hidden by a boolean published field, but the AlgoliaBundle doesn't know about that yet.
I added index_if: isPublished to the configuration and added a new public function isPublished(): bool to the Tool entity.
# All available configuration can be found here:
# https://www.algolia.com/doc/api-client/symfony/configuration/
algolia_search:
# ...
indices:
- name: tools
class: App\Entity\Tool
enable_serializer_groups: true
index_if: isPublished
Indexed 59 / 64 App\Entity\Tool entities into stackselect_dev_tools index
Done!
Perfect!
Configuring the environment with webpack encore
I'm not going to go into the frontend configuration too much, because it's not my area of expertise, and the documentation does a much better job than I can. I used the Vue InstantSearch package. If you want some extra details in particular, leave a comment.
A small difficulty was getting the API key to be populated from an environment variable. In the webpack.config.js file, I used the dotenv to add some extra configuration.
var dotenv = require('dotenv');
//...
.configureDefinePlugin(options => {
const env = dotenv.config();
if (env.error) {
throw env.error;
}
options['process.env'].ALGOLIA_SEARCH_KEY = JSON.stringify(env.parsed.ALGOLIA_SEARCH_KEY);
})
Importantly, this doesn't read environment files using the same prioritisation and environment logic as Symfony. A .env.dev.local won't be read by this configuration. After some trial and error, I ended up simplifying this to use .env and .env.dist, with the dist version being the only committed file.
This works with both Symfony and the dotenv package. I'll just make sure each environment has either this file, or real environment variables. .env.test still exists for the PHPUnit tests.
Displaying an image in the Algolia search results
Currently the images for each service are committed as static files and are included in the webpack build process. On production this results in images named like tools/{slug}/favicon.abc1234.png. That hash is part of the Webpack Encore cache busting mechanism.
The client-side code for the search is written in Vue. When Algolia returns a record, we have access to item.slug representing the slug of the tool but we can't convert that into the correct hashed version at runtime.
Symfony has asset functions in Twig which will find the right asset location, by reading the manifest but since we're in JS code at this point, this isn't available.
I solved this by adding a new route, /tools/{slug/favicon which would locate the right file and redirect to it. I added cache headers to speed this up, to ensure Cloudflare would serve the images as quickly as possible.
I'm not going to share that code, because the experience was terrible. It was really slow. Each letter typed resulted in a new list, with new img tags causing a request, redirecting to another request. It just wasn't good enough.
Second attempt (fast loading of images in search)
It bothered me that there were too many image requests, and I wanted to reduce that number. In the end I found a way to do it with zero image requests.
These icons are favicons. They're pretty small. I decided to index the icon data itself and have Algolia immediately return it.
Before explaining in detail, I'll paste the code. It's not pretty, but it works. There's plenty of things yet to be abstracted out of this class. But this illustrates the logic, all in one place.
<?php
declare(strict_types=1);
namespace App\Serializer\Normalizer;
use Algolia\SearchBundle\Searchable;
use App\Entity\Tool;
use Imagine\Image\Box;
use Imagine\Imagick\Imagine;
use League\Flysystem\Filesystem;
use Symfony\Component\Serializer\Normalizer\NormalizerInterface;
class ToolNormalizer implements NormalizerInterface
{
/**
* @var Filesystem
*/
private Filesystem $tools;
public function __construct(Filesystem $tools)
{
$this->tools = $tools;
}
public function normalize($tool, string $format = null, array $context = [])
{
/** @var Tool $tool */
$faviconPath = "{$tool->slug()}/favicon.png";
$dataUrl = null;
if ($this->tools->has($faviconPath)) {
$imagine = new Imagine();
$faviconContentFullSize = $this->tools->read($faviconPath);
try {
$favicon = $imagine->load($faviconContentFullSize);
} catch (\RuntimeException $e) {
throw new \LogicException("Cannot read {$tool->slug()}");
}
$favicon->resize(new Box(30, 30));
ob_start();
$favicon->show('png');
$png = ob_get_clean();
$dataUrl = "data://image/png;base64," . base64_encode($png);
}
return [
'slug' => $tool->slug(),
'name' => $tool->name(),
'summaryLine' => $tool->summaryLine(),
'tags' => $tool->tags(),
'dataUrl' => $dataUrl
];
}
public function supportsNormalization($data, string $format = null)
{
return $data instanceof Tool && Searchable::NORMALIZATION_FORMAT === $format;
}
}
This class replaces the annotations. They don't support what we need here, since we're reading from files, and doing extra conversion we can't do in an entity class. This is an example of why I don't like much the annotation classes, since it limits the flexibility, and often doesn't cover more complex cases. The PHP version is explicit in what it does, even if it's more verbose.
We're using Flysystem here to source the images, then we're passing that into the Imagine library, before resizing the images. Some of the icons actually have large dimensions, and it turns out there's a 10kb limit on the record size in Algolia. Since these icons are meant to be small, even with the extra size of converting them to base64 this is plenty.
After resizing we do a little conversion so that the image is represented in a data uri. When returned by Algolia this can be put straight into the src attribute of the img tag and will be shown instantly.
The extra controller we added earlier was also removed. It never felt right anyway.
The final result
I'm pretty pleased with how it works in the end. Try it out at the top of the page. What do you think? Have you used another service or tool for search which worked well?
Follow along on Twitter for more articles like this, and for great tools you can use in your next project.
Algolia helps businesses across industries quickly create relevant, scalable, and lightning fast Search and Discovery experiences.
