Forest Admin's update live on NodeJS

Edited in September 2024: In the moment of publishing this article, the new Agent was only available in NodeJS. Since then we released the new versions of Forest Admin Agents for Python, PHP and Ruby on Rails, with the same capabilities as explained below.

Every product is bound to change, to evolve. Forest Admin is no exception.

The solution we offer our clients to support them in their daily activities is boosted by what we internally call “the Agent.” It is the piece of open source code that our team has built and that our clients must add in their project to use our internal tool solution. Now five years after our launch, our technology has matured, and so has our Agent.

The history

From an engineering standpoint, the design of any system requires compromising between performance, flexibility and learning curve.

Forest Admin released its Agent six years ago, when Express.JS was the most popular web framework and typing systems were scarcer. The predominant way to do async programming then was with callbacks.

In the ecosystem of the time, the design of our Agent was sound.

Its API had a low learning curve because it was based on express.js and sequelize.js, which were popular at the time.
The accessibility of our API meant a bigger deal than its performance because Forest Admin customers had smaller projects – being easy to use was more important than speed.
The company focused on a reduced number of data sources (PostgreSQL, MySQL, Microsoft SQL, MongoDB).

In the meantime, JavaScript’s ecosystem has substantially evolved, and although we could’ve introduced changes in our API at every turn, this would have greatly disturbed our customers’ daily operations.

We have been working on a new version of the agent for a while now, and we are happy to announce that it is now available.

The Agent’s job

The Agent is the beating heart of Forest Admin.

Looking at the infographic below, the Agent resides on the customer’s server.

The Agent is, once again, a piece of open source code the customer must plug into its database so that it can work its magic. Think of it as a well-trained butler trained by Forest in order to manipulate and serve you your data whatever way you wish to.

It is crucial to understand though that this agent is working to the service of the customer, not ours. This means that it manipulates data at will but never leaks it.

The old Agent is available for 5 different stacks so far: Express/Sequelize, Express/Mongoose, Ruby on Rails, Django and Laravel – each of them having been through various stages of development.

The new Agent, what has changed

Web development’s ecosystem has grown considerably since we first released our product.

Frameworks such as Fastify, Nest.js and Hapi have launched – so did Object Relational Mappers (ORM) like TypeORM, MikroORM, Prisma, and Knex.

This partly explains why we have decided to make our new Agent more flexible, so as to allow our customers to choose the technologies they want to use.

This new Agent no longer cares about either the web framework or the ORM in use – it natively integrates with Express.js, Fastify, Nest.js, and Koa.

Customers working on other frameworks can still use the Agent by mounting it in standalone mode and by using a reverse proxy.

You can also connect to your database directly and use automatic model introspection: you will no longer need to maintain Sequelize.js models if you are using another ORM for your production app.

We still support Sequelize.js and Mongoose.js, which gives you additional benefits – such as improving code reuse from your application, even though it isn’t required anymore.

Support multiple data sources

With the advent of the Software-as-a-Service (SaaS) industry, the number of data sources that you need to manage has increased.

Many apps now use specific data sources for specific features – ElasticSearch for search, Redis for caching, etc – and mix relational and non-relational databases.

With our first Agent, the paradigm was to have a single database connection, and Forest Admin released a different library for each data source type.

We now support multiple data sources in the same project: you can connect as many as you need, combine relational and non-relational databases, and use different ORMs for each one.

You are also free to implement your own connectors to any data source that you want, even to your in-house APIs.

Autocompletion and typing system

In 2016, the vast majority of projects did not use a typing system.

This is no longer the case and we adapted to that change as well: the new agent is completely written in TypeScript, and automatically generates type definitions on the databases that are plugged into your agent.

When coding you will get autocompletion and type-checking on your models.

Functional approach

We sell a product made by developers for developers.

As such, our customers have a unique understanding of the value and limits of our product, which makes their feedback more invaluable than in any other industry.

We've made a major change in the way we design our API and are now using a more functional approach: instead of having a generated codebase that you modify, you now start from an empty project and add custom behaviors by registering your logic with side-effect-free functions.

As a customer, the new API is much higher-level: you won't be dealing with low-level concepts such as "routes" or "query-string", but instead you will be dealing with "actions", "fields", and "segments".

Performance

Our first customers were small startups that have grown alongside us. They now harbor dozens of collections and millions of users – we need to be able to scale with them.

Many of the performance improvements that come with this new Agent would not have been possible with the first one because of API stability.

The first Agent’s API is full of compromises between performance and accessibility. Some of those compromises are no longer relevant today, so we have decided to head in a different direction.

Let’s focus on only one example here. To define a smart field with the first Agent, you had to define a getter function that would be called with a record as parameter.

This was the easiest way to implement smart fields for our customers but, let’s be honest, it was also the slowest. Displaying a table with 30 records required us to call the code written by a given customer 30 times, which in turn might call the database 30 times.

To avoid that:

The new agent allows you to define functions that will be called in batches: this drastically reduces the number of database calls.
Smart-field dependencies are defined in a declarative way: this decreases the number of columns that are fetched from the database – since we no longer need to fetch the complete records.

See comparison below.

A Field declaration in the old generation agents.

A Field declaration in the new generation agents.

Onward

Although in its infancy, this new Agent has the potential to fast track adoption as it both eases integration and opens the door to more complex scenarios – Pledger is a prime example.

We, at Forest Admin, are always trying to improve our solution and do not hesitate to take risks if it means better serving those who put their trust in us – just as we did with our new pricing policy.