Building a Rule Engine With TypeScript
What is a Rule Engine?
A rule engine reads from a place where rules are defined (e.g. in a JSON file, a key-value database etc), checks rule conditions against an object or array of objects, and executes the rules for objects that match the conditions. With a rule engine, we get to manage business behaviour separately from code architecture. This means that we’re able to change business behaviour when the need arises without touching application source code.
When Do You Need A Rule Engine?
• Is there a part(s) of your application where you perform some operations on an object or array of objects based on predefined conditions?
• Do those conditions change? Are they decided by people outside your development team?
• Do you want to accommodate the changes without modifying application source code?
If your answer is “yes” to some or all of the questions above, you should consider a rule engine.
Rule Engine Use Cases
Let’s delve into some cases where it makes perfect sense to use a rule engine. Although most of the use cases we’re about to consider are imaginary, they’re based-off real world scenarios.
Personalised Search
Consider an imaginary music streaming app where a family subscription allows you to add up to 3 members of your family. This app allows the account admin to update app preferences such as filtering out music tracks with strong language from search results when the logged-in user’s age is below 13. In addition, a user can download music tracks to their local device. Where an already downloaded song appears in search results, that song gets a “downloaded” badge.
While we could use if/else statements to handle the business logic above, using a rule engine abstracts the rules from our application source code. If there’s the need to add more rules (e.g. if a music track was previously rated 4 out of 5 stars by the app user, the stars should show for that music track in search results), our implementation stays dynamic.
Handling Externally-Controlled Business Logic
You’re working on a flight booking app that allows end-users search for flights, see results and book an itinerary. The app relies on third-party APIs to provide end-users with flight data, and lets end-users book only flights they’re eligible for. The product team wants a different booking experience for end users: if an end user is not eligible for an itinerary, then we shouldn’t include it in search results. We don’t want to give them what they cannot book.
A search result may include multiple airlines, and the airlines have different conditions around who can fly and what is required to fly. So, we collect some data about the end-user who wants to book a flight and keep it in a FlightBooker object. The data includes what visas the end-user holds, and the maximum number of stops the person wants. Then for each paginated search result, we compare the FlightBooker object to rules defined for specific airlines, and omit itineraries that the end-user is not eligible for. The app also allows end users to view omitted itineraries, including the reasons why they were omitted e.g “This itinerary requires a UK transit visa which you made us understand that you don’t have”.
The rules guarding airline operations are defined by aviation standards, government policies as well as internal management decisions. For example, Airline X may disallow flying with infants that are less than 1 year old, but Airline Y has no problem with that. Airline Q may require a transit visa when flying to specific locations, but Airline R does not require such transit visa. These condition could change later. When it changes and how often it does is clearly controlled outside your business domain. You should not bake the conditions into your application code because if and when they change, your code changes too. It makes sense to define rules for the conditions, and create a rule engine to apply the rules accordingly.
Pre-computed Ranking
An imaginary app called “Elite Estate” connects home seekers to real estate agents. However, the app has decided to honour top 10 agents by listing them on the homepage based on some rule set by the management of Elite Software Group LLC. Everyday, a background job runs to calculate the ranking score for each estate agent. Every time the job starts, the ranking score of all agents is zero by default. The rules for scoring agents include:
- For each property sold, we increase ranking score by 0.5.
- If the agent has sold more than 100,000 USD for the day, we increase score by 1.
- If the average rating given by real estate buyers/prospective clients who have interacted with an agent is 5/5, we increase the agent’s score by 1.
For this use case, we want to define rules that accommodate the present logic for scoring agents. The rules for ranking the real estate agents may change from time-to-time, as decided by management.
Some Factors to Consider When Designing a Rule Engine
Structure of the Target Object
[
{
"title": "Heal the World",
"artist": "Michael Jackson",
"year": 1991,
"album": "Dangerous",
"tags": ["pop", "society"]
},
{
"title": "The Girl From Last Night",
"artist": "Jackson Pepper",
"year": 2017,
"album": "Slot Machine",
"tags": ["rap", "strong language"]
},
...
]Above is a sample search result for the imaginary music streaming app use case. It’s important to know the structure of the object we intend to write rules for, and how that object may change. The beauty about a rule engine is that if the object changes, we only need to adjust the rules we defined, and not our rule engine implementation.
Rule Structure and Format
How do we structure rules? Should we define them in JSON or YAML format? How do we optimise rule definitions for reading/fetching? We’ll obviously read rules more often than we create rules, right? For the purpose of this tutorial, we’ll define our rules in JSON format. We’ll go into details of the structure for rules later.
Where to Store the Rules
Where do we keep rules to ensure fast read access? Do we want an object storage such as AWS S3 or Minio where may store a JSON file, update, fetch and parse it when necessary? Or is a SQL or NoSQL database preferable? How will a relational or non-relational database option impact latency in our application since we have to query a database to fetch rules? Or do we prefer a key-value store that offers real-time read access such as Redis? Are we interested in keeping version history to track changes in rules? These and more are the questions we need to ask. For the purpose of this tutorial, we’ll use a rules.json file kept in an object storage. Keeping the rules.json file within our project is also an option; however, this would mean that a deployment is necessary whenever a new rule is added or updated to replace the rules.json file on our server.
A solution that ensures read-speed would be to use an in-memory data store such as Redis, and adopt persistence to save rules to disk so that they can be restored if the Redis server crashes or is shutdown.
Actions to Run When an Object Matches a Rule
What actions do we intend to run when an object matches a rule? Are we modifying the object based on some logic? It’s salient to have a list of possible actions before we build a rule engine. These actions may include:
- Increment: add to the numeric value of an object key
- Decrement: subtract from the numeric value of an object key
- Add: add a key to an object and set the value
- Replace: replace the value of an object key
- Omit: exclude an object from an array of objects
- Omit with silent error: exclude an object from an array of objects, and return an error object that explains why it was omitted.
The actions could be more, depending on your use case. Now let’s move on to how to structure rules.
The Structure for Rules
We need to come up with a clean, readable, testable rule structure that makes it easy to add and update rules without changing the logic in our application source code. Consider the rule structure below for the imaginary music streaming app:
{
"trackHasStrongLanguage": {
"conditions": [
["$.loggedInUser.age", "lessThan", 13],
["$.tags", "contains", "strong language"]
],
"effect": {
"action": "omit"
}
},
"trackIsDownloaded": {
"conditions": [
["$.downloadedTracks", "hasKey", "$.title"]
],
"effect": {
"action": "add",
"property": "downloaded",
"value": true
}
}
}Our rules structure looks like a dictionary. We opt for a dictionary approach so that a rule can be fetched in constant time using it’s unique key. Each rule has one or more conditions. conditions is a tuple having a fixed length of three items where the first and third are either object keys or values, while the value in the middle is the operator. Effect defines what should happen if an object satisfies all conditions defined for a rule. For example, the trackHasStrongLanguage rule is matched only when the logged-in user’s age is less than 13, and the track object has a “tags” key whose value is an array of strings that includes “strong language”. The effect of that rule is that any matched music track is omitted from search results.
Did you notice the use of a dollar sign in $.loggedInUser.age? This tells the rule engine that loggedInUser.age refers to a key in a music track object. We’ll build our rule engine to understand the dollar symbol. Without the dollar symbol, the rule engine will treat it as a string. It’s also noteworthy that you can use the dot notation syntax to access nested object properties as used in the trackHasStrongLanguage rule. Our rule engine will be built to handle this.
Defining Rule Structure With Types
Using TypeScript, we define rule structure as follows:
Rule and RuleError
type Rule = Record<string, {
conditions: Condition[];
effect: Effect;
}>;
type RuleError = { message: string };Condition with Operator enum
type Condition = [string, Operator, string];
enum Operator {
CONTAINS = "contains",
EQUALS = "equals",
GREATER_THAN = "greaterThan",
GREATER_THAN_OR_EQUALS = "greaterThanOrEquals",
LESS_THAN = "lessThan",
LESS_THAN_OR_EQUALS = "lessThanOrEquals"
}Effect with Action enum
type Effect = {
action: Action;
property?: string;
value?: number;
error?: { message: string; }
};
enum Action {
ADD = "add",
DECREMENT = "decrement",
INCREMENT = "increment",
OMIT = "omit",
OMIT_WITH_SILENT_ERROR = "omit_with_silent_error",
REPLACE = "replace"
}The Rule Engine Class
Our rule engine will be a Node module that exports a class. The class will include a conditions checker method checkForMatchingRules, and an effect runner method runEffect. The conditions checker takes an object (a track object in this case) and checks for whether it matches any rule defined in rules.json. For each rule matched, the effect runner is called. The effect runner takes the object and the rule effect as arguments, runs the effect on the object, and returns a modified copy of the object.
export class RuleEngine {
private readonly rules: Rule;
constructor(rules: Rule) {
this.rules = rules;
}
getRule(ruleName: string): { conditions: Condition[]; effect: Effect } {
if (this.rules[ruleName]) return this.rules[ruleName];
throw new Error(`No rule found for ${ruleName}`);
}
getOperator(operator: Operator): string {
switch (operator) {
case Operator.CONTAINS:
return Operator.CONTAINS;
case Operator.EQUALS:
return '===';
case Operator.GREATER_THAN:
return '>';
case Operator.GREATER_THAN_OR_EQUALS:
return '>=';
case Operator.LESS_THAN:
return '<';
case Operator.LESS_THAN_OR_EQUALS:
return '<=';
default:
throw new Error(
`No matching operator found for ${operator}`,
);
}
}
getFieldValue<T, F>(field: string, object: T, fallbackObject?: F): string {
if (field[0] === '$') {
const value = this.getObjectKeyValue<T>(field.substring(2), object);
if (value) return value;
if (!fallbackObject)
throw new Error(
`Object has no ${field.substring(2)} key, and no fallback object was provided`,
);
return this.getObjectKeyValue<T>(field.substring(2), fallbackObject);
}
return field;
}
getObjectKeyValue<T>(key: string, object: T): string {
const keys: string[] = key.split(".");
const value = keys.reduce((accumulator, currentValue) => {
if (!accumulator[currentValue])
throw new Error(`Object has no ${currentValue} key`);
return accumulator[currentValue];
}, object);
return value;
}
conditionIsTruthy(left: string | string[], operator: Operator, right: string): boolean {
const comparator = this.getOperator(operator);
if (comparator === Operator.CONTAINS) {
return Function(
`"use strict"; return ('${left}'.includes('${right}'))`,
)();
}
return Function(
`"use strict"; return ('${left}' ${comparator} '${right}')`,
)();
}
runEffect<T>(object: T, effect: Effect): T | null {
const clonedObject = <T>structuredClone(object);
switch (effect.action) {
case Action.INCREMENT:
clonedObject[effect.property] += effect.value;
break;
case Action.DECREMENT:
clonedObject[effect.property] -= effect.value;
break;
case Action.REPLACE:
clonedObject[effect.property] = effect.value;
break;
case Action.OMIT:
case Action.OMIT_WITH_SILENT_ERROR:
return effect.action;
default:
throw new Error(
`${effect.action} not found in effect runner`,
);
}
return clonedObject;
}
checkForMatchingRules<T, F>(object: T, fallback?: F): string[] {
const matchedRules: string[] = [];
for (const ruleName in this.rules) {
const { conditions } = this.getRule(ruleName);
let numberOfRulesMatched = 0;
for (const condition of conditions) {
const [leftField, operator, rightField] = condition;
const left = this.getFieldValue<T>(leftField, object, fallback);
const right = this.getFieldValue<T>(rightField, object, fallback);
if (this.conditionIsTruthy(left, operator, right)) {
numberOfRulesMatched++;
} else break;
}
if (numberOfRulesMatched === conditions.length) {
matchedRules.push(ruleName);
}
}
return matchedRules;
}
applyRules<T, F>(objects: T[], fallback?: F): ApplyRulesResponse<T> {
const results: T[] = [];
const omitted: [T, RuleError][] = [];
for (const object of objects) {
const matchedRules: string[]
= Engine.checkForMatchingRules<T, F>(object, fallback);
if (!matchedRules.length)
return results.push(object) && { results };
for (const ruleName of matchedRules) {
const { effect } = Engine.getRule(ruleName);
const feedback = Engine.runEffect<T>(object, effect);
switch (feedback) {
// if feedback is "omit" or "omit_with_silent_error",
// then we should omit the current object by not pushing
// it to results array.
case Action.OMIT:
omitted.push([object, null]);
continue;
case Action.OMIT_WITH_SILENT_ERROR:
omitted.push([object, { message: effect.error.message }]);
continue;
default:
// push to results array by default
results.push(object);
}
}
}
return { results, omitted };
}
}getRule
The getRule method simply fetches a rule from rules.json using the rule name. If no rule matches the provided rule name, an error is thrown.
getOperator
The getOperator method returns the operator that corresponds to the provided operator name. It throws an error if no operator that matches the provided operator name is found.
getFieldValue
This method takes a string field, an object object of type T, and an optional object fallbackObject. The first two characters of the field string is checked to see if it equals $. e.g. if the string is $.downloadedTracks, it uses the getObjectKeyValue method to check the provided track object for a downloadedTracks key. If the key is found, the value is returned. Else, it checks the fallback object for a downloadedTracks key. However, if the provided field string does not start with $., that string is returned as is.
You should provide a fallback object where one or more rules compare values across different objects. For example, let’s revisit the conditions defined for the trackHasStrongLanguage rule:
[
["$.loggedInUser.age", "lessThan", 13],
["$.tags", "contains", "strong language"]
]The first rule requires that the object passed to getFieldValue() contains a loggedInUser object that has a age key, while the second condition requires that the object passed to getFieldValue() contains a tags key. It makes sense for a Track object to have a tags key, but not a loggedInUser key. This is the reason why provision is made to use a fallback object. If a key is not found in the provided Track object, the rule engine checks the fallback object for that key. It would make much sense to provide the logged-in user as a fallback User object in order to prevent checks for the ["$.loggedInUser.age", "lessThan", 13] condition from throwing an error.
To conclude on getFieldValue(), notice the use of generic types T and F for the method arguments. It allows our rule engine the flexibility of working for more than Track objects. We could do getFieldValue<Track, User>() when working with a track and a user object, or getFieldValue<RecordLabel>() if we decide to work with a RecordLabel object.
getObjectKeyValue
This method takes a key string that follows the dot notation syntax to access object properties (object.key.subKey), and an object. It splits the key on the . character, and attempts to recursively access the property of the provided object.
conditionIsTruthy
This method constructs a string expression and evaluates whether the expression is truthy. It uses Function(), a close relative to JavaScript eval(), to evaluate the expression based on any of the operations defined under Operation enum earlier.
runEffect
This method clones the object passed to it (in our case, a Track object), and performs the provided effect. Recall that an effect defines what should happen if an object satisfies all conditions defined for a rule. runEffect uses a switch statement to define expressions to be executed for each effect action.
checkForMatchingRules
This method takes a generic object object (a Track object in our case) and another generic object fallback, runs the music track against every rule in our rule engine instance, and returns an array of rules whose conditions are satisfied by the Track object. If a key specified in a rule does not exist in the object, our rule engine looks up the key in fallback.
applyRules
This method applies the rules passed to our rule engine instance on an array of objects. The use of generic types T and F for the applyRules function arguments allows us to provide the object and fallback object types when calling this method. applyRules returns an ApplyRulesResponse object that has a results key and an optional omitted key. results is an array of objects having the same generic type T mentioned earlier. omitted is an array of tuples where each tuple has its first item as an object of the same generic type T, while the second item is an object of type RuleError. See the definition below:
type ApplyRulesResponse<T> = {
results: T[];
omitted?: [T, RuleError][];
};Testing the Rules
One of our objectives is to develop testable rules and a testable rule engine. While we won’t write unit tests that cover the whole Rule Engine here, we will write a sample test for the trackHasStrongLanguage rule to be sure that the rule does what we expect.
describe("rules", () => {
describe("trackHasStrongLanguage", () => {
const rules = {
trackHasStrongLanguage: {
conditions: [
["$.loggedInUser.age", "lessThan", 13],
["$.tags", "contains", "strong language"],
],
effect: {
action: "omit",
},
},
};
const tracks = [
{
title: "Heal the World",
artist: "Michael Jackson",
year: 1991,
album: "Dangerous",
tags: ["pop", "society"],
},
{
title: "The Girl From Last Night",
artist: "Jackson Pepper",
year: 2017,
album: "Slot Machine",
tags: ["rap", "strong language"],
},
];
it("should filter out tracks with strong language if logged-in user's age is below age set in rule", () => {
const ineligible = {
loggedInUser: {
id: 1,
age: 11,
},
};
const Engine = new RuleEngine(rules);
const { results, omitted } = Engine.applyRules<Track, CurrentUser>(tracks, ineligible);
const [ omittedTracks, ] = omitted;
expect(results.length).toEqual(1);
expect(results[0].title).toEqual("Heal the World");
expect(omittedTracks[0].title).toEqual("The Girl From Last Night");
});
it("should NOT filter out tracks with strong language if logged-in user's age is above age set in rule", () => {
const eligible = {
loggedInUser: {
id: 1,
age: 19,
},
};
const Engine = new RuleEngine(rules);
const { results, omitted } = Engine.applyRules<Track, CurrentUser>(tracks, eligible);
const [ omittedTracks, ] = omitted;
expect(results.length).toEqual(2);
expect(results[0].title).toEqual("Heal the World");
expect(results[1].title).toEqual("The Girl From Last Night");
expect(omittedTracks.length).toEqual(0);
});
it("should NOT filter out tracks with strong language if logged-in user's age is the same as age set in rule", () => {
const eligible = {
loggedInUser: {
id: 1,
age: 13,
},
};
const Engine = new RuleEngine(rules);
const { results, omitted } = Engine.applyRules<Track, CurrentUser>(tracks, eligible);
const [ omittedTracks, ] = omitted;
expect(results.length).toEqual(2);
expect(results[0].title).toEqual("Heal the World");
expect(results[1].title).toEqual("The Girl From Last Night");
expect(omittedTracks.length).toEqual(0);
});
});
});It’s important to test every rule we create to give us the confidence that it does what we expect it to do.
Conclusion
There are a few, open-source rule engine libraries out there that save you the trouble of writing and maintaining a custom rule engine. However, some were written in a different programming language, and would require much work to integrate with a Node.js application. In addition, some of the most starred libraries have a chain of issues reported, but haven’t seen a new commit in years. Hence the custom, no-NPM-dependencies implementation we did in this tutorial. When you understand how a rule engine works, you can create one in Rust, Scala or any programming language of choice.
Eventually, we delved into only the music streaming app use case. Nonetheless, with the flexibility of our rule engine, we should be able to define rules for the flight booking app use case as follows, and have our rule engine work just fine:
{
"infantIsLessThanAgeOne": {
"conditions": [
["$.airline", "equals", "Rafiki Airlines"],
["$.infants.ageOfYoungestInYears", "lessThan", 1],
],
"effect": {
"action": "omit_with_silent_error",
"error": {
"message": "This airline prohibits flying with an infant less than 1 year old"
}
},
},
}Then we can apply rules to a flight search result array like this:
const flightBooker: FlightBooker = {
visasHeld: [
{ country: "UK", validTill: "2024-06-17" }
],
maximumTransitHoursWanted: 15,
maximumStopsWanted: 1,
infants: {
count: 1,
ageOfYoungestInYears: 0
}
};
const Engine = new RuleEngine(rules);
const { results, omitted } = Engine.applyRules<FlightItinerary, FlightBooker>(
flightSearchResults,
flightBooker
)
Finally, a note on performance: despite ensuring that we work majorly with objects which guarantee a constant time efficiency, our rule engine has a time efficiency with a worst case scenario O(n⁴) — you definitely don’t want to use this on very large data sets between a HTTP request-response cycle.
See you next time 👋
