Advocate API – Slack Engineering

Slack, as a product, presents many alternatives for suggestion, the place we are able to make solutions to simplify the consumer expertise and make it extra pleasant. Each looks as if a terrific use case for machine studying, however it isn’t sensible for us to create a bespoke answer for every.

As an alternative, we developed a unified framework we name the Advocate API, which permits us to shortly bootstrap new suggestion use instances behind an API which is definitely accessible to engineers at Slack. Behind the scenes, these recommenders reuse a standard set of infrastructure for each a part of the advice engine, reminiscent of information processing, mannequin coaching, candidate era, and monitoring. This has allowed us to ship a variety of completely different suggestion fashions throughout the product, driving improved buyer expertise in a wide range of contexts.

Extra than simply ML fashions

We purpose to deploy and keep ML fashions in manufacturing reliably and effectively, which is termed MLOps. This represents nearly all of our staff’s work, whereas mannequin coaching is a comparatively small piece within the puzzle. In case you take a look at Matt Turck’s 2021 review speaking in regards to the ML and information panorama, you’ll see for every part of MLOps, there are extra instruments than you may select from available on the market, which is a sign that business requirements are nonetheless creating within the space. As a matter of truth, stories present a majority (as much as 88%) of company AI initiatives are struggling to maneuver past check phases. Corporations reminiscent of Facebook, Netflix, Uber often implement their very own in-house methods, which is analogous right here in Slack.

Luckily, most frequently we don’t have to hassle with choosing the proper instrument, because of Slack’s well-maintained information warehouse ecosystem that permits us to:

• Schedule varied duties to course of sequentially in Airflow
• Ingest and course of information from the database, in addition to logs from servers, shoppers, and job queues
• Question information and create dashboards to visualise and observe information
• Lookup computed information from the Function Retailer with major keys
• Run experiments to facilitate characteristic launching with the A/B testing framework

Different groups in Slack, reminiscent of Cloud Providers, Cloud Foundations, and Monitoring, have offered us with extra infrastructure and tooling which can also be crucial to construct the Advocate API.

The place is ML utilized in Slack?

The ML companies staff at Slack companions with different groups throughout the product to ship impactful and pleasant product modifications, wherever incorporating machine studying is smart. What this appears like in observe is a variety of completely different quality-of-life enhancements, the place machine studying smoothes tough edges of the product, simplifying consumer workflows and expertise. Listed below are a number of the areas the place you’ll find this form of suggestion expertise within the product:

An essential results of the Advocate API — even apart from the present use instances that may be discovered within the product — are the close to equal variety of use instances we’re at present testing internally, or now we have tried and deserted. With easy instruments to bootstrap new recommenders, we’ve empowered product groups to observe a core product design precept at Slack of “prototyping the trail”, testing and discovering the place machine studying is smart in our product.

Machine studying, when constructed up from nothing, can require heavy funding and could be fairly hit-or-miss, so beforehand we averted making an attempt out many use instances which may have made sense merely out of a concern of failure. Now, we’ve seen a proliferation of ML prototypes by eradicating that up-front value, and are netting out extra use instances for machine studying and suggestion from it.

Unified ML workflow throughout product

With such a wide range of use instances of advice fashions, now we have to be deliberate about how we manage and take into consideration the varied parts. At a excessive degree, recommenders are categorized based on “corpus”, after which “supply”. A corpus is a kind of entity — e.g. a Slack channel or consumer — and a supply represents a selected a part of the Slack product. A corpus can correspond to a number of sources — e.g. Slackbot channel solutions and Channel browser suggestion every correspond to a definite supply, however the identical corpus channel.

No matter corpus and use case although, the therapy of every suggestion request is fairly comparable. At a excessive degree:

• Our fundamental backend serves the request, taking in a question, corpus, and supply and returning a set of suggestions that we additionally log.
• When these outcomes are interacted with in our shopper’s frontend, we log these interactions.
• Offline, in our information warehouse (Airflow), we mix these logs into coaching information to coach new fashions, that are subsequently served to our backend as a part of returning suggestions.

Here’s what that workflow appears like in whole:

Backend

Every supply is related to a “recommender” the place we implement a sequence of steps to generate an inventory of suggestions, which:

• Fetch related candidates from varied sources, together with our embeddings service the place comparable entities have shut vector representations
• Filter candidates based mostly on relevancy, possession, or visibility, e.g. personal channels
• Increase options reminiscent of entities’ attributes and actions utilizing our Function Retailer
• Rating and type the candidates with the predictions generated by corresponding ML fashions
• Rerank candidates based mostly on extra guidelines

Every of those steps is constructed as a standardized class which is reusable between recommenders, and every of those recommenders is in flip constructed as a sequence of those steps. Whereas use instances may require bespoke new parts for these steps, typically creating a brand new recommender from current parts is so simple as writing one thing like this:

last class RecommenderChannel extends Recommender 
   public perform __construct() 
	   mum or dad::__construct(
		   /* fetchers */ vec[new RecommendChannelFetcher()],
		   /* filters  */ vec[new RecommendChannelFilterPrivate()],
		   /* mannequin    */ new RecommendLinearModel(
			   RecommendHandTunedModels::CHANNEL,
			   /** additional options to extract **/
			   RecommendFeatureExtractor::ALL_CHANNEL_FEATURES,
		   ),
		   /* reranker */ vec[new RecommendChannelReranker()],
	   );
   

Knowledge processing pipelines

Apart from having the ability to serve suggestions based mostly on these feedback, our base recommender additionally handles important logging, reminiscent of monitoring the preliminary request that was made to the API, the outcomes returned from it, and the options our machine studying mannequin used at scoring time. We then output the outcomes by means of the Advocate API to the frontend the place consumer responses, reminiscent of clicks, are additionally logged.

With that, we schedule Airflow duties to hitch logs from backend (server) offering options, and frontend (shopper) offering responses to generate the coaching information for machine studying.

Mannequin coaching pipelines

Fashions are then scheduled to be educated in Airflow by working Kubernetes Jobs and served on Kubernetes Clusters. With that we rating the candidates and full the cycle, thereafter beginning a brand new cycle of logging, coaching, and serving once more.

For every supply we frequently experiment with varied fashions, reminiscent of Logistic Regression and XGBoost. We set issues up to verify it’s simple so as to add and productionize new fashions. Within the following, you may see the six fashions in whole we’re experimenting with for people-browser because the supply and the quantity of Python code wanted to coach the XGBoost rating mannequin.

ModelArtifact(
	identify="people_browser_v0_xgbr",
	mannequin=RecommendationRankingModel(
		pipeline=create_recommendation_pipeline(
			XGBRanker(
				**
					"goal": "rank:map",
					"n_estimators": 500,
				
			)
		),
		input_config=RecommenderInputConfig(
			supply="people-browser",
			corpus=Corpus.USER,
			feature_specification=UserFeatures.get_base_features(),
		),
	),
)

Monitoring

We additionally output metrics in several parts in order that we are able to get an total image on how the fashions are performing. When a brand new mannequin is productionized, the metrics can be mechanically up to date to trace its efficiency.

• Reliability metrics: Prometheus metrics from the backend to trace the variety of requests and errors
• Effectivity metrics: Prometheus metrics from the mannequin serving service, reminiscent of throughput and latency, to verify we’re responding quick sufficient to all of the requests
• On-line metrics: enterprise metrics which we share with exterior stakeholders. Some most essential metrics we observe are the clickthrough fee (CTR), and rating metrics reminiscent of discounted cumulative gain (DCG). On-line metrics are ceaselessly checked and monitored to verify the mannequin, plus the general end-to-end course of, is working correctly in manufacturing
• Offline metrics: metrics to match varied fashions throughout coaching time and determine which one we doubtlessly wish to experiment and productionize. We put aside the validation information, aside from the coaching information, in order that we all know the mannequin can carry out properly on information it hasn’t seen but. We observe widespread classification and rating metrics for each coaching and validation information
• Function stats: metrics to observe characteristic distribution and have significance, upon which we run anomaly detection to forestall distribution shift

Iteration and experimentation

So as to practice a mannequin, we want information, each options and responses. Most frequently, our work will goal lively Slack customers so we often have options to work with. Nonetheless, with out the mannequin, we received’t have the ability to generate suggestions for customers to work together so as to get the responses. That is one variant of the cold start drawback which is prevalent in constructing suggestion engines, and it’s the place our hand-tuned mannequin comes into play.

Through the first iteration we’ll typically depend on a hand-tuned mannequin which is predicated on widespread data and easy heuristics, e.g. for send-time optimization, we usually tend to ship invite reminders when the staff or inviter is extra lively. On the identical time, we brainstorm related options and start extracting from the Function Retailer and logging them. It will give us the primary batch of coaching information to iteratively enhance upon.

We depend on intensive A/B testings to verify ML fashions are doing their job to enhance the advice high quality. Every time we swap from hand-tuned to mannequin based mostly suggestions, or experiment with completely different units of options or extra difficult fashions, we run experiments and ensure the change is boosting the important thing enterprise metrics. We’ll typically be taking a look at metrics reminiscent of CTR, profitable groups, or different metrics associated to particular components of Slack.

Following is an inventory of current wins we’ve made to the aforementioned ML powered options, measured in CTR.

• Composer DMs: +38.86% when migrating from hand-tuned mannequin to logistic regression and extra just lately +5.70% with XGBoost classification mannequin and extra characteristic set
• Creator invite movement: +15.31% when migrating from hand-tuned mannequin to logistic regression
• Slackbot channel solutions: +123.57% for depart and +31.92% for archive solutions when migrating from hand-tuned mannequin to XGBoost classification mannequin
• Channel browser suggestion: +14.76% when migrating from hand-tuned mannequin to XGBoost classification mannequin. Beneath we are able to see the impression of the channel browser experiment over the time:

Closing ideas

The Advocate API has been used to serve ML fashions during the last couple of years, although it took for much longer to construct the groundwork of varied companies backing up the infrastructure. The unified strategy of Advocate API makes it doable to quickly prototype and productionize ML fashions throughout the product. In the meantime, we’re continually improving:

• Knowledge logging and preprocessing course of, so that may be prolonged to extra use instances
• Mannequin coaching infrastructure, e.g. scaling, {hardware} acceleration, and debuggability
• Mannequin explainability and mannequin introspection tooling using SHAP

We’re additionally reaching out to numerous groups inside the Slack group for extra alternatives to collaborate on new components of the product that could possibly be improved with ML.

Acknowledgments

We needed to provide a shout out to all of the those who have contributed to this journey: Fiona Condon, Xander Johnson, Kyle Jablon

Eager about taking up attention-grabbing initiatives, making folks’s work lives simpler, or simply constructing some fairly cool kinds? We’re hiring! 💼 Apply now