Oct 4, 2022
|
1h 21min
|
3.2k views
Model Monitors and Alerting at Scale with RStudio Connect | Adam Austin, Socure
Deploying a predictive model signals the end of one long journey and the start of another. Monitoring model performance is crucial for ensuring the long-term quality of your work.
A monitor should provide insights into model inputs and output scores, and should send alerts when something goes wrong. However, as the number of deployed models increases or your customer base grows, the maintenance of your monitor portfolio becomes daunting. In this talk we’ll explore a solution for orchestrating monitor deployment and maintenance using RStudio Connect. I will show how applications of R Markdown, Shiny, and Plumber can unburden data scientists of time-consuming report upkeep by empowering end-users to deploy, update, and own their monitors.
Timestamps: 2:01 - Start of presentation 3:43 - About Socure 6:12 - Model performance matters, deployment isn’t the end of the story 7:26 - What is a monitor? 8:55 - What is an alert? 11:00 - Monitor example 18:09 - Firing an alert from RStudio Connect 19:00 - Why monitor from RStudio Connect? 24:33 - How monitoring drives success at Socure 30:00 - Git-backed deployment in RStudio Connect 36:00 - Shiny app that their account managers see 46:00 - Architecture of a monitoring system 56:00 - Connect hot tip System-wide packages 57:00 - Why did we try Connect for monitoring 59:02 - Why do we keep using it for that:)
Resources shared: Blastula package: https://github.com/rstudio/blastula connectapi package: https://github.com/rstudio/connectapi rsconnect package: https://rstudio.github.io/rsconnect/ Intro to APIs blog post: https://www.rstudio.com/blog/creating-apis-for-data-science-with-plumber/
Speaker Bio: Adam Austin is a senior data scientist and RStudio administrator at Socure, a leading provider of identity verification and fraud prevention services. His work focuses on data science enablement through tools, automation, and reporting
blastula
connectapi
plumber
rsconnect
rstudio