About the company
Overview
Payoneer uses our solution to ingest and centralize logs for over 1,000 services running in production, and leverages the technology to aggregate and convert the logs to metric data which can easily be monitored for anomalous behavior.
Together with Anodot and PagerDuty, the team has a complete view of their production systems, fewer false positives for production errors, and a more streamlined troubleshooting process.
The Challenge
Before switching to our platform, Payoneer didn’t have a logging solution that could support their scaling requirements. As their systems grew, they needed a more powerful approach that would combine log data from hundreds of services and allow them to perform more dynamic calls to the system.
“When we first started to build our monitoring capabilities, we were looking at text in the logs. We were looking for specific buzzwords in certain logs, such as ‘Exception in abc’ or ‘Errors in xyz’ and so on. It was pretty cumbersome. And then we thought, how can we do better?”
The team was also relying heavily on static thresholds which were used to trigger incidents via PagerDuty. Using this strategy, the team was overwhelmed with false positives which were unavoidable due to the dynamic nature of their services and user behavior.
The Solution
With our innovation, Payoneer’s systems grew from hundreds to more than a thousand services. As the log data is ingested, the individual logs are analyzed and grouped into templates which can be used to generate metrics such as error volumes, rates, and more.
“Instead of looking at hundreds of logs, they help us to focus on the log itself that creates the noise.”
Once the data is ingested and analyzed, Payoneer sends the resulting metric data to Anodot for autonomous anomaly detection and a deeper understanding of business impact.
How the Platform Integrates to the Daily Workflow
The platform is ingesting and analyzing all of Payoneer’s log data. Using the Loggregation and Logs2Metrics features, Payoneer can pinpoint noisy logs that need to be investigated by producing metrics around them.
These metrics are then sent to Anodot, the autonomous business monitoring tool, which is then able to run their ML model and add business insights. When Anodot’s system detects anomalous behavior in the metrics, a PagerDuty alert is triggered. With this combination of tools in place, Payoneer has achieved a near-zero false positive rate.
Within the alert, there’s an option to click directly into the platform to see and investigate the errors. This also enables issues to be easily forwarded to relevant members of the team.
Overall, this solution has not only supported massive growth, it’s also improved quality of life for engineers at Payoneer who are no longer being bombarded by alerts and have a faster and easier way to investigate them.
“It also helped the developers to be more confident when a new thing is being released to production, because they know that there is something there monitoring its behavior. They trust it and they know that they’re being covered.”
“Instead of looking at hundreds of logs, the platform helps us to focus on the log itself that creates the noise.”
Yuval Molnar – Sr. Director of Production Services at Payoneer, Inc.
Summary
The team is using the platform to collect and aggregate all log data from more than 1,000 production services for more advanced anomaly detection. By reducing log data to patterned templates and metric data, Payoneer can more accurately track anomalous behavior in their applications.
More to explore
Puma
AGS
