Ever since we attended PyCon this year, we’ve been hard at work rebuilding our python agent. We took the feedback we received from the event, compiled it with the ongoing input from our beta users and set out to redefine how we manage APM for Python. The result is a more than ten times more efficient agent. To accomplish these gains, we changed how we perform performance analysis, ship data, and serialize packets.
Python Performance Analysis
The most significant amount of feedback we received was that our agent created an unacceptable amount of overhead for almost every user’s app. This is why reducing the impact of our performance monitoring was essential to this release. We had fun and learned a lot doing the necessary refactor of function monitoring and data transmission. In the end, we were able to maintain the accuracy of our performance issue identification while obtaining the 10x improvement we were striving to achieve. Although this was a significant improvement, we continue to work towards having zero impact on customer applications.
Shipping Data From Python with MQTT
We’ve been planning to move to MQTT for a while and took this opportunity to take the plunge. MQTT provides all the functionality we need to transmit data securely and aligns with our roadmap almost perfectly. MQTT allows us to continue to capture unhandled exceptions and poorly performing functions while devices are not remotely accessible. This offline support means, as with our first implementation, we never have to miss a problem.
MQTT also enables us to provide better device identity management. With this new identity management we’re now able to identify the following problems:
- Troublesome devices whose application is performing worse than the rest of the fleet
- Regional & environment based performance issues
- Function performance anomaly detection relating to multiple devices
In our first iteration of the python agent, we utilized JSON for serialization because of its integrated python support. This out of the box support enabled customers to refrain from adding more third-party packages than they already required. However, over the course of our beta, we chatted with our customers and found that another third party package wasn’t a big deal. Knowing that JSON wasn’t exactly size or performance friendly, and having this input drove us to make the shift.
Based on most of our user’s input we settled on investigating Protocol Buffer and MessagePack. Due to Protocol Buffer’s support from Google, having the most requests, and being slightly more performant we started down the path of integrating it. Our first red flag was the comment on the README that stated “[Python Protobuff] may be more buggy, and it is known to be pretty slow at this time.” We had heard great things about it, so we didn’t want to discount it without at least giving it a try. Unfortunately, after a week of getting it integrated some nasty benchmarks, we decided we should transition over to MessagePack due to these exact issues.
After moving to MessagePack, we couldn’t be happier. MessagePack serialization on average produces a packet that is 33% smaller than the JSON equivalent. This data reduction has been a massive win for our beta users. Paired with some additional modifications to our backend we’ve been able to drastically reduce the amount of data our agents produce. All of this comes with the added benefit of a reduction in the serialization overhead itself. Which in turn reduces the impact Auklet has on customer’s applications.
Automotive Safety Integrity Level B Compliance
Aside from making improvements to the python agent’s performance we also improved the overall test coverage. We undertook this effort to adhere to our internal quality standards, reduce the potential of our agent introducing problems into applications, and to meet the demands of our Automotive customers. With these improvements, we are proud to announce our 0.6.0 release meets all ASIL B requirements. We will maintain this level of integrity throughout all future releases of the Python agent. If there are other compliances you need to meet to use an APM tool, please reach out.
1.0 Release Coming
With these reductions in overhead, data transmission, and issues we’re finally ready to draw our beta to a close. It has been a fantastic ride, and we cannot thank our beta users enough. Everyone has been integral to helping us pinpoint bugs and areas for improvements. We’ve received amazing input on what helps developers solve problems before their customers find them and couldn’t be more thankful. We’ll be releasing version 1.0 of our python agent before the end of September. This transition does not affect our existing beta users or any users that sign up in the interim. It also has no impact on our engagement with our customers. Please keep the feedback coming. We always love hearing your thoughts!