We recently released an API for our Email Validation platform. API's are all the rage these days, so I thought I would stop and explain a little of the how and why of our API.
Why make an API?
Our software revolves around people giving us lists of email addresses, checking those email addresses to see if they are deliverable, and then getting the results back to our users.
Our website DataValidation.com does a great job of doing this, but sometimes you'd like to do things in a more automated fashion. That's where this API comes in: Users big or small can now use our API to script the process of uploading and analyzing email addresses. This allows the information to be transferred directly between our systems and the customer's system with no human intervention.
The first start
When we decided to make this new API, we quickly identified necessary endpoints based on how we expected people to use our system. We also quickly realized that having a "Cookbook" that showed details of how to use our system would probably be very helpful when integrating with new partners.
We started building out the new API pretty quickly. As the development process rolled along we began to understand that some of our assumptions about how this API would fit into our architecture were not ideal. Coincidentally, around this time we also sent 5 members of our team to the API Strategy conference that was being put on by our API Management Vendor 3Scale.
The conference was a lot of fun and very educational for the whole team. We had developer and marketing folks attend this, and everyone learned lots of interesting things that we were able to roll into our API. This influx of ideas (and hearing from others who had done similar things) lead us to stop and rethink the design of our system and how we were building it.
The second start
One major lesson we learned the hard way (and this was also brought up many times at the API conference) was the importance of documentation. In software development documentation is usually given a pretty low priority, but in an API it’s actually very important: Your documentation is your contract with the outside world about how to communicate with your systems.
This contract can also be used to ensure that all team members understand what is being built, and how it must function. To build our documentation we used a format called "Swagger", which turned out to be a very powerful choice for us. Not only is it a semi-standard that is well understood in the API space, we were also able to use it to help verify the correctness of our API as we built it!
Our development stack loads our swagger docs on startup and confirms that the Swagger documentation is syntactically correct, and then as data flows through our system it ensures that the right fields and data types are present. We favor the Python programming language for our backend work, and it is a dynamically typed language where things can get a little lax when it comes to ensuring type safety. The Swagger validation checks now act as a sort of a type-checking system to ensure we don't return lists where we should return strings (and other simple programming mistakes).
The other major revelation we had was that other parts of our internal infrastructure could be turned into API's. This would allow more flexibility with our code by lessening some dependencies while allowing us to seamlessly scale systems to handle larger data volumes. This API-ification of our services turned out to have other benefits, such as cutting down on code duplication and allowing for better testing due to separation-of-concerns that happened as we broke the code apart.
Checks and balances
While development was progressing on the API, our marketing team was able to recruit a friendly customer who was quite interested in the capabilities offered by our new API. Conversations with this new partner helped steer the API development and also served as the perfect test of our new service. When the API went live, our (very patient) partner was able to give us invaluable feedback that ranged from code fixes to new recipes for our Cookbook.
The end result was that by the end of our testing and evaluation period the partner was able to load large volumes of data into and out of our system without any "real-time" assistance from our team. We consider this a major milestone because this means we have a system (both our API and the Cookbook) that is usable and useful!
As the partner was testing and evaluating our system we wrote a series of automated tests that would exercise the live system as well as our development systems. These tests helped establish confidence that any changes we were making to the system would not negatively impact our current system. This, combined with our Swagger validation, lead to the API solidifying very quickly and almost no regression bugs as we rolled out code updates.
Another valuable step we took was involving a 3rd developer to write our documentation and an additional set of tests. Ryan had not contributed to the design of the API nor did he attend the API Conference, so he was able to bring a truly fresh set of eyes to the process. Along the way, he found several bugs that had simply been overlooked due to everyone else's familiarity with the system. His efforts helped put the final bit of polish on the system to truly make it shine.
We started this API project in late August, had our first partner using it by October 15th, and had it ready for public signups by early November. Along the way there were several lessons learned that we will be incorporating into the next API we build:
- Write your documentation first, and preferably in Swagger. That contract will help keep your project on track.
- Writing a Cookbook can be challenging, but doing this early on can help clarify any misconceptions the team has about the API. It also is a great touchstone to show everyone what the end goal is.
- Enable Swagger validation in your code as soon as you can.
- The better the error messages, the happier everyone will be. Verbosity isn't a requirement, but explaining that an integer was seen when you expected a boolean helps your developers and partners immensely.
- Test early, test often. Unit tests are great for checking your code, but a full end-to-end integration test is invaluable. We have setup our Continuous Integration server with these tests so it can monitor our production servers and ensure that it is up and running (and that our new code doesn't break anything).
In addition to our normal Python development tools, we also used Postman, cURL, and swagger-ui for testing and debugging the API. These tools were very helpful for finding out what went wrong as we developed the system. We use 3Scale for our API provisioning, management, and monitoring (major thanks to Amelia and Yossi at 3Scale for all of their help!). I highly recommend using an API management service like 3Scale. They handle all of the grunt work around the API and allow us to focus on making sure the API functionality served our clients’ needs.