Written by Hitesh Chitalia, VP of Technology at Itemize
As the VP of Technology at Itemize, decisions I make can greatly impact our ability to deliver product features. Itemize provides payment documents (receipts, hotel folios) data extraction services for clients large and small. We also have an expense management offering. While our product offering is straight-forward, the products are feature-rich. The core Itemize technology is a complex engine utilizing proprietary OCR and machine learning to extract relevant data from financial transaction documents. In this post, I will talk about the three infrastructure principles that help us maintain our service levels, reduce outages, reduce costs, and most importantly, keep the CEO smiling.
Principle 1: Stability at Scale
At Itemize, we support the processing of tens of thousands of receipt documents per month. In many instances, our clients upload in batches, causing spikes in load. In order for us to accommodate (and sleep better at night), we have chosen to host all processing “in the cloud,” specifically, Amazon Web Services (AWS). I am not promoting one cloud service provider over another, however, I do advocate using cloud services to support unpredictable workloads.
When I first started at Itemize, the cloud infrastructure was in pretty good shape, but we had instances where servers did not scale properly or at the right time. Our processing queues would back up, and we lived with it, knowing that the servers would scale eventually. This unfortunately is not satisfactory for operations and support, and definitely not for customers. We needed to be smarter about scaling.
Over the course of a few weeks, we determined the times of day that spikes happened and at what load we need to scale machines to support the workloads. We also needed to scale down at the right time. After gathering all the metrics around queue backups, outages, scaling actions, and reviewing AWS best practices, we modified our scaling policies to support the data collected. The results were clear and we now rarely observe queue backups. Not resting on success, we monitor scaling and workload spikes and adjust as we see changes in usage.
Principle 2: Cost Efficiency
One area I have paid close attention to is our AWS cost. While cloud services have cost benefits if used properly, costs can spiral out of control if not managed tightly.
The three areas we have focused on to help lower our overall costs are
- Right-sizing instance types
- Balancing reserved and spot instances
- Going serverless where appropriate.
We have moved our static and, in some cases, dynamic sites (built on ReactJS) to S3 and Cloudfront. For me, this was a big step. I no longer had to maintain machines for static sites, nor worry about scaling them. We have also made use of Lambda and API Gateway for those processes that are batch and schedule oriented. Finally, we scale with spot instances while keeping a set of reserved “based” instances. Over the course of the year, we have reduced our AWS expenses by 30%. As we grow over the course of 2018, our infrastructure will grow to match the demand while maintaining the gains in cost efficiency.
Principle 3: Reducing Complexity
Cloud service providers offer many tools and can overwhelm quickly. My approach at Itemize has been to keep things simple. The more tools I use, the more things grow in complexity. A couple of questions I ask myself when investigating a new offering are 1) Does the development team have the necessary skill set to build and support on this new offering and 2) Does it help reduce the cost, complexity, and time to delivery? We have not used containers or docker at Itemize, yet. However, we have pushed forward serverless and microservice architectures. We have also taken infrastructure “out of the equation” for the development team so they can focus on code and easily deploy to staging and production utilizing Code Pipeline. What used to take 2-4 hours to manually deploy one of our services now takes minutes.
These are not the only technology principles at Itemize, but they are definitely in focus at the moment. As we grow and move further into 2018, another principle will be center-stage, reducing the focus of one of these three (maintenance mode). Business is not static and neither should our approach to IT be static. I am very proud of the team here at Itemize. The development and product teams are working closely together and discussing more features to add to our product offerings and the development team is delivering more predictably.
If you would like to learn more about Itemize and try out our API, request a trial.
