Taking Google’s Services for Granted
Recently I was on boarding a large department into Resgrid which is a cloud service company, SaaS deployed on Microsoft Azure, providing logistics and management tools to first responder organizations like volunteer fire, career fire, EMS, search and rescue, public safety, disaster relief organizations, etc. It was founded in late 2012 by myself and Jason Jarrett (staxmanade).
I was in a Join.me with them for about an hour. As I was going through it I noticed that the BigBoard, which is used to display a snapshot of data usually on a monitor mounted on a wall, wasn’t setting the correct address, so it was defaulting to Carson City (nice feature for a department in another country). So I’m completely baffled why the map on the BigBoard won’t center correctly. I knew it was working at one point, the code tested and validated, but I went along and continued to onboard them, making a note to check this out latter.
Latter in the day I pulled down the latest DB from Azure and fired it up, and logged into the department, pulled up the BigBoard and bam, it’s working. I’m seeing a map not of the dessert of Carson City, but of their location. No data changes, everything is the same as prod, I doubled checked all the data, reviewed all my git commit history to ensure I didn’t change anything between prod’s deployment and master, nothing. So I move along, nothing to see here, and worked on some other issues they reported. I left the BigBoard open on a tab while I worked, a couple of hours later I look at it and it’s back to Carson City. So began the process of stepping through the code to see where in the hell the bug in. After a couple of hours I discovered that the geocoders stopped working.
Currently Resgrid uses 2 geocorders, Bing and Google. Yahoo shut theirs down and I haven’t gotten around to enabling MapQuest or Nokia HERE yet. We use multiple because sometimes one or another falls down on geocoding an address, Yahoo was actually pretty good at UK and Australia addresses compared to Google. I dove into the Google geocoder to see what’s up and saw the request was denied with an OVER_QUOTA error, oh snap. We had been using the free/public version of Google’s geocoder which only gives 2,000 geocoding transitions per day. Every time the geocode was called it would call out to Google. Which means if someone sat on the BigBoard it would generate 1440 geocode requests just but itself, every day!
So began the mad dash, low-carb monster fueled coding binge. I reorderd the geocode code, signed up for an API key on Google so we could track request counts and most importantly enabled our Azure Redis cache to cache the request of a geocode request, both for reverse and forward. The Bing Geocoder returned nothing, so I think they banned that api key, so I signed up for a new one as well.
As a friend Dale would say this was one of the first of many ‘uptown problems’ Resgrid encountered as we grow. But the more I look at the problem it should have never been an ‘uptown’ problem. For so long I’ve been doing the same thing, just relying on Google’s services to work all the time for free. I didn’t bother caching the result incase it was requested again. This was just being a poor netcitizen, this data isn’t in flux, forward and reverse geocoding results are what they are, the latitude and longitude of my house still equals my house and vice versa. This is a clear cut golden case for caching the results.
The things I can no longer take for granted are growing in number every day. Developing a global SaaS app has ruined DateTime handing, TimeZones, EF CodeFirst and services like Geocoding for me. I think I developed a few more gray hairs with just this issue alone and the fact is that it could have been avoided.
