Putting the economists out-of-work. A bit self-defeating perhaps. Most of my career has been about analyzing data, shedding light on hypotheses, and develop compelling data-driven narratives that can yield insight and enable informed leadership action.
This case was about that. At least a sub-set of automating analytics of government financials, resulting in a number of products for different users, i.e Minister of Finance, Prime Minister, technical staff, and a public facing webpage.
The aim was to develop a fully autonomous solution, run at a no-cost infrastructure, to be updated as frequently as daily if needed, very low barrier operator interface, fully human recoverable, and perhaps most challenging of all, achieve a sense of dynamism and compelling storyline.
Executing this type of analytical package ad-hoc, can take a highly skilled team weeks. Even when using the sort of semi-automated internal production systems that good consultant houses will have built.
With the core government team, we explored several concepts, and eventually arrived at resolving three main problems:
Automating the Mechanics
The advantage of working with fiscal analytics is that it is based upon relatively structured data input from the governments accounting system. There is a logic to the structure. Yet, the data comes in different forms, and for different purposes, some intended for internal management consumption, some intended for public disclosure, and the periodicity will naturally change for every update.
So the scripts had to be programmed to recognize the different types of data sources, recognize the periodicity (day, month, year and so forth), and to send them in the right direction for further processing, depending upon what type of product and user.
From there it was a question of building datasets, tabulations and cross-tabulations, to feed the analytical narrative. Endless strings of calculations and formulas, but not all that unfamiliar to those who do spreadsheet programming. Same principles. Just a lot of it.
The Analytical Narrative and Visualization
A simplification in this case is that the macro-narrative of fiscal analysis is somewhat given. There is revenue, and expenditures. There is time. There are departments, geographies and types of cost.
Visualizations is something we are passionate about. But it is difficult to build visualizations that respond to shifting data, time periods, and maintain high quality in response to different data, without requiring any manual human adjustments. Also, we were aiming at quite a portfolio of visualizations and formats for management reports, different languages, and formats for powerpoint, document reports and for web.
Web-page data-visualization is a surprisingly undeveloped area of technology. After some exploration of commercial packages, we found them to rigid and confined, and developed this from scratch using raw javascript and open source libraries. Real fun to work with. Ever grateful to the open source communities. The end result is visible to all. Check it out. Quite dynamic and engaging!
Cognitive capabilities and meaningful observations
Analytics need a good narrative to be useful and compelling.
Humans do that well. Robots do not.
Sticking to our fully autonomous concept, we aimed at developing comments, headings and observations responsive to actual changes in the data. Basically, having the machine recognize whether there is a trend, ranks, clusters, majorities, and outliers. And then develop a text string describing such phenomenon as they are found in the data.
This goes some way in the machine interpreting meaning and significance of that pattern. Sounds a bit like AI, but that would be an exaggeration. But certainly, involved complicated logics of IF ELSE statements and dynamically adjusting the parameters of key variables. Also explored actually changing the storyline and presentation outline in response to findings.
What amazed us is how much is actually possible using conventional tools and scripts but in slightly unconventional ways. I think much more is possible here. Especially when dealing with ex-ante familiar entities and structures of data. If dealing with truly unknown data, one would need real AI to automate analytics. And no-one has really succeeded in that quite yet.
And beyond those issues, it was also a bit of fun to build this on entirely no-cost consumer grade free infrastructure served in the cloud. Its unbelievable what is possible these days, whether you are in Mogadishu, Bangalore or San Jose.
The government now has a very powerful tool at their disposal. And they have already put it to good use. The system can perform repetitive, yet complex analysis, repetitively and extremely fast. And the team at the Ministry of Finance can spend their time on many, and much more complicated tasks, that only humans can do.
