Statistics Estonia Efficiently Integrates over 70 Public Data Sources, Supporting Critical Government Decisions That Shape Economic and Social Policies
With data growing exponentially, manual processing was falling short, so Statistics Estonia called on HULFT.
Since 1921, Statistics Estonia has collected and analysed vital national data, helping to inform economic and social policies that have significant impacts on, and deliver important benefits to, Estonia’s population. Its work is crucial for demographic, social, economic, and environmental development.
Data moved up a gear – but the data gateway didn’t.
For Statistics Estonia’s Chief Data Officer Tauno Tamm, finding a more efficient way to tackle huge volumes of multi-channel data, from many different sources, in many disparate formats, was a key priority.
Every business owner in Estonia, for example, has to submit information relating to their business performance via different Government portals. From there, the Statistics Estonia team processes data from around 70 sources and generates more than 50 discrete statistics. A complex mix of both structured and unstructured data – xml files, csv files, Excel, email, SharePoint, and many others – is the order of the day.
To collect this maelstrom of data, prior to integrating it into a form that Statistics Estonia analysts can work with, the team built a ‘data gateway’ that receives the data to minimize the manual work for loading, cleansing, restructuring, and validation.
This process has necessitated three skilled developers working full-time on individual SQL scripts to transform the disparate data into a resource that can be made accessible to data scientists and statisticians in the data warehouse.
As Tamm observes, the maelstrom is becoming a tsunami, not only as more data are produced, but as requirements shift from recent and retrospective data to real-time. “Much of the data we have typically processed was from a preceding period,” he says, “but what a phenomenon like COVID showed was the criticality of being able to integrate real-time data, which was coming at us with greater speed and velocity – faster than our developers could respond to.”
In short, despite Statistics Estonia’s hundred-year history of meticulously collected and processed data, the manual gateways now in use risked becoming a bottleneck.
A more automated approach was needed.
HULFT Integrate – more and better data output, less coding, same headcount
Statistics Estonia chose HULFT Integrate to enable it to produce, as Tamm puts it, “faster numbers,” and to respond to the growing tide of both retrospective and real-time data entering its gateway.
The key benefit of HULFT Integrate was its ability to reduce the organisation’s dependency on the use of manual SQL scripts – and the developers who code them – by automating and therefore accelerating the validation and restructuring of data in the gateway, before it passes to the central staging area.
Previously, this process relied on multiple disparate scripts and piecemeal manual integrations, and the drawbacks were many.
Developments of this type tend to be point-to-point, meaning that they are slow, and easily broken by even the slightest change not properly replicated across both points. This makes them error-prone, which in turn makes them still more time-consuming and onerous to manage, sapping productivity and forcing up operational costs.
Additionally, such integrations lack both transparency and operational resilience; only a developer can understand their workings, and they are, moreover, typically reliant on the knowledge of the specific developer who built them.
With HULFT Integrate, the process is not only faster, meaning much more data can be processed in the same timeframe, it also decouples integration from development, and requires no separate workflow build – both of which mean developers can be deployed more strategically on other business-critical projects.
But the positive outcomes go well beyond this, delivering benefits the organisation had never previously envisaged. Tamm reports, for example, that HULFT Integrate’s built-in API tools enabled his team to create connections to new data sources, something they could not have considered before, ‘because we didn’t have the programmers for it’.
HULFT Integrate’s intuitive, drag-and-drop interface also minimized the need for the Statistics Estonia team to develop new code. All of these represent productivity gains, enabling Tamm’s team, as he puts it, to “produce more with the same amount of people.”
Why HULFT, how was it, and what’s next?
As you might expect from a project of national importance, Tamm’s team considered a number of potential suppliers, including Alteryx, Apache NiFi, Dataiku, Informatica, Pentaho, and Talend.
What drew the team to HULFT was, as Tamm puts it, “the optimum combination of price, quality, and customer service, delivering the best value overall.”
Speaking of the implementation and the learning curve, he comments that both were “pretty smooth,” whilst acknowledging that the organisation’s particular needs in working with Governmental IP (Intellectual Property) might necessitate additional “special development.”
He also points out that his team needed to adjust its way of thinking away from their previous scripts and rules tables and towards HULFT Integrate’s visual drag-and-drop approach.
As for the future, Tamm is clear that there will be other applications and use cases under consideration. “The pressure to produce more and better data isn’t going to relent,” he says. “This is the data that tells our Government whether the economy is growing or shrinking, what the risk of COVID is, and the consequences this has had and could have in the future.”
About HULFT Integrate
HULFT Integrate is an automated, no-code data integration solution that takes actionable data and delivers it into a business application, intelligence tool, or workflow to achieve better business outcomes. It eliminates error-prone, time-consuming, productivity-sapping manual integration processes.
It is securely compatible with all major data destinations through hundreds of diverse connectors. Whereas most data extraction tools require significant manual interaction such as uploading the file, waiting for results, downloading the data, and then manipulating it to the right format, HULFT Integrate connects that data to any internal or external database directly.
HULFT Integrate is part of a broader solutions portfolio: HULFT Data Capture, which leverages leading OCR technologies to help customers extract and validate critical data; and HULFT Transfer (HULFT8), the most time-tested managed file transfer solution on the market today. Together, HULFT provides a one-stop-shop to take businesses from data disparity to data dominance.