Data Wrangling & ETL on DataBaselines

Trade Data via the Census API and SSIS

Wed, 05 Dec 2018 00:00:00 +0000

This article continues an exploration of the Census Trade data sets, but with an eye for doing larger “bulk-loading” of data the Census makes available. Specifically, we will:

Retrieve export data from the US Census by using their Application Programming Interface (API),
Import the API-generated data file into a database using Microsoft SQL Server Integration Services.

In a previous article, I described a method of generating export data using the Census USA Trade Online website. That method is great for initial discovery and extraction of one-time data. But if you are ingesting Census data on a recurring schedule - you need a process you can repeat easily and consistently. The Census API is useful for this purpose.

Georgia Exports, A Tutorial for USA Trade Online

Thu, 13 Sep 2018 00:00:00 +0000

Trade exports accounted for $37 billion of Georgia’s economy in 2017. That translates to over 180,000 jobs in our state. With so many revisions to tariffs and NAFTA taking place, you may wonder: “What effect will these changes have on our State’s economy?”

Starting with this blog article – I’d like to share what I’ve learned about the subject. Together, we’ll evaluate the most authoritative and detailed source of US import/export data that is available: the US Census Trade data. In this article I’ll show you how to extract data from the Census using USA Trade Online. This is an easy-to-use and powerful method of retrieving trade-related data. In later articles I’ll address:

Improving Data Quality

Wed, 03 Jan 2018 00:00:00 +0000

Our customer address information is inaccurate and it contributes to delayed deliveries and frustrated customers. What general resources are available to improve the quality of our data?

There is a rather elegant solution to the problem of address standardization that is described later in this article. But I’d like to take the opportunity to talk more generally about data quality.

Data quality is one of those quirky things that you notice most when it is not there. Lousy data introduces all manner of inefficiencies and costs within an organization. Yet the time and expense to clean up data can seem overwhelming.