Dark data discovery and solar eclipses

The recent “Great American Eclipse” is a good metaphor for the “dark data” being generated and stored in federal agencies today. To view an eclipse, you need eclipse glasses—without them, you not only endanger your eyes, but you can’t see the moon traveling across the sun because the sun’s glare is too bright.

Likewise, for data—without the right use of technology, procedures and processes, not only is the risk of a security breach greater, but the value of the data is greatly diminished because it’s not easily accessible or usable.

Research and advisory company Gartner, Inc. defines dark data as:

“The information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes.”

Gartner turns to a different science—physics—for another metaphor, saying that just as the universe is comprised mostly of dark matter, an organization’s universe of information is greatly comprised of dark data—data retained for compliance, or just because it can be so easily retained.

Dealing with the growth of dark data

We all know about the challenge of growing data volume in federal agencies. 80% of respondents in a recent GovLoop survey said that “data is growing exponentially at their organization” which of course is no surprise.

Our interest is in the reasons why this data goes dark in the first place:

• 49% — simply lack time to address data issues strategically.
• 27% — users treat storage systems as a “data dumping ground.”
• 10% — IT budgets and strategies are based on data volume, not value.
• 9% — automated applications generate data that’s not removed when no longer needed.

Unfortunately, these issues show up throughout the federal and private industry sectors. Part of our expertise is in making data more accessible to all users and discoverable when needed. When these concerns become significant enough, service providers like LAC Federal are brought in for support.

Dealing with data storage obsolescence

Another dark data source that continues to linger in some federal government departments and agencies is technology obsolescence. Old storage media holds information that may be useful and valuable, but what do you do if the technology needed to read the media isn’t available in good working order?

Managing obsolescence has become a librarianship requirement, and as the pace of new technology accelerates, the need to keep up becomes a greater priority. Moreover, unlike the commercial sector, the government sector is rarely able to stay on the cutting edge. It’s a wake-up call to monitor old data, identify what’s worth keeping and get that information transferred to modern systems now, while the old systems you need to accomplish that are still working.

Why librarians get called in for dark data discovery

The move to digital information is building a bridge between the disciplines of LIS (Library Information Science) and IT (Information Technology) for natural reasons. Librarians have always been dedicated to the sorting and classifying of information to make it easily discoverable and accessible. That information used to be encoded on paper; now it’s mostly digitally encoded. Even librarians who continue to work in traditional library settings are becoming digital information experts.

For example, LAC Federal librarians, working with NLE (National Library of Education) leadership, developed a system of dedicated state web portals to empower NLE users to easily view pending legislation, local news and other state-level educational information, which was too time-consuming to search on a case-by-case basis.

Increasingly our work is devoted to digital information discovery and access. Librarians are creating metadata, curating information to help determine data lifecycles while adding additional value—all with the purpose of bringing dark data into the light.