Dark data discovery and solar eclipses

Home Blog Dark data discovery and solar eclipses
Solar eclipse

The recent “Great American Eclipse” is a good metaphor for the “dark data” being generated and stored in federal agencies today. To view an eclipse, you need eclipse glasses—without them, you not only endanger your eyes, but you can’t see the moon traveling across the sun because the sun’s glare is too bright.

Likewise, for data—without the right use of technology, procedures and processes, not only is the risk of a security breach greater, but the value of the data is greatly diminished because it’s not easily accessible or usable.

Research and advisory company Gartner, Inc. defines dark data as:

“The information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes.”

Gartner turns to a different science—physics—for another metaphor, saying that just as the universe is comprised mostly of dark matter, an organization’s universe of information is greatly comprised of dark data—data retained for compliance, or just because it can be so easily retained.

Dealing with the growth of dark data

We all know about the challenge of growing data volume in federal agencies. 80% of respondents in a recent GovLoop survey said that “data is growing exponentially at their organization” which of course is no surprise.

Our interest is in the reasons why this data goes dark in the first place:

  • 49% — simply lack time to address data issues strategically.
  • 27% — users treat storage systems as a “data dumping ground.”
  • 10% — IT budgets and strategies are based on data volume, not value.
  • 9% — automated applications generate data that’s not removed when no longer needed.

Unfortunately, these issues show up throughout the federal and private industry sectors. Part of our expertise is in making data more accessible to all users and discoverable when needed. When these concerns become significant enough, service providers like LAC Federal are brought in for support.

Dealing with data storage obsolescence

old printers

Another dark data source that continues to linger in some federal government departments and agencies is technology obsolescence. Old storage media holds information that may be useful and valuable, but what do you do if the technology needed to read the media isn’t available in good working order?

Managing obsolescence has become a librarianship requirement, and as the pace of new technology accelerates, the need to keep up becomes a greater priority. Moreover, unlike the commercial sector, the government sector is rarely able to stay on the cutting edge. It’s a wake-up call to monitor old data, identify what’s worth keeping and get that information transferred to modern systems now, while the old systems you need to accomplish that are still working.

Why librarians get called in for dark data discovery

The move to digital information is building a bridge between the disciplines of LIS (Library Information Science) and IT (Information Technology) for natural reasons. Librarians have always been dedicated to the sorting and classifying of information to make it easily discoverable and accessible. That information used to be encoded on paper; now it’s mostly digitally encoded. Even librarians who continue to work in traditional library settings are becoming digital information experts.

For example, LAC Federal librarians, working with NLE (National Library of Education) leadership, developed a system of dedicated state web portals to empower NLE users to easily view pending legislation, local news and other state-level educational information, which was too time-consuming to search on a case-by-case basis.

Increasingly our work is devoted to digital information discovery and access. Librarians are creating metadata, curating information to help determine data lifecycles while adding additional value—all with the purpose of bringing dark data into the light.

Audrey McKay

Audrey McKay

Deputy Director, Federal Services at LAC Federal.
Audrey McKay

Latest posts by Audrey McKay (see all)

Related posts

Data visualization trends and tools

Anselm Spoerri is a data visualization expert who has been conducting research and teaching information visualization for the last 20…

Read more
Data management for federal agencies

The need to tame and manage the proliferation of data will define 21st century federal government libraries. Harnessing the information…

Read more