The basics of enterprise archival systems

Some of your most valuable enterprise data might be sitting around in file cabinets, old hard drives, and legacy devices—but only if it can be found in the first place.

Every enterprise worth their salt keeps backups, and they might superficially resemble archives. But the two forms of storage “are definitely not the same and cannot really serve as one another,” Denis Leconte, VP of technology at archival service Iron Mountain, wrote to IT Brew.

Backups are primarily designed to protect against accidental or malicious data deletion and ensuing disruption in the course of everyday business, Leconte wrote, while “archival timescales are much longer—decades, definitely multi-generational, and this is when the usual technologies run out of longevity.” He added they’re also packaged and curated for individuals who may have “no connection with the technical and cultural norms of the time when the archive was created.”

Unfortunately, businesses often aren’t clear on why or how they should properly curate data for long-term preservation, experts told IT Brew.

Spring cleaning

Leconte wrote that while commonplace data storage solutions are “incredibly well-suited” to business time horizons of up to three to five years, data retained longer than that needs special attention as “pretty much all modern information storage technologies, analog or digital, fail eventually.”

To address the long-term preservation challenges Leconte presented, one of the standard deliverables offered by digital archival services is a digital assets management (DAM) system, which provides a categorizable and queryable database for managing content over long timescales. Depending on client needs, DAMs can be anything from an on-prem storage bank to a cloud-hosted interface.

“Definitely things are getting much more advanced,” Shana Scott, an archives technician at Anderson Archival, told IT Brew. “Most of the actual technology that archives use are all open source, and even some of the DAMs are based on that same open source. They just put them together in a much easier to use package.”

Getting from point A to point B is often a messy process, though. Ryan Donegan, SVP of operations at Illinois-based archival services agency Heritage Werks, warned “that a lot of times when you have a corporate archive, the word corporate goes before archive.”

Donegan said outsourcing archives to a managed services provider has several advantages, including assisting clients sort through volumes of unstructured data.

“That’s usually the time-consuming part,” Scott said, even with the help of special software like AI-powered search or duplication checkers. “Unless you have a very expensive DAM that can do this auto tagging, and it can do it well, the best way to get searchable data is to have all that metadata, and that usually takes time.”

“We have a kind of adage,” Donegan said. “A company comes to us and says, ‘We’ve got 50 terabytes of material.’ The reality is, you’re only going to use about one terabyte of that throughout the entire lifetime of one of your employees.”

Often the DAM is like the tip of an iceberg, Donegan said: visible and readily accessible, but only containing data that is used often. Progressively larger amounts of data often end up in nearline storage (available on short notice) or relegated to cold storage.

Value add

In Donegan’s experience, clients have often left valuable data—old marketing campaigns, WIP projects, sales records, and historical corporate materials, to name a few—to gather dust. They also tend to relegate tasks like data categorization to interns.

That’s the opposite of the ideal, Donegan said, as the best time to organize data is shortly after it’s created.

“Sometimes [clients] are shocked by the values,” Donegan said. “Until they can kind of get that, it’s almost sometimes just looked at as junk, stuff that takes up space.”

For example, Heritage Werks’s clients have included professional sports teams. Donegan pointed to situations like a player’s death or admission to a hall of fame.

If historical data like photos or career stats aren’t stored in a structured way, “You’re not able to respond in a timely manner,” Donegan said. “That’s when the value is very evident.”

Scott noted DAMs often come with features to ensure long-term accessibility, such as automatic upgrades of file formats. They also tend to be “much more visual” than a standard database, she added, as they’re often intended as much or more for external consumption than the organization itself.

“A lot of them are meant to be shown as search libraries or exhibitions where they can put up some of the pictures from the DAM into different websites,” Scott added.