Bridging the Gap: How DIGGS DATA PROCESSING is Democratizing Geotechnical Data Exchange

Breaking down silos between Excel spreadsheets, SQL databases, and industry-standard DIGGS XML files

Nicholas Miller, P.E.

In the world of geotechnical engineering, data is everything. From soil boring logs to laboratory test results, the information gathered from subsurface investigations forms the foundation—quite literally—of every infrastructure project. Yet for decades, this critical data has been trapped in various formats, scattered across incompatible systems, and locked away in organizational silos. Data allows you to reduce costs, reduce risk, reduce uncertainty, increase profitability, and create better, safer, more cost-effective infrastructure, but only when it can be accessed, shared, and utilized effectively.

Enter DIGGS DATA PROCESSING, an open-source solution that targets data management, so practitioners have everything they need to make impactful decisions at their fingertips.

The Problem: Islands of Information

Anyone who's worked in geotechnical engineering knows the frustration. Project data lives in Excel spreadsheets on individual laptops. Laboratory results arrive in various formats that can't be easily imported. Historical boring logs are stored as pdfs in unknown locations or Gint files no one knows how to open. When it's time to share data with consultants, regulators, or other stakeholders, teams often resort to PDF exports or manual re-entry—a process that's both time-consuming and error-prone.

Data Interchange for Geotechnical and Geoenvironmental Specialists (DIGGS) is a data exchange standard developed through the cooperation of organizations, academics, and industry, designed specifically to solve this problem. It is anticipated that DIGGS will save state and federal agencies, and other public and private organizations millions of dollars. Savings will be realized through a combination of avoided drilling and laboratory testing costs, and efficiencies afforded by the availability of geotechnical data for multiple projects in a standard format.

But there's been a gap: while DIGGS provides the standard, many practitioners still struggle with the technical implementation. That's where DIGGS DATA PROCESSING comes in.

The Solution: A Bridge Between Worlds

DIGGS DATA PROCESSING  is an open-source project developed in collaboration with the Geo-Institute of ASCE that serves as a comprehensive bridge between the tools geotechnical engineers already use and the industry-standard data formats they need to adopt. At its core, it's a dataflow system that seamlessly connects three critical components:

• Excel interfaces for familiar data entry and visualization
• SQLite databases for robust data storage and querying
• DIGGS 2.6 compliant XML for standardized data exchange

What makes this project remarkable isn't just its technical capabilities, it's how thoughtfully it's designed around real-world workflows.

Built for Real Workflows

The developers of DIGGS DATA PROCESSING understand that adoption of new technologies in engineering requires more than just technical capability—it requires integration with existing processes. The system follows a practical three-step workflow:

Step 1: Start Where You Are Generate standardized Excel templates that include all the necessary sheets for comprehensive geotechnical projects: Project details, borehole information, test methods, samples, field and final stratigraphy, rock coring data, and laboratory results including gradation, consolidation, strength testing, permeability, and more.

Step 2: Centralize and Organize Convert populated Excel files into normalized SQLite databases that eliminate data redundancy, ensure referential integrity, and provide powerful querying capabilities—all while maintaining the flexibility that geotechnical data demands.

Step 3: Share and Archive Export data as DIGGS 2.6 compliant XML files that can be shared with any organization or imported into any DIGGS-compatible system. The exported files include proper XML namespaces, schema validation, units of measure for all measurements, and complete observation wrappers for test data.

But the system is equally powerful in reverse: import existing DIGGS XML files, work with them in familiar databases and spreadsheet environments, then re-export with enhanced compliance and validation.

Open Source and Community-Driven

One of the most compelling aspects of DIGGS DATA PROCESSING is its commitment to open-source development. In an industry where proprietary software often creates vendor lock-in and limits innovation, this project takes a different approach. The entire codebase is freely available on GitHub, encouraging community contributions and ensuring that improvements benefit everyone.

The timing couldn't be better. The 2025 DIGGS Code Jam brought together vendors and developers across the geotechnical data management space to put the DIGGS schema to the test, highlighting both the potential and the challenges of widespread DIGGS adoption. All vendors successfully imported DIGGS files and generated mostly accurate reports. However, only one vendor out of seven was able to complete a full round-trip of the dataset, demonstrating the critical need for tools like DIGGS DATA PROCESSING that prioritize compliance and validation.

The project is under active development, with regular updates addressing both user feedback and evolving industry standards. Its modular architecture, built on the Abstract Factory design pattern, makes it easy for developers to contribute new processors, extend functionality, and adapt the system to emerging needs.

Technical Excellence Meets Practical Design

While DIGGS DATA PROCESSING is built for practical use, it doesn't compromise on technical quality. The system includes:

• Comprehensive data validation that removes invalid entries, ensures proper units of measure, and validates foreign key relationships
• Industry standard compliance with ASTM standards (D1586 for SPT, D4318 for Atterberg Limits), USCS and AASHTO classification systems
• Robust error handling and user-friendly feedback for troubleshooting
• Cross-platform compatibility with both command-line and GUI interfaces

The project also includes a standalone desktop application with a professional interface, drag-and-drop support, real-time progress tracking, and the ability to create enterprise-ready executables that require no Python installation.

Integration Into Current Workflows

One of DIGGS DATA PROCESSING 's greatest strengths is how easily it integrates into existing organizational workflows. For companies already using Excel for data collection, the transition is almost seamless—they can continue using familiar interfaces while gaining the benefits of standardized data management and exchange.

For organizations with existing databases, the system can import DIGGS XML files from other sources, allowing teams to work with external data in their preferred environment before re-exporting with enhanced compliance.

The backend of the software can be run independently from the executable interface through the command line interface making it easy to integrate DIGGS DATA PROCESSING into automated workflows, while the GUI application provides an accessible option for occasional users or those who prefer visual interfaces.

The Future: AI and Advanced Analytics

Perhaps most exciting thing is what DIGGS DATA PROCESSING  enables for the future. AI, machine learning, and data mining are driving this revolution, but data is the fuel that is powering it. Geotechnical engineers have a unique opportunity to leverage these same capabilities, but they must first begin managing data effectively and using data interchange to communicate the data to other organizations.

The development roadmap for DIGGS DATA PROCESSING includes integration with machine learning platforms and design software, positioning it as a foundation for the next generation of geotechnical analysis tools. When data is properly structured and standardized, it becomes possible to:

• Train AI models on historical geotechnical data to improve site characterization and reduce uncertainty
• Automate routine analyses like soil classification and design calculations
• Integrate with design software for seamless transfer from investigation to analysis to design
• Enable predictive analytics that can identify potential construction issues before they occur

Getting Started

For organizations ready to modernize their geotechnical data management, DIGGS DATA PROCESSING offers multiple entry points:

• Researchers and developers can clone the repository and contribute to ongoing development
• Organizations can implement the system gradually, starting with Excel template generation and progressing to full database integration
• Individual practitioners can use the desktop application for project-specific data management and sharing

The project includes comprehensive documentation, sample data, and templates that make it easy to get started regardless of technical background.

A Community Effort

DIGGS DATA PROCESSING represents more than just a software tool—it's part of a broader movement toward open, standardized data management in geotechnical engineering. By providing free, accessible tools that implement industry standards along with the easily accessible SQLite format, projects like this help level the playing field and ensure that best practices in data management aren't limited to organizations with large IT budgets.

The project welcomes contributions from the community, whether that's code contributions, bug reports, feature requests, or simply sharing experiences with implementation. This collaborative approach ensures that the tool continues to evolve to meet real-world needs.

Conclusion: Building the Foundation for Tomorrow

In an industry built on understanding what lies beneath the surface, DIGGS DATA PROCESSING is helping to uncover the hidden value in geotechnical data. By bridging the gap between familiar tools and industry standards, it's making it easier for organizations of all sizes to adopt best practices in data management and exchange.

As the geotechnical engineering community continues to embrace digital transformation, tools like DIGGS DATA PROCESSING will play a crucial role in ensuring that transformation is inclusive, standardized, and built on solid foundations. The future of geotechnical engineering isn't just about better drilling techniques or more sophisticated laboratory tests—it's about unlocking the collective knowledge embedded in decades of subsurface investigations.

DIGGS will revolutionize the way data is shared and transferred for geotechnical engineering. With DIGGS DATA PROCESSING, that revolution is already underway, one Excel spreadsheet and one SQLite database at a time.

For any questions on implementation or comments on how to improve, feel free to reach out to me on LinkedIn at https://www.linkedin.com/in/geotech-nick

Ready to get started? Visit the DIGGS DATA PROCESSING  GitHub repository to download the latest version, explore the documentation, and join the growing community of contributors. Whether you're looking to standardize your organization's data management or contribute to the future of geotechnical data exchange, there's never been a better time to get involved.