Skip to content

Release Notes: v0.7.0 - Foundation & Quality

Release Date: November 8, 2025 Type: Minor Release (Feature + Quality) Status: Stable

Overview

Version 0.7.0 represents a significant milestone in the PopHealth Observatory's evolution, focusing on robustness, reliability, and extensibility. This release introduces the foundational infrastructure for pesticide biomonitoring analysis while dramatically improving code quality, test coverage, and system stability.

Key Highlights

  • ✅ 96% Test Coverage (up from 90%) - Comprehensive validation and quality assurance
  • 🧪 Pesticide Laboratory Module - Backend foundation for NHANES pesticide analyte ingestion
  • 🔬 Data Validation Framework - Automated CDC data integrity verification
  • 🐛 Critical Bug Fixes - Streamlit UI improvements and manifest generation stability
  • 📊 Enhanced Observatory Coverage - Extended test suite for core functionality

New Features

Pesticide Laboratory Ingestion Module

Status: Backend Foundation (UI in development)

A robust, cycle-aware ingestion pipeline for NHANES pesticide laboratory data:

  • Multi-component support: Handles various pesticide analyte files (urine metabolites, serum organochlorines)
  • Flexible URL fallback: Automatically attempts multiple CDC URL patterns for reliable downloads
  • Derived metrics: Computes log-transformed concentrations and detection flags
  • Reference metadata: Integrates analyte naming and classification metadata
  • Cycle validation: Enforces proper NHANES cycle format with informative error messages

API Example:

from pophealth_observatory.laboratory_pesticides import get_pesticide_metabolites

# Load pesticide metabolite data for a cycle
df = get_pesticide_metabolites("2017-2018")

# Returns long-format DataFrame with:
# - participant_id, cycle, analyte_code
# - concentration_raw, log_concentration, detected_flag
# - source_file metadata

Coverage: 99% (108/109 statements tested)

Data Validation Framework

Programmatic validation against official CDC sources:

  • URL correctness verification: Ensures generated URLs match CDC patterns
  • Row count validation: Compares downloaded data against CDC-reported record counts
  • Component-level checks: Validates demographics, body measures, blood pressure, etc.
  • Multi-component reports: Aggregates validation results with PASS/WARN/FAIL status propagation

API Example:

from pophealth_observatory.validation import run_validation
from pophealth_observatory.observatory import NHANESExplorer

explorer = NHANESExplorer()
report = run_validation(explorer, "2017-2018", ["demographics", "body_measures"])

print(report)  # Human-readable summary
report_dict = report.to_dict()  # Machine-readable JSON

Coverage: 100% (120/120 statements tested)

Improvements

Test Coverage Expansion

Overall Project: 90% → 96% (+6 percentage points)

Module Before After Gain New Tests
laboratory_pesticides.py 83% 99% +16% 6 micro-tests
validation.py 31% 100% +69% 23 comprehensive tests
observatory.py 85% 86% +1% 6 edge case tests

New Test Classes: - TestDownloadXPTFlexible - URL fallback logic, network failures - TestMapToReference - Metadata enrichment edge cases - TestScrapeCDCMetadata - CDC page parsing (9 tests) - TestValidateComponent - Component validation logic (17 tests) - TestRunValidation - Multi-component pipeline (4 tests)

Observatory Enhancements

New Test Coverage: - HTML parsing edge cases (malformed rows, empty cells) - Year range filtering with malformed spans - Manifest generation exception handling - Force refresh cache clearing

Functions Tested: - _parse_component_table robustness - get_detailed_component_manifest error recovery - Year filtering with missing underscore separators

Bug Fixes

Streamlit Application

Line Length Compliance (E501): - Refactored long help strings and filter summaries - Fixed multi-line string formatting for better readability - Ensured black & ruff compliance across UI code

Repository Hygiene

Pre-commit Hook Compliance: - Resolved F841 unused variable warnings - Fixed B007 unused loop variable names - Added B904 exception chaining for better error context - Corrected E712 boolean comparison style

Documentation

New Planning Document: - Added docs/pesticide_biomonitoring_plan.md with comprehensive roadmap - Outlined 6 phased milestones for pesticide feature development - Documented data contracts, edge cases, and testing strategy

What's Next (Roadmap)

Version 0.7.0 lays the foundation for the complete pesticide biomonitoring feature, planned for future releases:

Phase 1 (Current - v0.7.0)

  • ✅ Internal pesticide analyte ingestion
  • ✅ Robust testing infrastructure
  • ✅ Reference metadata loading

Phase 2 (Planned - v0.8.0)

  • 🔄 Streamlit "Pesticide Biomonitoring" tab
  • 🔄 Multi-analyte trend visualization
  • 🔄 Distribution heatmaps
  • 🔄 Demographic disparity views

Phase 3 (Planned - v0.9.0)

  • 🔄 External data source scaffolding (USGS, PDP)
  • 🔄 Agricultural use context integration
  • 🔄 Commodity residue monitoring

Phase 4+ (Future)

  • 🔄 Exploratory health correlation panels
  • 🔄 Mixture/co-exposure analysis
  • 🔄 Advanced survey design weighting

See docs/pesticide_biomonitoring_plan.md for full details.

Breaking Changes

None. This release is fully backward-compatible with v0.6.x.

Upgrade Guide

From v0.6.x to v0.7.0

No action required. All existing code will continue to work without modification.

Optional - New Features:

To use the new pesticide ingestion module:

pip install --upgrade pophealth-observatory

from pophealth_observatory.laboratory_pesticides import (
    get_pesticide_metabolites,
    load_pesticide_reference
)

To use the validation framework:

from pophealth_observatory.validation import run_validation

Technical Details

Dependencies

No new required dependencies. All features work with existing dependency set: - pandas >= 1.3.0 - requests >= 2.25.0 - beautifulsoup4 >= 4.9.0 (for validation scraping)

Testing

Test Suite Stats: - Total Tests: 144 (up from 109) - All Passing: ✅ - Execution Time: ~35 seconds - Coverage: 96% (2174/2256 statements)

Run Tests:

pytest
coverage run -m pytest
coverage report

Performance

No performance regressions. Pesticide ingestion caching maintains sub-second response for repeated queries.

Contributors

This release includes contributions focused on: - Testing infrastructure expansion - Code quality improvements - Foundation for pesticide analytics - Documentation enhancements

Installation

PyPI (when published)

pip install pophealth-observatory==0.7.0

From Source

git clone https://github.com/paulboys/PopHealth-Observatory.git
cd PopHealth-Observatory
git checkout v0.7.0
pip install -e .

Development Install

pip install -e .[dev,docs]

Acknowledgments

This release builds upon the solid foundation of previous versions while prioritizing quality, stability, and extensibility. Special focus on achieving >95% test coverage ensures confidence in future feature development.

Questions or Issues?

Please open an issue on GitHub or refer to the documentation for detailed usage examples and troubleshooting guidance.