DUNE merge-utils Documentation

Indices and tables

• Index

• Module Index

• Search Page

Contents

setup

• Installation and Startup
  • Installation
  • Testing
  • Dependencies

Configuration

• Configuration
  • defaults.yaml

python API

• do_merge
  • checksums()
  • main()
  • merge()
  • merge_hadd()
  • merge_hdf5()
  • merge_lar()
  • merge_tar()
• config
  • load()
  • recursive_update()
  • update()
• io_utils
  • find_fcl()
  • get_inputs()
  • get_timestamp()
  • log_dict()
  • log_list()
  • pkg_dir()
  • read_config_file()
  • set_log_level()
  • setup_log()
  • src_dir()
• metacat_utils
  • MetaCatRetriever
    • MetaCatRetriever.connect()
    • MetaCatRetriever.input_batches()
  • list_extensions()
  • list_field_values()
• meta
  • MergeMetaAll
    • MergeMetaAll.add()
    • MergeMetaAll.valid
    • MergeMetaAll.value
  • MergeMetaMax
    • MergeMetaMax.add()
    • MergeMetaMax.valid
  • MergeMetaMin
    • MergeMetaMin.add()
    • MergeMetaMin.valid
  • MergeMetaNameDict
  • MergeMetaOverride
    • MergeMetaOverride.add()
    • MergeMetaOverride.valid
  • MergeMetaSum
    • MergeMetaSum.add()
    • MergeMetaSum.valid
  • MergeMetaUnion
    • MergeMetaUnion.add()
    • MergeMetaUnion.valid
    • MergeMetaUnion.value
  • MergeMetaUnique
    • MergeMetaUnique.add()
    • MergeMetaUnique.valid
  • check_required()
  • fix()
  • make_name()
  • merged_keys()
  • parents()
  • validate()
• rucio_utils
  • RucioRetriever
    • RucioRetriever.checksum()
    • RucioRetriever.connect()
    • RucioRetriever.dupes
    • RucioRetriever.files
    • RucioRetriever.input_batches()
    • RucioRetriever.missing
    • RucioRetriever.output_chunks()
    • RucioRetriever.process()
    • RucioRetriever.run()
• justin_utils
  • get_distances()
• merge_rse
  • MergeRSE
    • MergeRSE.distance()
    • MergeRSE.get_ping()
    • MergeRSE.nearest_site()
    • MergeRSE.pfns
  • MergeRSEs
    • MergeRSEs.add_pfn()
    • MergeRSEs.add_replicas()
    • MergeRSEs.cleanup()
    • MergeRSEs.connect()
    • MergeRSEs.get_pfns()
    • MergeRSEs.optimize_pfns()
    • MergeRSEs.set_rse_sites()
    • MergeRSEs.site_pfns()
  • check_path()
• merge_set
  • MergeChunk
    • MergeChunk.add()
    • MergeChunk.chunk()
    • MergeChunk.inputs
    • MergeChunk.json
    • MergeChunk.metadata
    • MergeChunk.name
    • MergeChunk.parents
    • MergeChunk.tier
  • MergeFile
    • MergeFile.did
    • MergeFile.format
    • MergeFile.get_fields()
    • MergeFile.name
    • MergeFile.namespace
  • MergeSet
    • MergeSet.add_file()
    • MergeSet.add_files()
    • MergeSet.check_consistency()
    • MergeSet.check_errors()
    • MergeSet.check_missing()
    • MergeSet.check_reachability()
    • MergeSet.check_uniqueness()
    • MergeSet.check_validity()
    • MergeSet.dupes
    • MergeSet.files
    • MergeSet.group_count()
    • MergeSet.group_size()
    • MergeSet.groups()
    • MergeSet.hash
    • MergeSet.set_unreachable()
    • MergeSet.size
  • check_remote_path()
• retriever
  • FileRetriever
    • FileRetriever.add()
    • FileRetriever.connect()
    • FileRetriever.dupes
    • FileRetriever.files
    • FileRetriever.input_batches()
    • FileRetriever.missing
    • FileRetriever.output_chunks()
    • FileRetriever.run()
  • LocalRetriever
    • LocalRetriever.connect()
    • LocalRetriever.get_metadata()
    • LocalRetriever.input_batches()
• scheduler
  • JobScheduler
    • JobScheduler.get_chunks()
    • JobScheduler.run()
  • JustinScheduler
    • JustinScheduler.get_chunks()
    • JustinScheduler.get_cmd()
    • JustinScheduler.json_name()
    • JustinScheduler.run()
    • JustinScheduler.upload_cfg()
  • LocalScheduler
    • LocalScheduler.get_chunks()
    • LocalScheduler.json_name()
    • LocalScheduler.run()
• CheckConfiguration
• MetaValidator
  • MetaValidator()

command line API

• merge
  • main()

references

• Design docs
  • Design considerations
    • Motivation
    • Inputs
      • At the list generation stage, we need methods that
    • Move from Jobsub to Justin for batch running
      • Justin questions 2025-03-24
      • how do we get Justin to run over chunks of files in a list.
      • Methods needed:
    • Delivery methods
    • Merging methods
      • Note on Event counting and tracking
    • Consistency checks
    • Metadata merge:
      • Metadata fields added:
      • enhancements to TypeChecker.py
      • Sam4users
    • Where this runs
    • Suggestions for future versions
    • Design ideas:
• Notes on sphinx
  • rst format
  • conda install
  • Github actions
  • Google indexing Notes
• Related documentation
  • metacat
  • Data Dispatcher (dd)
  • samweb
  • JustIn

• Acknowledgements