Proactive Data Containers (PDC)¶
Proactive Data Containers (PDC) software provides an object-focused data management API, a runtime system with a set of scalable data object management services, and tools for managing data objects stored in the PDC system. The PDC API allows efficient and transparent data movement in complex memory and storage hierarchy. The PDC runtime system performs data movement asynchronously and provides scalable metadata operations to find and manipulate data objects. PDC revolutionizes how data is managed and accessed by using object-centric abstractions to represent data that moves in the high-performance computing (HPC) memory and storage subsystems. PDC manages extensive metadata to describe data objects to find desired data efficiently as well as to store information in the data objects.
More information and publications of PDC is available at https://sdm.lbl.gov/pdc
If you use PDC in your research, please use the following citation:
Byna, Suren, Dong, Bin, Tang, Houjun, Koziol, Quincey, Mu, Jingqing, Soumagne, Jerome, Vishwanath, Venkat, Warren, Richard, and Tessier, François. Proactive Data Containers (PDC) v0.1. Computer Software. https://github.com/hpc-io/pdc. USDOE. 11 May. 2017. Web. doi:10.11578/dc.20210325.1.
Getting Started
Resources
- Hello PDC Example
- API Documentation with Examples
- PDC general APIs
- PDC container APIs
- PDC object APIs
- PDC region APIs
- PDC property APIs
- PDC query APIs
- PDC hist APIs
- PDC Data types
- Basic types
- Histogram structure
- Container info
- Container life time
- Object property public
- Object property
- Object info
- Object structure
- Region info
- Access type
- Query operators
- Query structures
- Selection structure
- Developers notes
- PDC Tools
Developer Documentation
- Developer Notes
- Containers
PDC_cont_close()PDC_cont_init()PDCcont_create()PDCcont_create_col()PDC_cont_create_local()PDC_cont_list_null()PDCcont_close()PDC_cont_end()PDCcont_open()PDCcont_open_col()PDC_cont_get_info()PDCcont_get_info()PDCcont_iter_start()PDCcont_iter_null()PDCcont_iter_next()PDCcont_iter_get_info()PDCcont_persist()PDCprop_set_cont_lifetime()
- Objects
PDC_obj_close()PDC_obj_init()PDCobj_create()PDC_Client_attach_metadata_to_local_obj()PDC_obj_create()PDC_obj_list_null()PDCobj_flush_start()PDCobj_flush_all_start()PDCobj_close()PDC_obj_end()PDCobj_open_common()PDCobj_open()PDCobj_open_col()PDCobj_iter_start()PDCobj_iter_null()PDCobj_iter_next()PDCobj_iter_get_info()PDCprop_set_obj_user_id()PDCprop_set_obj_app_name()PDCprop_set_obj_time_step()PDCprop_set_obj_data_loc()PDCprop_set_obj_tags()PDCprop_set_obj_dims()PDCprop_set_obj_type()PDCprop_set_obj_transfer_region_type()PDCprop_set_obj_consistency_semantics()PDCprop_set_obj_buf()PDCobj_set_dims()PDCobj_get_dims()PDCobj_buf_retrieve()PDC_obj_get_info()PDC_free_obj_info()PDCobj_get_info()PDCobj_del()PDCobj_create_mpi()PDCobj_encode()PDCobj_decode()
- Preperties
- Regions
pdc_region_close()pdc_transfer_request_close()PDC_region_init()PDC_transfer_request_init()PDC_region_list_null()PDCregion_close()PDC_region_end()PDCregion_create()PDCbuf_obj_map()PDCregion_get_info()PDCbuf_obj_unmap()PDCreg_obtain_lock()PDCreg_release_lock()pdc_transfer_requestpdc_transfer_request_start_all_pkgpdc_transfer_request_wait_all_pkgsort_by_data_server_start_all()sort_by_metadata_server_start_all()sort_by_data_server_wait_all()PDCregion_transfer_create()PDCregion_transfer_close()attach_local_transfer_request()remove_local_transfer_request()static_region_partition()pack_region_buffer()set_obj_server_bufs()pack_region_metadata_query()unpack_region_metadata_query()register_metadata()prepare_start_all_requests()finish_start_all_requests()PDC_Client_pack_all_requests()PDC_Client_start_all_requests()PDCregion_transfer_start_all()sorted_array_unions()PDCregion_transfer_start()release_region_buffer()PDCregion_transfer_status()PDCregion_transfer_wait_all()PDCregion_transfer_wait()
- Query
- Analysis
PDC_Server_get_region_data_ptr()iterator_init()PDCobj_data_block_iterator_create()PDCobj_data_iter_create()PDC_get_argv0_()PDC_get_realpath()PDCobj_analysis_register()PDCobj_data_getSliceCount()PDCobj_data_getNextBlock()PDC_analysis_end()PDC_iterator_end()default_pdc_analysis_libcompare_gt()pdc_analysis_registry_init_()pdc_transform_registry_init_()pdc_analysis_registry_finalize_()PDC_add_analysis_ptr_to_registry_()PDCiter_get_nextId()PDC_check_transform()PDC_get_transforms()PDC_add_transform_ptr_to_registry_()PDC_update_transform_server_meta_index()PDC_set_execution_locus()PDC_get_execution_locus()PDC_get_ftnPtr_()HG_TEST_RPC_CB()HG_TEST_RPC_CB()HG_TEST_THREAD_CB()PDC_obj_data_iterator_register()PDC_free_analysis_registry()PDC_free_transform_registry()PDC_free_iterator_cache()analysis_registry_sizetransform_registry_sizeregistered_transform_ftn_count_gi_cache_freediterator_cache_entriesi_cache_indexi_free_indexexecution_locushg_test_thread_pool_ghg_test_thread_pool_fs_gpdc_region_analysis_registrypdc_region_transform_registryclient_register_iterator_rpc_cb()client_register_analysis_rpc_cb()client_register_transform_rpc_cb()PDC_free_obj_info()PDC_Client_send_iter_recv_id()PDC_Client_register_obj_analysis()send_context_gwork_todo_ganalysis_ftn_register_id_gtransform_ftn_register_id_gserver_transform_ftn_register_id_gobject_data_iterator_register_id_gPDC_sample_min_max()ceil_power_of_2()floor_power_of_2()PDC_create_hist()PDC_hist_incr_all()PDC_gen_hist()PDC_free_hist()PDC_print_hist()PDC_dup_hist()PDC_merge_hist()PDC_copy_hist()
- Transformation